title: feeding a 2D intermediate feature and not a 2D image with 3 channels to Inception V3 for fine-tuning in case of gigapixel images
body: <p>I have gigapixel images which requires me to patch them into either of <code>256x256</code>, <code>512x512</code>, or <code>1024x1024</code> smaller images (2D images with 3 channels). I pass these images to a raw (frozen) ResNet18 to extract their 1D embeddings (feature vectors) of size 512. Here' the model I am trying to use for fine-tuning of a pre-trained Inception V3 on a 2-class binary classification problem which has inputs of shapes batch_size , number of patches, resnet18 extracted feature dimension (512 here), so if <code>batch_size=64</code> and the large initial image has 6000 patches, I end up with 64, 6000, 512 as an <strong>intermediate representation</strong>. I want to use Inception V3 now to fine-tune on my own labels and my own intermediate representations. How can I do this? current error is <code>RuntimeError: Expected 4-dimensional input for 4-dimensional weight [32, 3, 3, 3], but got 3-dimensional input of size [16, 3, 512] instead</code> shown fully at the end of the post.</p>
  <pre><code>model_ft = models.inception_v3(pretrained=True)
  for param in model_ft.parameters():
      param.requires_grad = False
  
  # Parameters of newly constructed modules have requires_grad=True by default
  # Handle the auxilary net
  num_ftrs = model_ft.AuxLogits.fc.in_features
  model_ft.AuxLogits.fc = nn.Linear(num_ftrs, 2)
  # Handle the primary net
  num_ftrs = model_ft.fc.in_features
  model_ft.fc = nn.Linear(num_ftrs, 2)
  model = model_ft
  model = nn.DataParallel(model)
  
  if args.resume:
      print('load model{}'.format(args.resume))
      model.load_state_dict(torch.load(args.resume))
  
  if torch.cuda.is_available():
      model = model.cuda()
  
  </code></pre>
  <p>and</p>
  <pre><code># training, validation, and test phases
  
  for epoch in range(num_epochs):
  
      train_loss = 0.
      total = 0.
  
      current_lr = optimizer.param_groups[0]['lr']
      print('\n=&gt;Epoches %i, learning rate = %.7f, previous best = %.4f' %
            (epoch+1, current_lr, best_val_acc))
      
      train_epoch_loss = 0
      train_epoch_labels = []
      train_epoch_preds = []
      
      val_epoch_loss = 0
      val_epoch_labels = []
      val_epoch_preds = []
      val_epoch_labels_arr = []
      val_epoch_preds_arr = []
      
  
      epoch_loss = 0
      epoch_accuracy = 0
      
      if train:
          exp_lr_scheduler.step()
          print('training...')
          torch.autograd.set_detect_anomaly(True)
          for i_batch, sample_batched in enumerate(dataloader_train):  
              print(type(sample_batched))
              print(sample_batched.keys())
              # sample_batched['image'] is an array of tensors with len of batch_size
              feats = torch.stack(sample_batched['image']) 
              print(&quot;feature size shape: &quot;, feats.shape)
              labels = torch.as_tensor(sample_batched['label']).cuda() 
              output = model(feats)
              loss = kornia.losses.focal_loss(output, labels, **kwargs)
              print('train loss is: ', loss)
              optimizer.zero_grad()
              loss.backward()
              optimizer.step()
              acc = (output.argmax(dim=1) == labels).float().mean()
              train_preds = output.argmax(dim=1)
              print('train preds are: ', train_preds)
              train_epoch_preds.extend(train_preds.cpu().numpy())
              train_epoch_labels.extend(labels.cpu().numpy())
              epoch_accuracy += acc / len(dataloader_train)
              epoch_loss += loss / len(dataloader_train)
              print('epoch accuracy: ', epoch_accuracy)
              
          train_epoch_accuracy = accuracy_score(train_epoch_labels, train_epoch_preds)
          print('train_epoch_accuracy: ', train_epoch_accuracy)
  
      if not test:
          print('not test')
  
      with torch.no_grad():
          print('Evaluating...')
          print('epoch is: ', epoch)
          
          epoch_val_accuracy = 0
          epoch_val_loss = 0
          epoch_val_preds = []
          epoch_val_labels = []
          
          #model.eval()
          print(&quot;evaluating...&quot;)
          total = 0.
          batch_idx = 0
          val_preds = []
          val_labels = []
          predictions = []
          actuals = []
          
          
          for i_batch, sample_batched in enumerate(dataloader_val):
              feats = torch.stack(sample_batched['image']) 
              labels = torch.as_tensor(sample_batched['label']).cuda() 
              val_output = model(feats)
              val_loss = kornia.losses.focal_loss(val_output, labels, **kwargs)
              acc = (val_output.argmax(dim=1) == labels).float().mean()
  
  </code></pre>
  <p>Error is:</p>
  <pre><code>
  =&gt;Epoches 1, learning rate = 0.0000000, previous best = 0.0000
  training...
  &lt;class 'dict'&gt;
  dict_keys(['image', 'label', 'id'])
  feature size shape:  torch.Size([64, 419, 512])
  
  ---------------------------------------------------------------------------
  RuntimeError                              Traceback (most recent call last)
  Input In [68], in &lt;module&gt;
       35 print(&quot;feature size shape: &quot;, feats.shape)
       36 labels = torch.as_tensor(sample_batched['label']).cuda() 
  ---&gt; 37 output = model(feats)
       38 loss = kornia.losses.focal_loss(output, labels, **kwargs)
       39 print('train loss is: ', loss)
  
  File ~/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py:1102, in Module._call_impl(self, *input, **kwargs)
     1098 # If we don't have any hooks, we want to skip the rest of the logic in
     1099 # this function, and just call forward.
     1100 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
     1101         or _global_forward_hooks or _global_forward_pre_hooks):
  -&gt; 1102     return forward_call(*input, **kwargs)
     1103 # Do not call functions when jit is used
     1104 full_backward_hooks, non_full_backward_hooks = [], []
  
  File ~/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py:168, in DataParallel.forward(self, *inputs, **kwargs)
      166     return self.module(*inputs[0], **kwargs[0])
      167 replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
  --&gt; 168 outputs = self.parallel_apply(replicas, inputs, kwargs)
      169 return self.gather(outputs, self.output_device)
  
  File ~/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py:178, in DataParallel.parallel_apply(self, replicas, inputs, kwargs)
      177 def parallel_apply(self, replicas, inputs, kwargs):
  --&gt; 178     return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  
  File ~/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py:86, in parallel_apply(modules, inputs, kwargs_tup, devices)
       84     output = results[i]
       85     if isinstance(output, ExceptionWrapper):
  ---&gt; 86         output.reraise()
       87     outputs.append(output)
       88 return outputs
  
  File ~/research/venv/dpcc/lib/python3.8/site-packages/torch/_utils.py:434, in ExceptionWrapper.reraise(self)
      430 except TypeError:
      431     # If the exception takes multiple arguments, don't try to
      432     # instantiate since we don't know how to
      433     raise RuntimeError(msg) from None
  --&gt; 434 raise exception
  
  RuntimeError: Caught RuntimeError in replica 0 on device 0.
  Original Traceback (most recent call last):
    File &quot;/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py&quot;, line 61, in _worker
      output = module(*input, **kwargs)
    File &quot;/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py&quot;, line 1102, in _call_impl
      return forward_call(*input, **kwargs)
    File &quot;/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torchvision/models/inception.py&quot;, line 200, in forward
      x, aux = self._forward(x)
    File &quot;/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torchvision/models/inception.py&quot;, line 139, in _forward
      x = self.Conv2d_1a_3x3(x)
    File &quot;/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py&quot;, line 1102, in _call_impl
      return forward_call(*input, **kwargs)
    File &quot;/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torchvision/models/inception.py&quot;, line 472, in forward
      x = self.conv(x)
    File &quot;/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py&quot;, line 1102, in _call_impl
      return forward_call(*input, **kwargs)
    File &quot;/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/conv.py&quot;, line 446, in forward
      return self._conv_forward(input, self.weight, self.bias)
    File &quot;/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/conv.py&quot;, line 442, in _conv_forward
      return F.conv2d(input, weight, bias, self.stride,
  RuntimeError: Expected 4-dimensional input for 4-dimensional weight [32, 3, 3, 3], but got 3-dimensional input of size [16, 3, 512] instead
  
  </code></pre>