Multiple parallel image-encoders in pytorch
I have an image encoder which can generate output features. I want to split my image into several patches (around 16) and feed each patch to a separate image encoder (the parameters for each encoder are different).
Initially, it's like:
Input -&gt; Encoder -&gt; Output
I want to change it like this:
patch-input1 -&gt; Encoder1 -&gt; output1
patch-input2 -&gt; Encoder2 -&gt; output2
...
patch-inputN -&gt; EncoderN -&gt; outputN

I'm using nn.Module to create the model class. The number of patches N is not fixed and is determined during the initialisation of the model so I wanted to use the list of objects (in __init__() function) to create multiple instances of the same encoder and call them over each part-input in a loop in the forward function.
When I run inference on a sample input, it doesn't show any error and I print out the number of trainable parameters using torchsummary module.
The problem is I do not see the parameters corresponding to the encoders in the number of parameters. More specifically in the single encoder pipeline there are around one million parameters and in the multiple encoder pipeline, params there are only around 200 thousand. The later's summary doesn't even show the description of various layers in the encoder.
A code snippet of my implementation would look like this:
class Patch(nn.Module):
   '''
   takes image tensor input
   returns a list of patch tensors
   '''
class Encoder(nn.Module):
   '''
   definition of the encoder
   '''
class Model(nn.Module):
   def __init__(self, patches=10, *kwargs):
      super().__init__()
      self.patch = Patch()
      self.enc = []
      for i in range(patches):
         enc.append(Encoder())
   def forward(self, x):
      '''
      patches is a list of tensors formed using an image 
      tensor
      '''
      patches = self.patch(x)
      output = []
      for i in range(patches):
         output.append(self.enc[i](patch[i]))
      output_feats = torch.cat(output, dim=0)


Is there a better way to implement parallel encoders? What am I doing wrong?

You need to use a nn.ModuleList in order to properly register the list of Patch modules inside Model nn.Module:
class Model(nn.Module):
   def __init__(self, patches=10, *kwargs):
      super().__init__()
      self.patch = Patch()
      enc = []
      for i in range(patches):
         enc.append(Encoder())
      self.enc = nn.ModuleList(enc)

In a slightly refactored form:
class Model(nn.Module):
   def __init__(self, patches=10, *kwargs):
      super().__init__()
      self.patch = Patch()
      self.enc = nn.ModuleList(Encoder() for _ in range(patches))

   def forward(self, x):
      patches = self.patch(x)
      output_feats = torch.cat([enc(patch) for enc, patch in zip(self.enc, patches])

