How to create properly shaped input using Dataloader?
I am trying to input an image and a vector as input to the model. The image has the correct shape of 4d, but the vector that I input doesn't have such shape. The image size is 424x512 while the vector is of shape (18,). After using dataloader, I get batches of shape (50x1x424x512) and (50x18). Model gives error as it needs the vector shape to be 4d too. How do I do that?
Here is my code : 

def loadTrainingData_B(args):
    fdm = []
    tdm = []
    parameters = []
    for i in image_files[:4]:
        try:
            false_dm = np.fromfile(join(ref, i), dtype=np.int32)
            false_dm = Image.fromarray(false_dm.reshape((424, 512, 9)).astype(np.uint8)[:,:,1])
            fdm.append(false_dm)
            true_dm = np.fromfile(join(ref, i), dtype=np.int32)
            true_dm = Image.fromarray(true_dm.reshape((424, 512, 9)).astype(np.uint8)[:,:,1])
            tdm.append(true_dm)
            pos = param_filenames.index(i)
            param = np.array(params[pos, 1:])
            param = np.where(param == '-point-light-source', 1, param).astype(np.float64)
            parameters.append(param)
        except:
            print('[!] File {} not found'.format(i))
    return (fdm, parameters, tdm)

class Flat_ModelB(Dataset):
    def __init__(self, args, train=True, transform=None):
        self.args = args
        if train == True:
            self.fdm, self.parameters, self.tdm = loadTrainingData_B(self.args)
        else:
            self.fdm, self.parameters, self.tdm = loadTestData_B(self.args)
        self.data_size = len(self.parameters)
        self.transform = transforms.Compose([transforms.ToTensor()])

    def __getitem__(self, index):
        return (self.transform(self.fdm[index]).double(), torch.from_numpy(self.parameters[index]).double(), self.transform(self.tdm[index]).double())

    def __len__(self):
        return self.data_size



The error I get is :

RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 1 5 5, but got 2-dimensional input of size [50, 18] instead


Here is the model : 

class Model_B(nn.Module):
    def __init__(self, config):
        super(Model_B, self).__init__()
        self.config = config
        # CNN layers for fdm
        self.layer1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(16))
        self.layer2 = nn.Sequential(
            nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(32))
        self.layer3 = nn.Sequential(
            nn.Conv2d(in_channels=32, out_channels=32, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(32))
        self.layer4 = nn.Sequential(
            nn.ConvTranspose2d(in_channels=32, out_channels=32, kernel_size=5, stride=2, padding=2, output_padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(32))
        self.layer5 = nn.Sequential(
            nn.ConvTranspose2d(in_channels=32, out_channels=16, kernel_size=5, stride=2, padding=2,output_padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(16))
        self.layer6 = nn.Sequential(
            nn.ConvTranspose2d(in_channels=16, out_channels=1, kernel_size=5, stride=2, padding=2, output_padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(1))
        # CNN layer for parameters
        self.param_layer1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=32, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(32))

    def forward(self, x, y):
        out = self.layer1(x)
        out_param = self.param_layer1(y)
        print("LayerParam 1 Output Shape : {}".format(out_param.shape))
        print("Layer 1 Output Shape : {}".format(out.shape))
        out = self.layer2(out)
        print("Layer 2 Output Shape : {}".format(out.shape))
        out = self.layer3(out)
        # out = torch.cat((out, out_param), dim=2)
        print("Layer 3 Output Shape : {}".format(out.shape))
        out = self.layer4(out)
        print("Layer 4 Output Shape : {}".format(out.shape))
        out = self.layer5(out)
        print("Layer 5 Output Shape : {}".format(out.shape))
        out = self.layer6(out)
        print("Layer 6 Output Shape : {}".format(out.shape))
        return out


and the method by which I access the data :

for batch_idx, (fdm, parameters) in enumerate(self.data):
            if self.config.gpu:
                fdm = fdm.to(device)
                parameters = parameters.to(device)
                print('shape of parameters for model a : {}'.format(parameters.shape))

            output = self.model(fdm)
            loss = self.criterion(output, parameters)


Edit : 
I think my code is incorrect as I am trying to apply convolutions over a vector of (18). I tried to copy the vector and make it (18x64) and then input it. It still doesnt work and gives this output : 

RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 1 5 5, but got 3-dimensional input of size [4, 18, 64] instead


I am not sure how to concatenate an 18 length vector to the output of layer 3, if I can't do any of these things.

Looks like you are training an autoencoder model and want to parameterize it with some additional vector input in the bottleneck layer. If you want to perform some transformations on it then you have to decide whether if you need any spatial dependencies. Given the constant input size (N, 1, 424, 512), the output of layer3 will have a shape (N, 32, 53, 64). You have a lot of options, depending on you desired model performance:


Use a nn.Linear with activations to transform the parameter vector. Then you might add extra spatial dimensions and repeat this vector in all spatial locations:


img = torch.rand((1, 1, 424, 512))
vec = torch.rand(1, 19)

layer3_out = model(img)
N, C, H, W = layer3_out.shape

param_encoder = nn.Sequential(nn.Linear(19, 30), nn.ReLU(), nn.Linear(30, 10))
param = param_encoder(vec)
param = param.unsqueeze(-1).unsqueeze(-1).expand(N, -1, H, W)
encoding = torch.cat([param, layer3_out], dim=1)



Use transposed convolutions to upsample your parameter vector to the size of layer3 output. But that would be harder to implement as you have to calculate exact output shape to fit with  (N, 32, 53, 64)
Transform input vector with MLP using nn.Linear to the size 2x of the channels in layer3 output. Then use so called Feature-wise transformations to scale and shift feature maps from layer3. 


I would recomend to start with the first option since this is the simplest one to implement and then try others. 
