Adding a linear layer to an existing model on Pytorch
I'm trying to add a new layer to an existing network (as the first layer) and train it on the original input. When I add a convolutional layer everything works perfectly but when I change it to linear it doesn't seem to train. Any ideas why?
Here is the whole network:

class ActorCritic(torch.nn.Module): #original model
    def __init__(self, num_inputs, action_space):
        super(ActorCritic, self).__init__()
        self.conv1 = nn.Conv2d(num_inputs, 32, 3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(32, 32, 3, stride=2, padding=1)
        self.conv3 = nn.Conv2d(32, 32, 3, stride=2, padding=1)
        self.conv4 = nn.Conv2d(32, 32, 3, stride=2, padding=1)

        self.lstm = nn.LSTMCell(32 * 3 * 3, 256)

        num_outputs = action_space.n
        self.critic_linear = nn.Linear(256, 1)
        self.actor_linear = nn.Linear(256, num_outputs)

    def forward(self, inputs):
        inputs, (hx, cx) = inputs
        x = F.elu(self.conv1(inputs))
        x = F.elu(self.conv2(x))
        x = F.elu(self.conv3(x))
        x = F.elu(self.conv4(x))
        x = x.view(-1, 32 * 3 * 3)
        hx, cx = self.lstm(x, (hx, cx))
        x = hx
        return self.critic_linear(x), self.actor_linear(x), (hx, cx)

class TLModel(torch.nn.Module): #new model
    def __init__(self, pretrained_model, num_inputs):
        super(TLModel, self).__init__()
        self.new_layer = nn.Linear(1*1*42*42, 1*1*42*42)
        self.pretrained_model = pretrained_model

    def forward(self, inputs):
        inputs, (hx, cx) = inputs
        x = F.elu(self.new_layer(inputs.view(-1, 1*1*42*42)))
        return self.pretrained_model((x.view(1,1,42,42), (hx, cx)))


I tried different activation functions (not just elu). it works with conv:

 class TLModel(torch.nn.Module):
    def __init__(self, pretrained_model, num_inputs):
        super(TLModel, self).__init__()
        self.new_layer = nn.Conv2d(num_inputs, num_inputs, 1)
        self.pretrained_model = pretrained_model

    def forward(self, inputs):
        inputs, (hx, cx) = inputs
        x = F.elu(self.new_layer(inputs))
        return self.pretrained_model((x, (hx, cx)))


The number of inputs is 1 and the size of an input is 1x1x42x42

It would be useful if you had supplied the error message. From what you have written, I can only guess that you have forgotten to squeeze your input. You write your input is of size 1x1x42x42, i.e. it is 4-dimensional. nn.Conv2D expects a 4-dimensional input. nn.Linear instead expects a 2-dimensional input. 

Therefore, try to call input = input.squeeze() before feeding it to your model. This removes singleton dimensions, and hence will make your input 2-dimensional as there are two singleton dimensions. 

As a side note, nn.Linear expects input of dimension  batch_size x feat_dim. Does a linear layer really make sense on your data?
As another side note, when people usually add layers to networks they put them in the end not the beginning, but I trust you have good reasons to do so and know what you're doing :) 

Good Luck!
