Why is my dynamic pytorch model definition not complete?
I tried to create a custom pytorch model class in a way that would allow variable number of hidden layers. Everything seems to &quot;work&quot; in the code, however even if I set 10 hidden layers, none of them show up in the print out of the model definition. I am wondering why this is happening? I can see obviously it has something to do with the way I use a loop to apply the hidden layers.
Code:
class FFModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(FFModel, self).__init__()
        self.input_layer = nn.Linear(input_dim, hidden_dim).to(device)
        self.hidden_layers = [ nn.Linear(hidden_dim, hidden_dim).to(device) for _ in range(layer_dim) ]
        self.activation = nn.LeakyReLU().to(device)
        self.output_layer = nn.Linear(hidden_dim, output_dim).to(device)

    def forward(self, x):
        y_hat = self.activation( self.input_layer(x) )
        for hidden_layer in self.hidden_layers:
            y_hat = self.activation( hidden_layer(y_hat) )
        out = self.output_layer(y_hat)
        return out

model = FFModel(2, 5, 10, 1)
model

Output:
FFModel(
  (input_layer): Linear(in_features=2, out_features=5, bias=True)
  (activation): LeakyReLU(negative_slope=0.01)
  (output_layer): Linear(in_features=5, out_features=1, bias=True)
)

Indeed, if I inspect deeper and parse through the parameters, even though this model should have 10 layers, the parameters are not correct for a model of 10 hidden layers. Clearly this model doesn't have 10 layers as I thought it would.
Why are the hidden layers all gone?
for param in model.parameters():
    print (param)

out:
Parameter containing:
tensor([[-0.1064, -0.0171],
        [-0.5892, -0.5391],
        [-0.6420, -0.5128],
        [ 0.0128,  0.2155],
        [ 0.3406,  0.1825]], device='cuda:0', requires_grad=True)
Parameter containing:
tensor([-0.5432, -0.0650,  0.6447,  0.4615,  0.4772], device='cuda:0',
       requires_grad=True)
Parameter containing:
tensor([[ 0.0615,  0.0901,  0.3399, -0.3341,  0.2029]], device='cuda:0',
       requires_grad=True)
Parameter containing:
tensor([0.1742], device='cuda:0', requires_grad=True)


PyTorch cannot see your list object. You need to use nn.ModuleList:
self.hidden_layers = nn.ModuleList([nn.Linear(hidden_dim, hidden_dim).to(device) for _ in range(layer_dim)])

