Is it mandatory in pytorch to add modules to ModuleList to access its parameters
I read some posts about ModuleList and all of them said that adding modules to ModuleList gives access to parameters of the Neural Network but in “Training a classifier” example of 60 mins blitz pytorch tutorial the modules are not added to any ModuleList and still the parameters could be accessed using

optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

This is confusing. Please clarify how the parameters are accessible even though the modules have not been added to any ModuleList

class mmodel(nn.Module):
    def __init__(self):
        super(mmodel,self).__init__()
        self.lst=[]
        self.lst.append(nn.Linear(1,1))
        self.lst.append(nn.Linear(1,1))
        self.mlist = nn.ModuleList([nn.Linear(2,1,bias=False),nn.Linear(3,1)])
    def forward(self,x):
        for m in self.lst:
            print(type(m))
            x = m(x)
        for m in self.mlist:
            print(type(m))

mm = mmodel()
mm(X)
print(list(mm.parameters())) 

&lt;class 'torch.nn.modules.linear.Linear'&gt;
&lt;class 'torch.nn.modules.linear.Linear'&gt;
&lt;class 'torch.nn.modules.linear.Linear'&gt;
&lt;class 'torch.nn.modules.linear.Linear'&gt;
[Parameter containing:
tensor([[0.2302, 0.3712]], requires_grad=True), Parameter containing:
tensor([[-0.3451, -0.0274,  0.3990]], requires_grad=True), Parameter containing:
tensor([0.3258], requires_grad=True)]


As evident from the above outputs the parameters of modules added in a python list are not visible in the forward method. Only the parameters of the modules added in the ModuleList are visible in the forward method

Calling module.parameters() lists all nn.Parameter of the module. Concretely, every attribute on the module that is an instance of nn.Parameter will be in that list. Additionally to listing all the parameters of that module, it will also list all parameters of the submodules (unless module.parameters(recurse=False) is used). That means it will also collect all parameters of every attribute that is an instance of nn.Module, which includes all subclasses.

However, if you assign an ordinary list of modules to your module, they won't be included since that is an instance of list, but not nn.Module. For convenience, nn.ModuleList can be used in place of a regular Python list. nn.ModuleList is an instance of nn.Module but acts similar to a list, albeit much more restricted.

Let's take a look at an example to understand what's considered a submodule:

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        # This is an nn.Module, therefore it is considered a submodule
        self.single_linear = nn.Linear(10, 20)

        # This is a list, that happens to contain nn.Modules.
        # This is not a submodule.
        self.linears = [nn.Linear(20, 10), nn.Linear(20, 30), nn.Linear(20, 40)]

        # A list with ints, clearly not a submodule either.
        self.numbers = [0, 1, 2, 3]

        # This is a module, but acts like an ordinary list
        # So it's a submodule as well.
        self.linears_module_list = nn.ModuleList(
            [nn.Linear(20, 10), nn.Linear(20, 30), nn.Linear(20, 40)]
        )


In this model we have assigned 4 new attributes, single_linear, linears, numbers and linears_module_list. When they are assigned, PyTorch checks whether they are instances of nn.Module, and if they are, they will be registered as submodules.

We can verify that (leaving out numbers, because that is pretty clear):

model = Model()

isinstance(model.single_linear, nn.Module)  # =&gt; True

isinstance(model.linears, nn.Module)  # =&gt; False
isinstance(model.linears, list)       # =&gt; True

isinstance(model.linears_module_list, nn.Module)  # =&gt; True
isinstance(model.linears_module_list, list)       # =&gt; False


Only single_linear and linears_module_list are in fact submodules, as they are the only attributes that are instances of nn.Module. The submodules can be seen with model._modules, but that's an implementation detail and you shouldn't rely on it.

# The registered modules
model._modules
# =&gt; OrderedDict([
#       ('single_linear', Linear(in_features=10, out_features=20, bias=True)),
#       ('linears_module_list', ModuleList(
#           (0): Linear(in_features=20, out_features=10, bias=True)
#           (1): Linear(in_features=20, out_features=30, bias=True)
#           (2): Linear(in_features=20, out_features=40, bias=True)
#       )
#   )])

