How to create a list of modulelists
Is it ok to create a python-list of PyTorch modulelists? 
If for example, I want to have a few Conv1d in a layer and then another layer with different Conv1d. In each layer I need to do a different manipulation on the output depending on the layer number.  What is the correct way to build this "python-list" of modulelists?

This way:

    class test(nn.Module):
        def __init__(...):
            self.modulelists = []
            for i in range(4):
                self.modulelists.append(nn.ModuleList([nn.Conv1d(10, 10, kernel_size=5) for _ in range(5)]))


or this way:

    class test(nn.Module):
        def __init__(...):
            self.modulelists = nn.ModuleList()
            for i in range(4):
                self.modulelists.append(nn.ModuleList([nn.Conv1d(10, 10, kernel_size=5) for _ in range(5)]))


Thanks

You need to register all sub-modules of your net properly so that pytorch can have access to their parameters, buffers etc.
This can be done only if you use proper containers.
If you store sub-modules in a simple pythonic list pytorch will have no idea there are sub modules there and they will be ignored.

So, if you use simple pythonic list to store the sub-modules, when you call, for instance, model.cuda() the parameters of the sub-modules in the list will not be transferred to GPU, but rather remain on CPU. If you call model.parameters() to pass all trainable parameters to an optimizer, all the sub-modules parameters will not be detected by pytorch and thus the optimizer will not "see" them.
