<p dir="auto">When using nn.ParameterList and nn.DataParallel at the same time, the length of ParameterList will be 0 after DataParallel.</p>
<p dir="auto">See and run this code (with pytorch=1.6+cu101 installed by pip):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.param = torch.nn.ParameterList([torch.nn.Parameter(torch.zeros(3,3)) for _ in range(16)])
        print('Before DataParallel: ' + str(len(self.param)))

    def forward(self, x):
        print('After DataParallel: ' + str(len(self.param)))
        return x


model = Model()
model = model.to(torch.device('cuda'))
model = torch.nn.DataParallel(model)
x = model(torch.zeros(3,3))"><pre class="notranslate"><code class="notranslate">import torch

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.param = torch.nn.ParameterList([torch.nn.Parameter(torch.zeros(3,3)) for _ in range(16)])
        print('Before DataParallel: ' + str(len(self.param)))

    def forward(self, x):
        print('After DataParallel: ' + str(len(self.param)))
        return x


model = Model()
model = model.to(torch.device('cuda'))
model = torch.nn.DataParallel(model)
x = model(torch.zeros(3,3))
</code></pre></div>
<p dir="auto">And the results are:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Before DataParallel: 16
After DataParallel: 0
After DataParallel: 0
After DataParallel: 0
After DataParallel: 0"><pre class="notranslate"><code class="notranslate">Before DataParallel: 16
After DataParallel: 0
After DataParallel: 0
After DataParallel: 0
After DataParallel: 0
</code></pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pietern/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pietern">@pietern</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrshenli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrshenli">@mrshenli</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pritamdamania87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pritamdamania87">@pritamdamania87</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zhaojuanmao/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zhaojuanmao">@zhaojuanmao</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/satgera/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/satgera">@satgera</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rohan-varma/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rohan-varma">@rohan-varma</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/aazzolini/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/aazzolini">@aazzolini</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xush6528/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xush6528">@xush6528</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/osalpekar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/osalpekar">@osalpekar</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jiayisuse/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jiayisuse">@jiayisuse</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/agolynski/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/agolynski">@agolynski</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/albanD/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/albanD">@albanD</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mruberry/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mruberry">@mruberry</a></p>
<h2 dir="auto">🐛 Bug</h2>
<p dir="auto">When I use nn.DataParallel to wrap an nn.Module X, nn.Parameter in X is not copied to gpus in the forward pass. I think nn.Parameter can be considered as a part of module parameters, so it should be treated like other nn.Module parameters in X as well. Is it an intentional design?</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">test.py:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sys
import torch
import torch.nn as nn
import torch.nn.functional as F


gpus = list(map(int, sys.argv[1].split(',')))


class Net(nn.Module):
    def __init__(self):
        super().__init__()

        self.alpha = nn.ParameterList()

        for i in range(4):
            self.alpha.append(nn.Parameter(1e-3*torch.randn(i+2, 5)))

        self.cnn = nn.Conv2d(1, 1, 1, 1, 1)


    def forward(self, x):
        print(self.alpha)
        print(self.cnn)
        return x


if __name__ == '__main__':
    net = Net().cuda()
    if len(gpus) &gt; 1:
        net = nn.DataParallel(net, device_ids=gpus)

    net(torch.rand(4, 5))"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">sys</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span> <span class="pl-k">as</span> <span class="pl-s1">nn</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">functional</span> <span class="pl-k">as</span> <span class="pl-v">F</span>


<span class="pl-s1">gpus</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-en">map</span>(<span class="pl-s1">int</span>, <span class="pl-s1">sys</span>.<span class="pl-s1">argv</span>[<span class="pl-c1">1</span>].<span class="pl-en">split</span>(<span class="pl-s">','</span>)))


<span class="pl-k">class</span> <span class="pl-v">Net</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>):
        <span class="pl-en">super</span>().<span class="pl-en">__init__</span>()

        <span class="pl-s1">self</span>.<span class="pl-s1">alpha</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">ParameterList</span>()

        <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">4</span>):
            <span class="pl-s1">self</span>.<span class="pl-s1">alpha</span>.<span class="pl-en">append</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Parameter</span>(<span class="pl-c1">1e-3</span><span class="pl-c1">*</span><span class="pl-s1">torch</span>.<span class="pl-en">randn</span>(<span class="pl-s1">i</span><span class="pl-c1">+</span><span class="pl-c1">2</span>, <span class="pl-c1">5</span>)))

        <span class="pl-s1">self</span>.<span class="pl-s1">cnn</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Conv2d</span>(<span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>)


    <span class="pl-k">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">x</span>):
        <span class="pl-en">print</span>(<span class="pl-s1">self</span>.<span class="pl-s1">alpha</span>)
        <span class="pl-en">print</span>(<span class="pl-s1">self</span>.<span class="pl-s1">cnn</span>)
        <span class="pl-k">return</span> <span class="pl-s1">x</span>


<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">'__main__'</span>:
    <span class="pl-s1">net</span> <span class="pl-c1">=</span> <span class="pl-v">Net</span>().<span class="pl-en">cuda</span>()
    <span class="pl-k">if</span> <span class="pl-en">len</span>(<span class="pl-s1">gpus</span>) <span class="pl-c1">&gt;</span> <span class="pl-c1">1</span>:
        <span class="pl-s1">net</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">DataParallel</span>(<span class="pl-s1">net</span>, <span class="pl-s1">device_ids</span><span class="pl-c1">=</span><span class="pl-s1">gpus</span>)

    <span class="pl-en">net</span>(<span class="pl-s1">torch</span>.<span class="pl-en">rand</span>(<span class="pl-c1">4</span>, <span class="pl-c1">5</span>))</pre></div>
<p dir="auto">When I run <code class="notranslate">python3 test.py 0</code> (which means device_id = [0]), the output is</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ParameterList(
    (0): Parameter containing: [torch.cuda.FloatTensor of size 2x5 (GPU 0)]
    (1): Parameter containing: [torch.cuda.FloatTensor of size 3x5 (GPU 0)]
    (2): Parameter containing: [torch.cuda.FloatTensor of size 4x5 (GPU 0)]
    (3): Parameter containing: [torch.cuda.FloatTensor of size 5x5 (GPU 0)]
)
Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1), padding=(1, 1))"><pre class="notranslate"><code class="notranslate">ParameterList(
    (0): Parameter containing: [torch.cuda.FloatTensor of size 2x5 (GPU 0)]
    (1): Parameter containing: [torch.cuda.FloatTensor of size 3x5 (GPU 0)]
    (2): Parameter containing: [torch.cuda.FloatTensor of size 4x5 (GPU 0)]
    (3): Parameter containing: [torch.cuda.FloatTensor of size 5x5 (GPU 0)]
)
Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1), padding=(1, 1))
</code></pre></div>
<p dir="auto">However, when I run <code class="notranslate">python3 test.py 0,1</code> (which means device_id = [0, 1]), the output is</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ParameterList()
Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1), padding=(1, 1))
ParameterList()
Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1), padding=(1, 1))"><pre class="notranslate"><code class="notranslate">ParameterList()
Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1), padding=(1, 1))
ParameterList()
Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1), padding=(1, 1))
</code></pre></div>
<p dir="auto">Only nn.Module is copied to gpus in forward pass.<br>
How can I use and train nn.Parameter just like nn.Module with nn.DataParallel?</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">When the nn.Module X is wrapped with nn.DataParallel, both nn.Module and nn.Parameter in X should be copied to gpus.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">PyTorch version: 1.6.0.dev20200401+cu101<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.1</p>
<p dir="auto">OS: Arch Linux<br>
GCC version: (Arch Linux 9.3.0-1) 9.3.0<br>
CMake version: version 3.17.0</p>
<p dir="auto">Python version: 3.8<br>
Is CUDA available: Yes<br>
CUDA runtime version: 10.2.89<br>
GPU models and configuration:<br>
GPU 0: GeForce GTX TITAN X<br>
GPU 1: GeForce GTX 1060 6GB</p>
<p dir="auto">Nvidia driver version: 440.64<br>
cuDNN version: /usr/lib/libcudnn.so.7.6.5</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip3] numpy==1.18.2<br>
[pip3] torch==1.6.0.dev20200401+cu101<br>
[pip3] torchexp==0.1.0<br>
[pip3] torchvision==0.6.0.dev20200401+cu101<br>
[conda] Could not collect</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/albanD/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/albanD">@albanD</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mruberry/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mruberry">@mruberry</a></p>