<h2 dir="auto">Issue description</h2>
<p dir="auto">nn.DataParallel uses memory on GPU 0 even if it's explicitly instructed to use 2 and 3.</p>
<h2 dir="auto">Code example</h2>
<p dir="auto">No line in this code snippet touches <code class="notranslate">cuda:0</code>.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import os
import torch
import torch.nn as nn

class MM(nn.Module):

    def __init__(self):
        super().__init__()
        self.block1 = nn.Linear(10, 20)

        # wrap block2 in DataParallel
        self.block2 = nn.Linear(20, 20)
        self.block2 = nn.DataParallel(self.block2, device_ids=[3, 2])

        self.block3 = nn.Linear(20, 20)

    def forward(self, x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.block3(x)
        return x

dev='cuda:3'

x=torch.zeros((64, 10), device=dev)
mm=MM()
mm.to(device=dev)

y=mm(x)
os.system('nvidia-smi')"><pre class="notranslate"><code class="notranslate">import os
import torch
import torch.nn as nn

class MM(nn.Module):

    def __init__(self):
        super().__init__()
        self.block1 = nn.Linear(10, 20)

        # wrap block2 in DataParallel
        self.block2 = nn.Linear(20, 20)
        self.block2 = nn.DataParallel(self.block2, device_ids=[3, 2])

        self.block3 = nn.Linear(20, 20)

    def forward(self, x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.block3(x)
        return x

dev='cuda:3'

x=torch.zeros((64, 10), device=dev)
mm=MM()
mm.to(device=dev)

y=mm(x)
os.system('nvidia-smi')
</code></pre></div>
<p dir="auto">However, nvidia-smi output indicates that GPU 0 is allocated memory</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2407      G   /usr/lib/xorg/Xorg                            14MiB |
|    0      5848      C   python                                       306MiB |
|    2      5848      C   python                                       324MiB |
|    3      5848      C   python                                       324MiB |
+-----------------------------------------------------------------------------+"><pre class="notranslate"><code class="notranslate">+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2407      G   /usr/lib/xorg/Xorg                            14MiB |
|    0      5848      C   python                                       306MiB |
|    2      5848      C   python                                       324MiB |
|    3      5848      C   python                                       324MiB |
+-----------------------------------------------------------------------------+
</code></pre></div>
<h2 dir="auto">System Info</h2>
<p dir="auto">PyTorch version: 0.4.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: 8.0.61</p>
<p dir="auto">OS: Ubuntu 16.04.2 LTS<br>
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609<br>
CMake version: version 3.5.1</p>
<p dir="auto">Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: 8.0.61<br>
GPU models and configuration:<br>
GPU 0: Tesla K80<br>
GPU 1: Tesla K80<br>
GPU 2: Tesla K80<br>
GPU 3: Tesla K80</p>
<p dir="auto">Nvidia driver version: 384.111<br>
cuDNN version: Probably one of the following:<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5.1.10<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn_static.a</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip3] msgpack-numpy (0.4.2)<br>
[pip3] numpy (1.13.3)<br>
[pip3] numpydoc (0.6.0)<br>
[pip3] torch (0.4.0)<br>
[pip3] torchvision (0.2.1)<br>
[pip3] torchx (0.0.1, /home/jimfan/code/torchx)<br>
[conda] cuda80                    1.0                           0    soumith<br>
[conda] pytorch                   0.4.0           py36_cuda8.0.61_cudnn7.1.2_1    pytorch<br>
[conda] torchvision               0.2.1                    py36_1    pytorch<br>
[conda] torchx                    0.0.1                     </p>
<h2 dir="auto">Issue description</h2>
<p dir="auto">on a multi GPU system, if a GPU != 0 is used, pytorch will still allocate some memory on the GPU 0 - see the nvidia-smi screenshot below.</p>
<h2 dir="auto">Code example</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.optim as optim........

torch.backends.cudnn.benchmark = True

class TestNet(nn.Module):
    def __init__(self):
        super(TestNet, self).__init__()
        self.net1 = nn.Linear(100, 200)
        self.net2 = nn.Linear(200, 1)
        self.bn = nn.BatchNorm1d(200)
        self.sigmoid = nn.Sigmoid()
        self.ReLU = nn.ReLU(inplace=False)
        self.drop = nn.Dropout(0.5)
...............
    def forward(self, V):
        return self.sigmoid(self.net2(self.drop(self.ReLU(self.bn(self.net1(V)))))).squeeze().


GPU = 1
net = TestNet()
criterion = nn.BCELoss()
net.cuda(GPU)
criterion.cuda(GPU)
V = Variable(torch.randn(100, 100)).cuda(GPU)
label = Variable(torch.randn(100)).cuda(GPU)
optim_betas = (0.9, 0.999)
optimizer = optim.Adam(filter(lambda p: p.requires_grad, net.parameters()), lr=0.01, betas=optim_betas)

net.train()
for i in range(0,1000000):
    net.zero_grad()
    r = net(V)
    err = criterion(r, label)
    err.backward()...
"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span> <span class="pl-k">as</span> <span class="pl-s1">nn</span>
<span class="pl-k">from</span> <span class="pl-s1">torch</span>.<span class="pl-s1">autograd</span> <span class="pl-k">import</span> <span class="pl-v">Variable</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>.<span class="pl-s1">optim</span> <span class="pl-k">as</span> <span class="pl-s1">optim</span>........

<span class="pl-s1">torch</span>.<span class="pl-s1">backends</span>.<span class="pl-s1">cudnn</span>.<span class="pl-s1">benchmark</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>

<span class="pl-k">class</span> <span class="pl-v">TestNet</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>):
        <span class="pl-en">super</span>(<span class="pl-v">TestNet</span>, <span class="pl-s1">self</span>).<span class="pl-en">__init__</span>()
        <span class="pl-s1">self</span>.<span class="pl-s1">net1</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Linear</span>(<span class="pl-c1">100</span>, <span class="pl-c1">200</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">net2</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Linear</span>(<span class="pl-c1">200</span>, <span class="pl-c1">1</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">bn</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">BatchNorm1d</span>(<span class="pl-c1">200</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">sigmoid</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Sigmoid</span>()
        <span class="pl-s1">self</span>.<span class="pl-v">ReLU</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">ReLU</span>(<span class="pl-s1">inplace</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">drop</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Dropout</span>(<span class="pl-c1">0.5</span>)
...............
    <span class="pl-s1">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-v">V</span>):
        <span class="pl-k">return</span> <span class="pl-s1">self</span>.<span class="pl-en">sigmoid</span>(<span class="pl-s1">self</span>.<span class="pl-en">net2</span>(<span class="pl-s1">self</span>.<span class="pl-en">drop</span>(<span class="pl-s1">self</span>.<span class="pl-v">ReLU</span>(<span class="pl-s1">self</span>.<span class="pl-en">bn</span>(<span class="pl-s1">self</span>.<span class="pl-en">net1</span>(<span class="pl-v">V</span>)))))).<span class="pl-en">squeeze</span>().


<span class="pl-v">GPU</span> <span class="pl-c1">=</span> <span class="pl-c1">1</span>
<span class="pl-s1">net</span> <span class="pl-c1">=</span> <span class="pl-v">TestNet</span>()
<span class="pl-s1">criterion</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">BCELoss</span>()
<span class="pl-s1">net</span>.<span class="pl-en">cuda</span>(<span class="pl-v">GPU</span>)
<span class="pl-s1">criterion</span>.<span class="pl-en">cuda</span>(<span class="pl-v">GPU</span>)
<span class="pl-v">V</span> <span class="pl-c1">=</span> <span class="pl-v">Variable</span>(<span class="pl-s1">torch</span>.<span class="pl-en">randn</span>(<span class="pl-c1">100</span>, <span class="pl-c1">100</span>)).<span class="pl-en">cuda</span>(<span class="pl-v">GPU</span>)
<span class="pl-s1">label</span> <span class="pl-c1">=</span> <span class="pl-v">Variable</span>(<span class="pl-s1">torch</span>.<span class="pl-en">randn</span>(<span class="pl-c1">100</span>)).<span class="pl-en">cuda</span>(<span class="pl-v">GPU</span>)
<span class="pl-s1">optim_betas</span> <span class="pl-c1">=</span> (<span class="pl-c1">0.9</span>, <span class="pl-c1">0.999</span>)
<span class="pl-s1">optimizer</span> <span class="pl-c1">=</span> <span class="pl-s1">optim</span>.<span class="pl-v">Adam</span>(<span class="pl-en">filter</span>(<span class="pl-k">lambda</span> <span class="pl-s1">p</span>: <span class="pl-s1">p</span>.<span class="pl-s1">requires_grad</span>, <span class="pl-s1">net</span>.<span class="pl-en">parameters</span>()), <span class="pl-s1">lr</span><span class="pl-c1">=</span><span class="pl-c1">0.01</span>, <span class="pl-s1">betas</span><span class="pl-c1">=</span><span class="pl-s1">optim_betas</span>)

<span class="pl-s1">net</span>.<span class="pl-en">train</span>()
<span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">0</span>,<span class="pl-c1">1000000</span>):
    <span class="pl-s1">net</span>.<span class="pl-en">zero_grad</span>()
    <span class="pl-s1">r</span> <span class="pl-c1">=</span> <span class="pl-en">net</span>(<span class="pl-v">V</span>)
    <span class="pl-s1">err</span> <span class="pl-c1">=</span> <span class="pl-en">criterion</span>(<span class="pl-s1">r</span>, <span class="pl-s1">label</span>)
    <span class="pl-s1">err</span>.<span class="pl-en">backward</span>()...</pre></div>
<p dir="auto">tester@NEWTITAN:~$ nvidia-smi<br>
Sat Apr 28 23:45:03 2018<br>
+-----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |<br>
|-------------------------------+----------------------+----------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|===============================+======================+======================|<br>
|   0  TITAN X (Pascal)    Off  | 00000000:05:00.0 Off |                  N/A |<br>
| 23%   33C    P2    57W / 250W |    401MiB / 12189MiB |      0%      Default |<br>
+-------------------------------+----------------------+----------------------+<br>
|   1  GeForce GTX 108...  Off  | 00000000:06:00.0 Off |                  N/A |<br>
| 29%   33C    P2    58W / 250W |    549MiB / 11172MiB |      6%      Default |<br>
+-------------------------------+----------------------+----------------------+<br>
|   2  GeForce GTX 108...  Off  | 00000000:09:00.0 Off |                  N/A |<br>
| 29%   36C    P8    10W / 250W |     10MiB / 11172MiB |      0%      Default |<br>
+-------------------------------+----------------------+----------------------+</p>
<p dir="auto">+-----------------------------------------------------------------------------+<br>
| Processes:                                                       GPU Memory |<br>
|  GPU       PID   Type   Process name                             Usage      |<br>
|=============================================================================|<br>
|    0      1262      C   python                                       391MiB |<br>
|    1      1262      C   python                                       539MiB |<br>
+-----------------------------------------------------------------------------+</p>