<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">We have a number of machines with Threadripper CPUs, and 2 NVIDIA GPUs, some have 1070ti cards some 1080 some 1080ti and one with titanXp, they all displayed this behavior, when switching to using data parallel, training would fail, i.e. accuracy would not go up. We first saw this in our code base, but it also happens on the imagnet example from the pytorch examples repo</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>run the imagnet example for the examples repo in pytorch with dataparallel</li>
</ol>
<p dir="auto">these error messages were found in the dmesg log:</p>
<p dir="auto">[1118468.873266] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000ea13a000 flags=0x0020]<br>
[1118468.942145] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000ea139068 flags=0x0020]<br>
[1118468.942189] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d0000040 flags=0x0020]<br>
[1118468.942227] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d00007c0 flags=0x0020]<br>
[1118468.942265] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d0001040 flags=0x0020]<br>
[1118468.942303] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d0000f40 flags=0x0020]<br>
[1118468.942340] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d00016c0 flags=0x0020]<br>
[1118468.942377] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d0002040 flags=0x0020]<br>
[1118468.942414] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d0001e40 flags=0x0020]<br>
[1118468.942452] nvidia 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x00000000d00025c0 flags=0x0020]<br>
[1118468.942489] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0003040 flags=0x0020]<br>
[1118468.942525] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0002d40 flags=0x0020]<br>
[1118468.942560] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d00034c0 flags=0x0020]<br>
[1118468.942596] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0004040 flags=0x0020]<br>
[1118468.942632] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0003c40 flags=0x0020]<br>
[1118468.942667] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d00043c0 flags=0x0020]<br>
[1118468.942703] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0005040 flags=0x0020]<br>
[1118468.942739] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d0004b40 flags=0x0020]<br>
[1118468.942774] AMD-Vi: Event logged [IO_PAGE_FAULT device=0a:00.0 domain=0x000f address=0x00000000d00052c0 flags=0x0020]</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">accuracy would not pick up in most cases, it never picked up for the validation set. We managed to work around this problem by turning off IOMMU in the bios.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">PyTorch version: 0.4.1<br>
Is debug build: No<br>
CUDA used to build PyTorch: 9.0.176</p>
<p dir="auto">OS: Ubuntu 16.04.5 LTS<br>
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609<br>
CMake version: version 3.5.1</p>
<p dir="auto">Python version: 2.7<br>
Is CUDA available: Yes<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: GeForce GTX 1080 Ti<br>
GPU 1: GeForce GTX 1080 Ti</p>
<p dir="auto">Nvidia driver version: 384.130<br>
cuDNN version: Probably one of the following:<br>
/usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudnn.so.7.0.5<br>
/usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudnn_static.a</p>
<p dir="auto">I'm having trouble getting multi-gpu via <code class="notranslate">DataParallel</code> across two Tesla K80 GPUs. The code I'm using is a modification of the MNIST example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable
from data_parallel import DataParallel

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=256, shuffle=True, num_workers=2, pin_memory=True)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2(x), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x)

model = DataParallel(Net())
model.cuda()

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
criterion = nn.NLLLoss().cuda()

model.train()
for batch_idx, (data, target) in enumerate(train_loader):
    input_var = Variable(data.cuda())
    target_var = Variable(target.cuda())

    print('Getting model output')
    output = model(input_var)
    print('Got model output')

    loss = criterion(output, target_var)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print('Finished')"><pre class="notranslate"><code class="notranslate">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable
from data_parallel import DataParallel

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=256, shuffle=True, num_workers=2, pin_memory=True)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2(x), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x)

model = DataParallel(Net())
model.cuda()

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
criterion = nn.NLLLoss().cuda()

model.train()
for batch_idx, (data, target) in enumerate(train_loader):
    input_var = Variable(data.cuda())
    target_var = Variable(target.cuda())

    print('Getting model output')
    output = model(input_var)
    print('Got model output')

    loss = criterion(output, target_var)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print('Finished')
</code></pre></div>
<p dir="auto">This doesn't throw an error, but hangs after it prints "Getting model output" and never returns. I traced this down to the <code class="notranslate">parallel_apply</code> spawning threads that then never finish. The line that hangs is <a href="https://github.com/pytorch/pytorch/blob/e50a1f19b3dc735f0710929b97b0af384aafe09b/torch/nn/parallel/parallel_apply.py#L25">here</a> where the threads are spawned using both GPU 0 and GPU 1, but never finish.</p>
<p dir="auto">This is only a problem when <code class="notranslate">CUDA_VISIBLE_DEVICES=0,1</code> as both GPU0 and GPU1 work perfectly well individually.</p>
<p dir="auto">Before running this, <code class="notranslate">nvidia-smi</code> shows</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="+------------------------------------------------------+
| NVIDIA-SMI 352.68     Driver Version: 352.68         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:06:00.0     Off |                    0 |
| N/A   40C    P0    57W / 149W |     55MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:07:00.0     Off |                    0 |
| N/A   35C    P0    76W / 149W |     55MiB / 11519MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+"><pre class="notranslate"><code class="notranslate">+------------------------------------------------------+
| NVIDIA-SMI 352.68     Driver Version: 352.68         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:06:00.0     Off |                    0 |
| N/A   40C    P0    57W / 149W |     55MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:07:00.0     Off |                    0 |
| N/A   35C    P0    76W / 149W |     55MiB / 11519MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre></div>
<p dir="auto">after running (while it hangs), <code class="notranslate">nvidia-smi</code> gives</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="+------------------------------------------------------+
| NVIDIA-SMI 352.68     Driver Version: 352.68         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:06:00.0     Off |                    0 |
| N/A   42C    P0    69W / 149W |    251MiB / 11519MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:07:00.0     Off |                    0 |
| N/A   36C    P0    90W / 149W |    249MiB / 11519MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      4785    C   python                                         194MiB |
|    1      4785    C   python                                         192MiB |
+-----------------------------------------------------------------------------+"><pre class="notranslate"><code class="notranslate">+------------------------------------------------------+
| NVIDIA-SMI 352.68     Driver Version: 352.68         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:06:00.0     Off |                    0 |
| N/A   42C    P0    69W / 149W |    251MiB / 11519MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:07:00.0     Off |                    0 |
| N/A   36C    P0    90W / 149W |    249MiB / 11519MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      4785    C   python                                         194MiB |
|    1      4785    C   python                                         192MiB |
+-----------------------------------------------------------------------------+
</code></pre></div>
<p dir="auto">and <code class="notranslate">top</code> shows the main python process and the two python subprocesses. Wondering if this could be something similar to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="202557169" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/554" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/554/hovercard" href="https://github.com/pytorch/pytorch/issues/554">#554</a>.</p>
<p dir="auto">Using <a href="https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py">this</a> TensorFlow example, I get linear speedup using multiple GPUs as I change <code class="notranslate">CUDA_VISIBLE_DEVICES</code> so multiple K80s should certainly be viable.</p>