<h2 dir="auto">🐛 Bug</h2>
<p dir="auto">I'm using the nightly build: 1.0.0.dev20181123. This issue is very similar to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="377413314" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/13569" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/13569/hovercard" href="https://github.com/pytorch/pytorch/issues/13569">#13569</a> .  When I instantiate two nn.Embedding, with DataParallel and with max_norm=1.0, I get the following assert</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="RuntimeError: output_nr_ == 0 ASSERT FAILED at /pytorch/torch/csrc/autograd/variable.cpp:196, please report a bug to PyTorch."><pre class="notranslate"><code class="notranslate">RuntimeError: output_nr_ == 0 ASSERT FAILED at /pytorch/torch/csrc/autograd/variable.cpp:196, please report a bug to PyTorch.
</code></pre></div>
<p dir="auto">If I remove the self.lut_dummy, the issue disappears.</p>
<h2 dir="auto">To Reproduce</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
import os

print torch.__version__
class Moda(torch.nn.Module):
    def __init__(self):
        super(Moda, self).__init__()
        self.lut_dummy = torch.nn.Embedding(1, 1, max_norm=1.0).to(&quot;cuda&quot;)
        self.lut_a = torch.nn.Embedding(22, 256, max_norm=1.0).to(&quot;cuda&quot;)

    def forward(self, src):
        print &quot;ok1&quot;
        ebd = self.lut_a(src)
        print &quot;ok2&quot;
        return ebd

def main():
    os.environ['CUDA_VISIBLE_DEVICES'] = &quot;0,1&quot;
    model = Moda()
    model = torch.nn.DataParallel(model)
    src = torch.randint(4,(2,)).to(&quot;cuda&quot;)
    output = model(src)

if __name__ == '__main__':
    main()"><pre class="notranslate"><code class="notranslate">import torch
import os

print torch.__version__
class Moda(torch.nn.Module):
    def __init__(self):
        super(Moda, self).__init__()
        self.lut_dummy = torch.nn.Embedding(1, 1, max_norm=1.0).to("cuda")
        self.lut_a = torch.nn.Embedding(22, 256, max_norm=1.0).to("cuda")

    def forward(self, src):
        print "ok1"
        ebd = self.lut_a(src)
        print "ok2"
        return ebd

def main():
    os.environ['CUDA_VISIBLE_DEVICES'] = "0,1"
    model = Moda()
    model = torch.nn.DataParallel(model)
    src = torch.randint(4,(2,)).to("cuda")
    output = model(src)

if __name__ == '__main__':
    main()
</code></pre></div>
<h2 dir="auto">Output</h2>
<p dir="auto">1.0.0.dev20181123<br>
/home/software/LM_stash/amitoj/pytorch1.0/local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py:25: UserWarning:<br>
There is an imbalance between your GPUs. You may want to exclude GPU 0 which<br>
has less than 75% of the memory or cores of GPU 1. You can do so by setting<br>
the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES<br>
environment variable.<br>
warnings.warn(imbalance_warn.format(device_ids[min_pos], device_ids[max_pos]))<br>
ok1<br>
ok1<br>
ok2<br>
Traceback (most recent call last):<br>
File "error_train.py", line 25, in <br>
main()<br>
File "error_train.py", line 22, in main<br>
output = model(src)<br>
File "/home/software/LM_stash/amitoj/pytorch1.0/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 479, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File "/home/software/LM_stash/amitoj/pytorch1.0/local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 143, in forward<br>
outputs = self.parallel_apply(replicas, inputs, kwargs)<br>
File "/home/software/LM_stash/amitoj/pytorch1.0/local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 153, in parallel_apply<br>
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])<br>
File "/home/software/LM_stash/amitoj/pytorch1.0/local/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply<br>
raise output<br>
RuntimeError: output_nr_ == 0 ASSERT FAILED at /pytorch/torch/csrc/autograd/variable.cpp:196, please report a bug to PyTorch.</p>
<h2 dir="auto">Additional context</h2>
<p dir="auto">Observations</p>
<ol dir="auto">
<li>If I comment out the self.lut_dummy, the issue disappears. Though in my real model, I am using the <code class="notranslate">lut_dummy</code>.</li>
<li>If max_norm is removed, the issue disappears again.</li>
<li>If I set os.environ["CUDA_VISIBLE_DEVICES"] to "0" or "1", again, the code works just fine.</li>
</ol>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ssnl/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ssnl">@ssnl</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/albanD/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/albanD">@albanD</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Gradients aren't backpropagated when all three conditions are met:</p>
<ol dir="auto">
<li>Use <code class="notranslate">torch.utils.checkpoint</code>.</li>
<li>Place <code class="notranslate">nn.InstanceNorm?d</code> right before any in-place module such as <code class="notranslate">nn.ReLU(inplace=True)</code>.</li>
<li>The checkpoint returns the result of the in-place module.</li>
</ol>
<h2 dir="auto">To Reproduce</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
from torch import nn
from torch.utils.checkpoint import checkpoint

x = torch.ones(3, 2, 1, requires_grad=True)

model = nn.Sequential(
    nn.InstanceNorm1d(2),
    nn.ReLU(inplace=True),
)
y = checkpoint(model, x)

y.norm().backward()

print(x.grad)
# Output: None
# Expected: tensor([[[0.],[0.]], ...])"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">from</span> <span class="pl-s1">torch</span> <span class="pl-k">import</span> <span class="pl-s1">nn</span>
<span class="pl-k">from</span> <span class="pl-s1">torch</span>.<span class="pl-s1">utils</span>.<span class="pl-s1">checkpoint</span> <span class="pl-k">import</span> <span class="pl-s1">checkpoint</span>

<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">ones</span>(<span class="pl-c1">3</span>, <span class="pl-c1">2</span>, <span class="pl-c1">1</span>, <span class="pl-s1">requires_grad</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)

<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Sequential</span>(
    <span class="pl-s1">nn</span>.<span class="pl-v">InstanceNorm1d</span>(<span class="pl-c1">2</span>),
    <span class="pl-s1">nn</span>.<span class="pl-v">ReLU</span>(<span class="pl-s1">inplace</span><span class="pl-c1">=</span><span class="pl-c1">True</span>),
)
<span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-en">checkpoint</span>(<span class="pl-s1">model</span>, <span class="pl-s1">x</span>)

<span class="pl-s1">y</span>.<span class="pl-en">norm</span>().<span class="pl-en">backward</span>()

<span class="pl-en">print</span>(<span class="pl-s1">x</span>.<span class="pl-s1">grad</span>)
<span class="pl-c"># Output: None</span>
<span class="pl-c"># Expected: tensor([[[0.],[0.]], ...])</span></pre></div>
<p dir="auto"><code class="notranslate">x</code> the input tensor doesn't receive gradients. <code class="notranslate">x.grad</code> is still <code class="notranslate">None</code>, which is not expected.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto"><code class="notranslate">x.grad</code> should be <code class="notranslate">torch.zeros(3, 2, 1)</code>. If we remove any of the conditions, the gradients are backpropagated correctly:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# No in-place operation
model = nn.Sequential(
    nn.InstanceNorm1d(2),
    nn.ReLU(inplace=False),
)
y = checkpoint(model, x)

# No checkpoint
model = nn.Sequential(
    nn.InstanceNorm1d(2),
    nn.ReLU(inplace=True),
)
y = model(x)

# Not at the end of checkpoint
model = nn.Sequential(
    nn.InstanceNorm1d(2),
    nn.ReLU(inplace=True),
    nn.Conv1d(2, 2, 1),
)
y = checkpoint(model, x)

# Other norm
model = nn.Sequential(
    nn.BatchNorm1d(2),
    nn.ReLU(inplace=True),
)
y = checkpoint(model, x)"><pre class="notranslate"><span class="pl-c"># No in-place operation</span>
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Sequential</span>(
    <span class="pl-s1">nn</span>.<span class="pl-v">InstanceNorm1d</span>(<span class="pl-c1">2</span>),
    <span class="pl-s1">nn</span>.<span class="pl-v">ReLU</span>(<span class="pl-s1">inplace</span><span class="pl-c1">=</span><span class="pl-c1">False</span>),
)
<span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-en">checkpoint</span>(<span class="pl-s1">model</span>, <span class="pl-s1">x</span>)

<span class="pl-c"># No checkpoint</span>
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Sequential</span>(
    <span class="pl-s1">nn</span>.<span class="pl-v">InstanceNorm1d</span>(<span class="pl-c1">2</span>),
    <span class="pl-s1">nn</span>.<span class="pl-v">ReLU</span>(<span class="pl-s1">inplace</span><span class="pl-c1">=</span><span class="pl-c1">True</span>),
)
<span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">x</span>)

<span class="pl-c"># Not at the end of checkpoint</span>
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Sequential</span>(
    <span class="pl-s1">nn</span>.<span class="pl-v">InstanceNorm1d</span>(<span class="pl-c1">2</span>),
    <span class="pl-s1">nn</span>.<span class="pl-v">ReLU</span>(<span class="pl-s1">inplace</span><span class="pl-c1">=</span><span class="pl-c1">True</span>),
    <span class="pl-s1">nn</span>.<span class="pl-v">Conv1d</span>(<span class="pl-c1">2</span>, <span class="pl-c1">2</span>, <span class="pl-c1">1</span>),
)
<span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-en">checkpoint</span>(<span class="pl-s1">model</span>, <span class="pl-s1">x</span>)

<span class="pl-c"># Other norm</span>
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Sequential</span>(
    <span class="pl-s1">nn</span>.<span class="pl-v">BatchNorm1d</span>(<span class="pl-c1">2</span>),
    <span class="pl-s1">nn</span>.<span class="pl-v">ReLU</span>(<span class="pl-s1">inplace</span><span class="pl-c1">=</span><span class="pl-c1">True</span>),
)
<span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-en">checkpoint</span>(<span class="pl-s1">model</span>, <span class="pl-s1">x</span>)</pre></div>
<h2 dir="auto">Environment</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.2 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: 
GPU 0: Tesla P40
GPU 1: Tesla P40
GPU 2: Tesla P40
GPU 3: Tesla P40
GPU 4: Tesla P40
GPU 5: Tesla P40
GPU 6: Tesla P40
GPU 7: Tesla P40

Nvidia driver version: 410.104
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.2

Versions of relevant libraries:
[pip3] numpy==1.16.4
[pip3] torch==1.2.0
[pip3] torchgpipe==0.0.3
[pip3] torchvision==0.4.0
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.4                      243  
[conda] mkl-include               2019.4                      243  
[conda] mkl-service               2.0.2            py36h7b6447c_0  
[conda] mkl_fft                   1.0.12           py36ha843d7b_0  
[conda] mkl_random                1.0.2            py36hd81dba3_0  
[conda] torch                     1.2.0                    pypi_0    pypi
[conda] torchgpipe                0.0.3                     dev_0    &lt;develop&gt;
[conda] torchvision               0.4.0                    pypi_0    pypi"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.2 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: 
GPU 0: Tesla P40
GPU 1: Tesla P40
GPU 2: Tesla P40
GPU 3: Tesla P40
GPU 4: Tesla P40
GPU 5: Tesla P40
GPU 6: Tesla P40
GPU 7: Tesla P40

Nvidia driver version: 410.104
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.2

Versions of relevant libraries:
[pip3] numpy==1.16.4
[pip3] torch==1.2.0
[pip3] torchgpipe==0.0.3
[pip3] torchvision==0.4.0
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.4                      243  
[conda] mkl-include               2019.4                      243  
[conda] mkl-service               2.0.2            py36h7b6447c_0  
[conda] mkl_fft                   1.0.12           py36ha843d7b_0  
[conda] mkl_random                1.0.2            py36hd81dba3_0  
[conda] torch                     1.2.0                    pypi_0    pypi
[conda] torchgpipe                0.0.3                     dev_0    &lt;develop&gt;
[conda] torchvision               0.4.0                    pypi_0    pypi
</code></pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ssnl/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ssnl">@ssnl</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/albanD/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/albanD">@albanD</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a></p>