<p dir="auto">(Reported by internal user)</p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">I'm observing some weird behaviors with the sparse mode of nn.Embedding. For instance, with the following code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="m = nn.Embedding(10, 3, sparse=True)
m.zero_grad()
m(torch.LongTensor([7, 1, 3])).sum().backward()
print(m.weight.grad)"><pre class="notranslate"><span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Embedding</span>(<span class="pl-c1">10</span>, <span class="pl-c1">3</span>, <span class="pl-s1">sparse</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">m</span>.<span class="pl-en">zero_grad</span>()
<span class="pl-en">m</span>(<span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>([<span class="pl-c1">7</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>])).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()
<span class="pl-en">print</span>(<span class="pl-s1">m</span>.<span class="pl-s1">weight</span>.<span class="pl-s1">grad</span>)</pre></div>
<p dir="auto">I'll get</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="torch.sparse.FloatTensor of size (10,3) with indices:
tensor([[7, 1, 3]])
and values:
tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]])"><pre class="notranslate"><span class="pl-s1">torch</span>.<span class="pl-s1">sparse</span>.<span class="pl-v">FloatTensor</span> <span class="pl-s1">of</span> <span class="pl-s1">size</span> (<span class="pl-c1">10</span>,<span class="pl-c1">3</span>) <span class="pl-k">with</span> <span class="pl-s1">indices</span>:
<span class="pl-en">tensor</span>([[<span class="pl-c1">7</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>]])
<span class="pl-s1">and</span> <span class="pl-s1">values</span>:
<span class="pl-en">tensor</span>([[<span class="pl-c1">1.</span>, <span class="pl-c1">1.</span>, <span class="pl-c1">1.</span>],
        [<span class="pl-c1">1.</span>, <span class="pl-c1">1.</span>, <span class="pl-c1">1.</span>],
        [<span class="pl-c1">1.</span>, <span class="pl-c1">1.</span>, <span class="pl-c1">1.</span>]])</pre></div>
<p dir="auto">as expected. But if I repeat the fwd/bwd line twice:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="m = nn.Embedding(10, 3, sparse=True)
m.zero_grad()
m(torch.LongTensor([7, 1, 3])).sum().backward()
m(torch.LongTensor([7, 1, 3])).sum().backward()
print(m.weight.grad)"><pre class="notranslate"><span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Embedding</span>(<span class="pl-c1">10</span>, <span class="pl-c1">3</span>, <span class="pl-s1">sparse</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">m</span>.<span class="pl-en">zero_grad</span>()
<span class="pl-en">m</span>(<span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>([<span class="pl-c1">7</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>])).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()
<span class="pl-en">m</span>(<span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>([<span class="pl-c1">7</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>])).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()
<span class="pl-en">print</span>(<span class="pl-s1">m</span>.<span class="pl-s1">weight</span>.<span class="pl-s1">grad</span>)</pre></div>
<p dir="auto">I get random outputs, such as:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="torch.sparse.FloatTensor of size (10,3) with indices:
tensor([[7, 1, 3]])
and values:
tensor([[  2.0000,   1.0000,   1.0000],
        [  1.0000,   1.0000,   1.0000],
        [  1.0000,   1.0000, 793.1436]])"><pre class="notranslate"><span class="pl-s1">torch</span>.<span class="pl-s1">sparse</span>.<span class="pl-v">FloatTensor</span> <span class="pl-s1">of</span> <span class="pl-s1">size</span> (<span class="pl-c1">10</span>,<span class="pl-c1">3</span>) <span class="pl-k">with</span> <span class="pl-s1">indices</span>:
<span class="pl-en">tensor</span>([[<span class="pl-c1">7</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>]])
<span class="pl-s1">and</span> <span class="pl-s1">values</span>:
<span class="pl-en">tensor</span>([[  <span class="pl-c1">2.0000</span>,   <span class="pl-c1">1.0000</span>,   <span class="pl-c1">1.0000</span>],
        [  <span class="pl-c1">1.0000</span>,   <span class="pl-c1">1.0000</span>,   <span class="pl-c1">1.0000</span>],
        [  <span class="pl-c1">1.0000</span>,   <span class="pl-c1">1.0000</span>, <span class="pl-c1">793.1436</span>]])</pre></div>
<p dir="auto">I guess this is not expected?<br>
Also I noticed that if the input is different, it behaves very differently:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="m = nn.Embedding(10, 3, sparse=True)
m.zero_grad()
m(torch.LongTensor([7, 1, 3])).sum().backward()
m(torch.LongTensor([8, 1, 3])).sum().backward()
print(m.weight.grad)"><pre class="notranslate"><span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Embedding</span>(<span class="pl-c1">10</span>, <span class="pl-c1">3</span>, <span class="pl-s1">sparse</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">m</span>.<span class="pl-en">zero_grad</span>()
<span class="pl-en">m</span>(<span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>([<span class="pl-c1">7</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>])).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()
<span class="pl-en">m</span>(<span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>([<span class="pl-c1">8</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>])).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()
<span class="pl-en">print</span>(<span class="pl-s1">m</span>.<span class="pl-s1">weight</span>.<span class="pl-s1">grad</span>)</pre></div>
<p dir="auto">Returns:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="torch.sparse.FloatTensor of size (10,3) with indices:
tensor([[7, 1, 3, 8, 1, 3]])
and values:
tensor([[1.0000, 1.0000, 1.0000],
        [1.0000, 1.0000, 1.0000],
        [1.0000, 1.0000, 1.0000],
        [1.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000]])"><pre class="notranslate"><span class="pl-s1">torch</span>.<span class="pl-s1">sparse</span>.<span class="pl-v">FloatTensor</span> <span class="pl-s1">of</span> <span class="pl-s1">size</span> (<span class="pl-c1">10</span>,<span class="pl-c1">3</span>) <span class="pl-k">with</span> <span class="pl-s1">indices</span>:
<span class="pl-en">tensor</span>([[<span class="pl-c1">7</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>, <span class="pl-c1">8</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>]])
<span class="pl-s1">and</span> <span class="pl-s1">values</span>:
<span class="pl-en">tensor</span>([[<span class="pl-c1">1.0000</span>, <span class="pl-c1">1.0000</span>, <span class="pl-c1">1.0000</span>],
        [<span class="pl-c1">1.0000</span>, <span class="pl-c1">1.0000</span>, <span class="pl-c1">1.0000</span>],
        [<span class="pl-c1">1.0000</span>, <span class="pl-c1">1.0000</span>, <span class="pl-c1">1.0000</span>],
        [<span class="pl-c1">1.0000</span>, <span class="pl-c1">0.0000</span>, <span class="pl-c1">0.0000</span>],
        [<span class="pl-c1">0.0000</span>, <span class="pl-c1">0.0000</span>, <span class="pl-c1">0.0000</span>],
        [<span class="pl-c1">0.0000</span>, <span class="pl-c1">0.0000</span>, <span class="pl-c1">0.0000</span>]])</pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">It should behave the same as the dense version of nn.Embedding (aka. <code class="notranslate">nn.Embedding(10, 3, sparse=False)</code>.</p>
<h2 dir="auto">Additional Context</h2>
<p dir="auto">This is reproducible on nightly builds as of 3/12/2019.</p>
<p dir="auto">I'm observing some weird behaviors with the sparse mode of nn.Embedding (both in PyTorch 0.4 and 1.0). For instance, with the following code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="m = nn.Embedding(10, 3, sparse=True)
m.zero_grad()
m(torch.LongTensor([7, 1, 3])).sum().backward()
print(m.weight.grad)"><pre class="notranslate"><span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Embedding</span>(<span class="pl-c1">10</span>, <span class="pl-c1">3</span>, <span class="pl-s1">sparse</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">m</span>.<span class="pl-en">zero_grad</span>()
<span class="pl-en">m</span>(<span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>([<span class="pl-c1">7</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>])).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()
<span class="pl-en">print</span>(<span class="pl-s1">m</span>.<span class="pl-s1">weight</span>.<span class="pl-s1">grad</span>)</pre></div>
<p dir="auto">I'll get</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="torch.sparse.FloatTensor of size (10,3) with indices:
tensor([[7, 1, 3]])
and values:
tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]])"><pre class="notranslate"><code class="notranslate">torch.sparse.FloatTensor of size (10,3) with indices:
tensor([[7, 1, 3]])
and values:
tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]])
</code></pre></div>
<p dir="auto">as expected. But if I repeat the fwd/bwd line twice:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="m = nn.Embedding(10, 3, sparse=True)
m.zero_grad()
m(torch.LongTensor([7, 1, 3])).sum().backward()
m(torch.LongTensor([7, 1, 3])).sum().backward()
print(m.weight.grad)"><pre class="notranslate"><span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Embedding</span>(<span class="pl-c1">10</span>, <span class="pl-c1">3</span>, <span class="pl-s1">sparse</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">m</span>.<span class="pl-en">zero_grad</span>()
<span class="pl-en">m</span>(<span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>([<span class="pl-c1">7</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>])).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()
<span class="pl-en">m</span>(<span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>([<span class="pl-c1">7</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>])).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()
<span class="pl-en">print</span>(<span class="pl-s1">m</span>.<span class="pl-s1">weight</span>.<span class="pl-s1">grad</span>)</pre></div>
<p dir="auto">I get random outputs, such as:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="torch.sparse.FloatTensor of size (10,3) with indices:
tensor([[7, 1, 3]])
and values:
tensor([[  2.0000,   1.0000,   1.0000],
        [  1.0000,   1.0000,   1.0000],
        [  1.0000,   1.0000, 793.1436]])"><pre class="notranslate"><code class="notranslate">torch.sparse.FloatTensor of size (10,3) with indices:
tensor([[7, 1, 3]])
and values:
tensor([[  2.0000,   1.0000,   1.0000],
        [  1.0000,   1.0000,   1.0000],
        [  1.0000,   1.0000, 793.1436]])
</code></pre></div>
<p dir="auto">I guess this is not expected?</p>
<p dir="auto">Also I noticed that if the input is different, it behaves very differently:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="m = nn.Embedding(10, 3, sparse=True)
m.zero_grad()
m(torch.LongTensor([7, 1, 3])).sum().backward()
m(torch.LongTensor([8, 1, 3])).sum().backward()
print(m.weight.grad)"><pre class="notranslate"><span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Embedding</span>(<span class="pl-c1">10</span>, <span class="pl-c1">3</span>, <span class="pl-s1">sparse</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">m</span>.<span class="pl-en">zero_grad</span>()
<span class="pl-en">m</span>(<span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>([<span class="pl-c1">7</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>])).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()
<span class="pl-en">m</span>(<span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>([<span class="pl-c1">8</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>])).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()
<span class="pl-en">print</span>(<span class="pl-s1">m</span>.<span class="pl-s1">weight</span>.<span class="pl-s1">grad</span>)</pre></div>
<p dir="auto">Returns:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="torch.sparse.FloatTensor of size (10,3) with indices:
tensor([[7, 1, 3, 8, 1, 3]])
and values:
tensor([[1.0000, 1.0000, 1.0000],
        [1.0000, 1.0000, 1.0000],
        [1.0000, 1.0000, 1.0000],
        [1.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000]])"><pre class="notranslate"><code class="notranslate">torch.sparse.FloatTensor of size (10,3) with indices:
tensor([[7, 1, 3, 8, 1, 3]])
and values:
tensor([[1.0000, 1.0000, 1.0000],
        [1.0000, 1.0000, 1.0000],
        [1.0000, 1.0000, 1.0000],
        [1.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000]])
</code></pre></div>