<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Multiplication of sparse with dense tensor on GPU breaks</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">I have the following implementation of page rank on pytorch using sparse tensors:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="i = torch.LongTensor(idx)
values = torch.FloatTensor([1] * len(idx))
M = torch.sparse.FloatTensor(i.t(), values, torch.Size([4847571, 4847571]))
N = M.shape[1]
v = torch.rand(N, 1).float()
values = torch.FloatTensor([(1 - d)/N] * len(indices))
temp = torch.sparse.FloatTensor(i.t(), values, torch.Size([4847571,
                                    4847571]))
if torch.cuda.is_available():
     v = v.cuda()
     M = M.cuda()
     temp = temp.cuda()

v = v / torch.norm(v, 1)
M_hat = self.d * M + temp
for i in range(num_iter):
     v = torch.mm(M_hat, v)"><pre class="notranslate"><code class="notranslate">i = torch.LongTensor(idx)
values = torch.FloatTensor([1] * len(idx))
M = torch.sparse.FloatTensor(i.t(), values, torch.Size([4847571, 4847571]))
N = M.shape[1]
v = torch.rand(N, 1).float()
values = torch.FloatTensor([(1 - d)/N] * len(indices))
temp = torch.sparse.FloatTensor(i.t(), values, torch.Size([4847571,
                                    4847571]))
if torch.cuda.is_available():
     v = v.cuda()
     M = M.cuda()
     temp = temp.cuda()

v = v / torch.norm(v, 1)
M_hat = self.d * M + temp
for i in range(num_iter):
     v = torch.mm(M_hat, v)
</code></pre></div>
<p dir="auto">The indices are gained by parsing <a href="https://snap.stanford.edu/data/soc-LiveJournal1.html" rel="nofollow">this</a> graph.</p>
<p dir="auto">On CPU everything runs fine. On GPU however I am receiving the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    v = torch.mm(M_hat, v)
RuntimeError: sub_iter.strides(0)[0] == 0 INTERNAL ASSERT FAILED at /pytorch/aten/src/ATen/native/cuda/Reduce.cuh:706, please report a bug to PyTorch."><pre class="notranslate"><code class="notranslate">    v = torch.mm(M_hat, v)
RuntimeError: sub_iter.strides(0)[0] == 0 INTERNAL ASSERT FAILED at /pytorch/aten/src/ATen/native/cuda/Reduce.cuh:706, please report a bug to PyTorch.
</code></pre></div>
<h2 dir="auto">Environment</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.2
OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2
Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla P100-PCIE-16GB
Nvidia driver version: 440.82
cuDNN version: Could not collect
Versions of relevant libraries:
[pip3] numpy==1.18.3
[pip3] torch==1.5.0"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.2
OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2
Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla P100-PCIE-16GB
Nvidia driver version: 440.82
cuDNN version: Could not collect
Versions of relevant libraries:
[pip3] numpy==1.18.3
[pip3] torch==1.5.0
</code></pre></div>
<p dir="auto">If it helps more my machine is the first one from <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/ncv2-series" rel="nofollow">here</a>.</p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>

<p dir="auto">Calling <code class="notranslate">.sum()</code> on large cuda tensors throws the error.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
t = torch.zeros(5, 14400, 14400).cuda()
!nvidia-smi
t.sum(dim=0)"><pre class="notranslate"><code class="notranslate">import torch
t = torch.zeros(5, 14400, 14400).cuda()
!nvidia-smi
t.sum(dim=0)
</code></pre></div>
<p dir="auto">shows the following nvidia info</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   40C    P0    32W / 250W |   4663MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+"><pre class="notranslate"><code class="notranslate">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   40C    P0    32W / 250W |   4663MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
</code></pre></div>
<p dir="auto">and the traceback is</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-1-bbe24183337c&gt; in &lt;module&gt;()
      2 t = torch.zeros(5, 14400, 14400).cuda()
      3 get_ipython().system('nvidia-smi')
----&gt; 4 t.sum(dim=0)

RuntimeError: sub_iter.strides(0)[0] == 0 INTERNAL ASSERT FAILED at /pytorch/aten/src/ATen/native/cuda/Reduce.cuh:706, please report a bug to PyTorch. "><pre class="notranslate"><code class="notranslate">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-1-bbe24183337c&gt; in &lt;module&gt;()
      2 t = torch.zeros(5, 14400, 14400).cuda()
      3 get_ipython().system('nvidia-smi')
----&gt; 4 t.sum(dim=0)

RuntimeError: sub_iter.strides(0)[0] == 0 INTERNAL ASSERT FAILED at /pytorch/aten/src/ATen/native/cuda/Reduce.cuh:706, please report a bug to PyTorch. 
</code></pre></div>
<p dir="auto">This behavior is reproduced with any <code class="notranslate">(n, 14400, 14400)</code> tensor for <code class="notranslate">n&gt;=5</code>.</p>
<p dir="auto"><strong>Note</strong>. I can easily create a (6, 14400, 14400) tensor - memory usage, in this case, is <code class="notranslate">5455MiB / 16280MiB</code>.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Returns a (14400, 14400) tensor</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">PyTorch version: 1.5.0+cu101<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.1</p>
<p dir="auto">OS: Ubuntu 18.04.3 LTS<br>
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0<br>
CMake version: version 3.12.0</p>
<p dir="auto">Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: 10.1.243<br>
GPU models and configuration: GPU 0: Tesla P100-PCIE-16GB<br>
Nvidia driver version: 418.67<br>
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip3] numpy==1.18.3<br>
[pip3] pytorch-mighty==0.1.0<br>
[pip3] torch==1.5.0+cu101<br>
[pip3] torchfile==0.1.0<br>
[pip3] torchsummary==1.5.1<br>
[pip3] torchtext==0.3.1<br>
[pip3] torchvision==0.6.0+cu101<br>
[conda] Could not collect</p>
<h2 dir="auto">Additional context</h2>
<p dir="auto">Google colab, GPU instance.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/bdhirsh/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/bdhirsh">@bdhirsh</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jbschlosser/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jbschlosser">@jbschlosser</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/anjali411/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/anjali411">@anjali411</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/heitorschueroff/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/heitorschueroff">@heitorschueroff</a></p>