<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Integer overflow when doing 1x1 convolution on very large tensor</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">This is the minimal code example that produces this bug.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
import torch.nn as nn

conv = nn.Conv2d(128, 3, kernel_size=1).half().cuda()

test_tensor = torch.rand((1, 128, 4096, 4096), device='cuda', dtype=torch.float16)

with torch.no_grad():
    out_tensor = conv(test_tensor)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span> <span class="pl-k">as</span> <span class="pl-s1">nn</span>

<span class="pl-s1">conv</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Conv2d</span>(<span class="pl-c1">128</span>, <span class="pl-c1">3</span>, <span class="pl-s1">kernel_size</span><span class="pl-c1">=</span><span class="pl-c1">1</span>).<span class="pl-en">half</span>().<span class="pl-en">cuda</span>()

<span class="pl-s1">test_tensor</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">rand</span>((<span class="pl-c1">1</span>, <span class="pl-c1">128</span>, <span class="pl-c1">4096</span>, <span class="pl-c1">4096</span>), <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s">'cuda'</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">torch</span>.<span class="pl-s1">float16</span>)

<span class="pl-k">with</span> <span class="pl-s1">torch</span>.<span class="pl-en">no_grad</span>():
    <span class="pl-s1">out_tensor</span> <span class="pl-c1">=</span> <span class="pl-en">conv</span>(<span class="pl-s1">test_tensor</span>)</pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-1-28bf39f6ead7&gt; in &lt;module&gt;()
      7 
      8 with torch.no_grad():
----&gt; 9     out_tensor = conv(test_tensor)

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--&gt; 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    417 
    418     def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 419         return self._conv_forward(input, self.weight)
    420 
    421 class Conv3d(_ConvNd):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    414                             _pair(0), self.dilation, self.groups)
    415         return F.conv2d(input, weight, self.bias, self.stride,
--&gt; 416                         self.padding, self.dilation, self.groups)
    417 
    418     def forward(self, input: Tensor) -&gt; Tensor:

RuntimeError: N &gt; 0 INTERNAL ASSERT FAILED at &quot;/pytorch/aten/src/ATen/cuda/detail/KernelUtils.h&quot;:27, please report a bug to PyTorch. CUDA kernel launch blocks must be positive, but got N=-2147483648"><pre class="notranslate"><code class="notranslate">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-1-28bf39f6ead7&gt; in &lt;module&gt;()
      7 
      8 with torch.no_grad():
----&gt; 9     out_tensor = conv(test_tensor)

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--&gt; 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    417 
    418     def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 419         return self._conv_forward(input, self.weight)
    420 
    421 class Conv3d(_ConvNd):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    414                             _pair(0), self.dilation, self.groups)
    415         return F.conv2d(input, weight, self.bias, self.stride,
--&gt; 416                         self.padding, self.dilation, self.groups)
    417 
    418     def forward(self, input: Tensor) -&gt; Tensor:

RuntimeError: N &gt; 0 INTERNAL ASSERT FAILED at "/pytorch/aten/src/ATen/cuda/detail/KernelUtils.h":27, please report a bug to PyTorch. CUDA kernel launch blocks must be positive, but got N=-2147483648
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">No RunTimeError, out_tensor is float16 tensor of shape 1x3x4096x4096.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">PyTorch version: 1.6.0+cu101<br>
Is debug build: False<br>
CUDA used to build PyTorch: 10.1</p>
<p dir="auto">OS: Ubuntu 18.04.3 LTS (x86_64)<br>
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0<br>
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)<br>
CMake version: version 3.12.0</p>
<p dir="auto">Python version: 3.6 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: 10.1.243<br>
GPU models and configuration: GPU 0: Tesla T4<br>
Nvidia driver version: 418.67<br>
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip3] numpy==1.18.5<br>
[pip3] torch==1.6.0+cu101<br>
[pip3] torchsummary==1.5.1<br>
[pip3] torchtext==0.3.1<br>
[pip3] torchvision==0.7.0+cu101<br>
[conda] Could not collect</p>
<h2 dir="auto">Additional context</h2>
<p dir="auto">I ran this in Google Colab.<br>
If I change number of groups in convolution to any N &gt; 1, everything works fine.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Get following error message when running <code class="notranslate">torch.nn.Conv3d.forward()</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="N &gt; 0 INTERNAL ASSERT FAILED at /tmp/pip-req-build-ltpwwdv7/aten/src/ATen/cuda/detail/KernelUtils.h:27, please report a bug to PyTorch. CUDA kernel launch blocks must be positive, but got N=-2147483648"><pre class="notranslate"><code class="notranslate">N &gt; 0 INTERNAL ASSERT FAILED at /tmp/pip-req-build-ltpwwdv7/aten/src/ATen/cuda/detail/KernelUtils.h:27, please report a bug to PyTorch. CUDA kernel launch blocks must be positive, but got N=-2147483648
</code></pre></div>
<h2 dir="auto">To Reproduce</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch

torch.backends.cudnn.enabled=False
x = torch.rand(1, 32, 512, 512, 256).to('cuda:0')
m = torch.nn.Conv3d(32, 1, kernel_size=1, padding=0,stride=1,bias=False).to('cuda:0')
x = m(x)  # Assert!!"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>

<span class="pl-s1">torch</span>.<span class="pl-s1">backends</span>.<span class="pl-s1">cudnn</span>.<span class="pl-s1">enabled</span><span class="pl-c1">=</span><span class="pl-c1">False</span>
<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">rand</span>(<span class="pl-c1">1</span>, <span class="pl-c1">32</span>, <span class="pl-c1">512</span>, <span class="pl-c1">512</span>, <span class="pl-c1">256</span>).<span class="pl-en">to</span>(<span class="pl-s">'cuda:0'</span>)
<span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">Conv3d</span>(<span class="pl-c1">32</span>, <span class="pl-c1">1</span>, <span class="pl-s1">kernel_size</span><span class="pl-c1">=</span><span class="pl-c1">1</span>, <span class="pl-s1">padding</span><span class="pl-c1">=</span><span class="pl-c1">0</span>,<span class="pl-s1">stride</span><span class="pl-c1">=</span><span class="pl-c1">1</span>,<span class="pl-s1">bias</span><span class="pl-c1">=</span><span class="pl-c1">False</span>).<span class="pl-en">to</span>(<span class="pl-s">'cuda:0'</span>)
<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-en">m</span>(<span class="pl-s1">x</span>)  <span class="pl-c"># Assert!!</span></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py in forward(self, input)
    478                             self.dilation, self.groups)
    479         return F.conv3d(input, self.weight, self.bias, self.stride,
--&gt; 480                         self.padding, self.dilation, self.groups)
    481 
    482 

RuntimeError: N &gt; 0 INTERNAL ASSERT FAILED at /tmp/pip-req-build-ltpwwdv7/aten/src/ATen/cuda/detail/KernelUtils.h:27, please report a bug to PyTorch. CUDA kernel launch blocks must be positive, but got N=-2147483648"><pre class="notranslate"><code class="notranslate">/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py in forward(self, input)
    478                             self.dilation, self.groups)
    479         return F.conv3d(input, self.weight, self.bias, self.stride,
--&gt; 480                         self.padding, self.dilation, self.groups)
    481 
    482 

RuntimeError: N &gt; 0 INTERNAL ASSERT FAILED at /tmp/pip-req-build-ltpwwdv7/aten/src/ATen/cuda/detail/KernelUtils.h:27, please report a bug to PyTorch. CUDA kernel launch blocks must be positive, but got N=-2147483648
</code></pre></div>
<h2 dir="auto">Environment</h2>
<p dir="auto">PyTorch version: 1.4.0a0+a5b4d78<br>
Is debug build: False<br>
CUDA used to build PyTorch: 10.2</p>
<p dir="auto">OS: Ubuntu 18.04.3 LTS (x86_64)<br>
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0<br>
Clang version: Could not collect<br>
CMake version: version 3.14.0</p>
<p dir="auto">Python version: 3.6 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: 10.2.89<br>
GPU models and configuration:<br>
GPU 0: Tesla V100-SXM2-32GB</p>
<p dir="auto">Nvidia driver version: 440.33.01<br>
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a></p>