<h2 dir="auto">Issue description</h2>
<p dir="auto">ResNet-50 (resolution of 128, batch size 8) crashes on PyTorch 0.4.1, CUDA 9.2 with fp16. It works with CUDA 9.0 and fp32.</p>
<h2 dir="auto">Code example</h2>
<p dir="auto"><a href="https://github.com/ddkang/fai-imagenet/tree/imagenet/imagenet_nv">https://github.com/ddkang/fai-imagenet/tree/imagenet/imagenet_nv</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Starting program: /home/daniel_d_kang/anaconda3/envs/dawnbench/bin/python main.py -a resnet50 --lr 0.40 --epochs 45 --small -j 8 --fp16 -b 8 --loss-scale 512 /mnt/disks/dawnbench/imagenet
[Thread debugging using libthread_db enabled]
Using host libthread_db library &quot;/lib/x86_64-linux-gnu/libthread_db.so.1&quot;.
~~epoch hours   top1Accuracy

[New Thread 0x7fffaf723700 (LWP 12826)]
[New Thread 0x7fffaef22700 (LWP 12827)]
0
[New Thread 0x7fff44a3f700 (LWP 13143)]
[New Thread 0x7fff426fe700 (LWP 13155)]
[New Thread 0x7fff4173f700 (LWP 13156)]
[New Thread 0x7fff40f3e700 (LWP 13157)]
[New Thread 0x7fff4073d700 (LWP 13158)]
[New Thread 0x7fff3ff3c700 (LWP 13159)]
[New Thread 0x7fff3f73b700 (LWP 13162)]
[New Thread 0x7fff3e77f700 (LWP 13164)]
[New Thread 0x7fff3df7e700 (LWP 13165)]
zero_grad
tensor(3648., device='cuda:0', dtype=torch.float16, grad_fn=&lt;MulBackward&gt;)
torch.Size([])
torch.cuda.HalfTensor
[New Thread 0x7fff3d77d700 (LWP 13216)]
[New Thread 0x7fff3cf7c700 (LWP 13217)]

Thread 14 &quot;python&quot; received signal SIGFPE, Arithmetic exception.
[Switching to Thread 0x7fff3cf7c700 (LWP 13217)]
0x00007fffc3151984 in cudnn::gemm::conv2d(cudnnContext*, void const*, cudnnTensor4dStruct*, void const*, cudnnFilter4dStruct*, void const*, cudnnConvolutionStruct*, cudnnConvWorkingStruct const*, void*, unsigned long, void const*, cudnnTensor4dStruct*, void*, cudnn::gemm::Conv2dType_t, cudnn::gemm::Conv2dConfig&amp;, bool, void const*, cudnnActivationStruct*, void*) () from /home/daniel_d_kang/anaconda3/envs/dawnbench/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so"><pre lang="(gdb)" class="notranslate"><code class="notranslate">Starting program: /home/daniel_d_kang/anaconda3/envs/dawnbench/bin/python main.py -a resnet50 --lr 0.40 --epochs 45 --small -j 8 --fp16 -b 8 --loss-scale 512 /mnt/disks/dawnbench/imagenet
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
~~epoch hours   top1Accuracy

[New Thread 0x7fffaf723700 (LWP 12826)]
[New Thread 0x7fffaef22700 (LWP 12827)]
0
[New Thread 0x7fff44a3f700 (LWP 13143)]
[New Thread 0x7fff426fe700 (LWP 13155)]
[New Thread 0x7fff4173f700 (LWP 13156)]
[New Thread 0x7fff40f3e700 (LWP 13157)]
[New Thread 0x7fff4073d700 (LWP 13158)]
[New Thread 0x7fff3ff3c700 (LWP 13159)]
[New Thread 0x7fff3f73b700 (LWP 13162)]
[New Thread 0x7fff3e77f700 (LWP 13164)]
[New Thread 0x7fff3df7e700 (LWP 13165)]
zero_grad
tensor(3648., device='cuda:0', dtype=torch.float16, grad_fn=&lt;MulBackward&gt;)
torch.Size([])
torch.cuda.HalfTensor
[New Thread 0x7fff3d77d700 (LWP 13216)]
[New Thread 0x7fff3cf7c700 (LWP 13217)]

Thread 14 "python" received signal SIGFPE, Arithmetic exception.
[Switching to Thread 0x7fff3cf7c700 (LWP 13217)]
0x00007fffc3151984 in cudnn::gemm::conv2d(cudnnContext*, void const*, cudnnTensor4dStruct*, void const*, cudnnFilter4dStruct*, void const*, cudnnConvolutionStruct*, cudnnConvWorkingStruct const*, void*, unsigned long, void const*, cudnnTensor4dStruct*, void*, cudnn::gemm::Conv2dType_t, cudnn::gemm::Conv2dConfig&amp;, bool, void const*, cudnnActivationStruct*, void*) () from /home/daniel_d_kang/anaconda3/envs/dawnbench/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so
</code></pre></div>
<h2 dir="auto">System Info</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.2.148

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla V100-SXM2-16GB
Nvidia driver version: 396.44
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_static_v7.a

Versions of relevant libraries:
[pip] numpy (1.14.3)
[pip] numpydoc (0.8.0)
[pip] torch (0.4.1)
[pip] torchvision (0.2.1)
[conda] cuda92                    1.0                           0    pytorch
[conda] pytorch                   0.4.1           py36_cuda9.2.148_cudnn7.1.4_1  [cuda92]  pytorch
[conda] torchvision               0.2.1                    py36_1    pytorch"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.2.148

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla V100-SXM2-16GB
Nvidia driver version: 396.44
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_static_v7.a

Versions of relevant libraries:
[pip] numpy (1.14.3)
[pip] numpydoc (0.8.0)
[pip] torch (0.4.1)
[pip] torchvision (0.2.1)
[conda] cuda92                    1.0                           0    pytorch
[conda] pytorch                   0.4.1           py36_cuda9.2.148_cudnn7.1.4_1  [cuda92]  pytorch
[conda] torchvision               0.2.1                    py36_1    pytorch
</code></pre></div>
<h2 dir="auto">Issue description</h2>
<p dir="auto">The following code produces a <code class="notranslate">Floating point exception (core dumped)</code> on Volta series with fp16:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
from torch.nn import Conv2d
conv=torch.nn.Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False).cuda().half()
x=torch.rand(2,256,196,196, requires_grad=True).cuda().half()
y=conv(x)
loss=y.min()
loss.backward()"><pre class="notranslate"><code class="notranslate">import torch
from torch.nn import Conv2d
conv=torch.nn.Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False).cuda().half()
x=torch.rand(2,256,196,196, requires_grad=True).cuda().half()
y=conv(x)
loss=y.min()
loss.backward()
</code></pre></div>
<p dir="auto">By changing the convolution parameters by +1/-1 the code works.</p>
<p dir="auto">I suspect this to be a cudnn bug.</p>
<h2 dir="auto">System Info</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 0.4.0a0+3749c58
Is debug build: No
CUDA used to build PyTorch: 9.2.88

OS: Ubuntu 16.04.4 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: 9.2.88
GPU models and configuration: 
GPU 0: TITAN V
GPU 1: TITAN V
GPU 2: TITAN V
GPU 3: TITAN V

Nvidia driver version: 396.26
cuDNN version: Probably one of the following:
/usr/local/cuda-9.2/cuda/lib64/libcudnn.so.7.1.4
/usr/local/cuda-9.2/cuda/lib64/libcudnn_static.a
/usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudnn.so.7.1.4
/usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudnn_static.a

Versions of relevant libraries:
[pip3] numpy (1.14.5)
[pip3] torch (0.4.0a0+3749c58)
[pip3] torchvision (0.2.1)
[conda] Could not collect"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 0.4.0a0+3749c58
Is debug build: No
CUDA used to build PyTorch: 9.2.88

OS: Ubuntu 16.04.4 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: 9.2.88
GPU models and configuration: 
GPU 0: TITAN V
GPU 1: TITAN V
GPU 2: TITAN V
GPU 3: TITAN V

Nvidia driver version: 396.26
cuDNN version: Probably one of the following:
/usr/local/cuda-9.2/cuda/lib64/libcudnn.so.7.1.4
/usr/local/cuda-9.2/cuda/lib64/libcudnn_static.a
/usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudnn.so.7.1.4
/usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudnn_static.a

Versions of relevant libraries:
[pip3] numpy (1.14.5)
[pip3] torch (0.4.0a0+3749c58)
[pip3] torchvision (0.2.1)
[conda] Could not collect
</code></pre></div>
<p dir="auto">Reproducible also on V100.</p>