<h2 dir="auto">🐛 Bug</h2>
<p dir="auto">Running a 2d convolution (no padding, stride=1) on a 3x1 input, sliced from a 3x3 tensor, and a 3x1 weight, sliced from a 3x3 tensor. When trying to run it on CPU the convolution result is wrong, but when moving to GPU (by adding .cuda() after the tensors), the result is correct!</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Run the code below to reproduce the bug. It's pretty self explanatory.</p>

<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
import torch.nn.functional as F

x = torch.Tensor([[[[0., 2., 2.],
                    [0., 0., 1.],
                    [2., 0., 0.]]]])
w = torch.Tensor([[[[1., 1., 0.],
                    [1., 0., 3.],
                    [0., 3., 1.]]]])

x1 = x[:,:,:,2:] # Using only the last column from x, containing [2,1,0]
w1 = w[:,:,:,2:] # Using only the last column from w, containing [0,3,1]
#The result should be ((2*0) + (1*3) + (0*1)) = 3

print(F.conv2d(x1, w1, None, padding=0, stride=1)) # tensor([[[[1.]]]]) WRONG!!!!
print(F.conv2d(x1.cuda(), w1.cuda(), None, padding=0, stride=1)) # tensor([[[[3.]]]], device='cuda:0') as expected"><pre class="notranslate"><code class="notranslate">import torch
import torch.nn.functional as F

x = torch.Tensor([[[[0., 2., 2.],
                    [0., 0., 1.],
                    [2., 0., 0.]]]])
w = torch.Tensor([[[[1., 1., 0.],
                    [1., 0., 3.],
                    [0., 3., 1.]]]])

x1 = x[:,:,:,2:] # Using only the last column from x, containing [2,1,0]
w1 = w[:,:,:,2:] # Using only the last column from w, containing [0,3,1]
#The result should be ((2*0) + (1*3) + (0*1)) = 3

print(F.conv2d(x1, w1, None, padding=0, stride=1)) # tensor([[[[1.]]]]) WRONG!!!!
print(F.conv2d(x1.cuda(), w1.cuda(), None, padding=0, stride=1)) # tensor([[[[3.]]]], device='cuda:0') as expected
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>

<p dir="auto">As seen in the code above, the CPU gives the result of 1, when the actual result should be 3.</p>
<h2 dir="auto">Environment</h2>
<ul dir="auto">
<li>PyTorch Version (e.g., 1.0): 1.0</li>
<li>OS (e.g., Linux): Ubuntu 16.04</li>
<li>How you installed PyTorch (<code class="notranslate">conda</code>, <code class="notranslate">pip</code>, source): pip</li>
<li>Build command you used (if compiling from source):</li>
<li>Python version: 3.5.2</li>
<li>CUDA/cuDNN version: 9.0</li>
<li>GPU models and configuration: GeForce GTX 1060</li>
<li>Any other relevant information:</li>
</ul>
<h2 dir="auto">Additional context</h2>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">If I load convolution weights from a NumPy float32 array, the result of the forward pass does not match that of a layer with the same weights loaded from a NumPy float64 array or of a Keras convolution layer with the same weights. Casting the layer to <code class="notranslate">double</code> fixes the discrepancy, even after casting back to <code class="notranslate">float</code>. Comparing the convolution weights before and after casting shows that they have not changed.</p>
<h2 dir="auto">To Reproduce</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import torch
from torch import nn

d_in = 25
d_out = 64
k = 7
B = 2
L = 10

conv = nn.Conv1d(d_in, d_out, k, stride=2, padding=0)
pad = nn.ConstantPad1d((2, 3), 0)
np.random.seed(1034)
X = np.random.random((B, L, d_in))
X = torch.as_tensor(X).transpose(-1, -2)
X = pad(X)
w = np.random.random((k, d_in, d_out))
w = w.astype('float32')
b = np.random.random((d_out, ))

w = torch.as_tensor(w).permute(2, 1, 0).float()
b = torch.as_tensor(b).float()

conv.weight = nn.Parameter(w)
conv.bias = nn.Parameter(b)

out1 = conv(X.float()).detach().numpy()
w1 = conv.weight.detach().numpy()
out2 = conv.double()(X).detach().numpy()
w2 = conv.weight.detach().numpy()
out3 = conv.float()(X.float()).detach().numpy()
w3 = conv.weight.detach().numpy()
print('w == w1?')
print(np.allclose(w.detach().numpy(), w1))
print('w1 == w2?')
print(np.allclose(w1, w2))
print('w2 == w3?')
print(np.allclose(w2, w3))
print('out1 == out2?')
print(np.allclose(out1, out2, atol=1e-6))
print('out2 == out3?')
print(np.allclose(out2, out3, atol=1e-6))"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">from</span> <span class="pl-s1">torch</span> <span class="pl-k">import</span> <span class="pl-s1">nn</span>

<span class="pl-s1">d_in</span> <span class="pl-c1">=</span> <span class="pl-c1">25</span>
<span class="pl-s1">d_out</span> <span class="pl-c1">=</span> <span class="pl-c1">64</span>
<span class="pl-s1">k</span> <span class="pl-c1">=</span> <span class="pl-c1">7</span>
<span class="pl-v">B</span> <span class="pl-c1">=</span> <span class="pl-c1">2</span>
<span class="pl-v">L</span> <span class="pl-c1">=</span> <span class="pl-c1">10</span>

<span class="pl-s1">conv</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Conv1d</span>(<span class="pl-s1">d_in</span>, <span class="pl-s1">d_out</span>, <span class="pl-s1">k</span>, <span class="pl-s1">stride</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, <span class="pl-s1">padding</span><span class="pl-c1">=</span><span class="pl-c1">0</span>)
<span class="pl-s1">pad</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">ConstantPad1d</span>((<span class="pl-c1">2</span>, <span class="pl-c1">3</span>), <span class="pl-c1">0</span>)
<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">seed</span>(<span class="pl-c1">1034</span>)
<span class="pl-v">X</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">random</span>((<span class="pl-v">B</span>, <span class="pl-v">L</span>, <span class="pl-s1">d_in</span>))
<span class="pl-v">X</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">as_tensor</span>(<span class="pl-v">X</span>).<span class="pl-en">transpose</span>(<span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-c1">-</span><span class="pl-c1">2</span>)
<span class="pl-v">X</span> <span class="pl-c1">=</span> <span class="pl-en">pad</span>(<span class="pl-v">X</span>)
<span class="pl-s1">w</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">random</span>((<span class="pl-s1">k</span>, <span class="pl-s1">d_in</span>, <span class="pl-s1">d_out</span>))
<span class="pl-s1">w</span> <span class="pl-c1">=</span> <span class="pl-s1">w</span>.<span class="pl-en">astype</span>(<span class="pl-s">'float32'</span>)
<span class="pl-s1">b</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">random</span>((<span class="pl-s1">d_out</span>, ))

<span class="pl-s1">w</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">as_tensor</span>(<span class="pl-s1">w</span>).<span class="pl-en">permute</span>(<span class="pl-c1">2</span>, <span class="pl-c1">1</span>, <span class="pl-c1">0</span>).<span class="pl-en">float</span>()
<span class="pl-s1">b</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">as_tensor</span>(<span class="pl-s1">b</span>).<span class="pl-en">float</span>()

<span class="pl-s1">conv</span>.<span class="pl-s1">weight</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Parameter</span>(<span class="pl-s1">w</span>)
<span class="pl-s1">conv</span>.<span class="pl-s1">bias</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Parameter</span>(<span class="pl-s1">b</span>)

<span class="pl-s1">out1</span> <span class="pl-c1">=</span> <span class="pl-en">conv</span>(<span class="pl-v">X</span>.<span class="pl-en">float</span>()).<span class="pl-en">detach</span>().<span class="pl-en">numpy</span>()
<span class="pl-s1">w1</span> <span class="pl-c1">=</span> <span class="pl-s1">conv</span>.<span class="pl-s1">weight</span>.<span class="pl-en">detach</span>().<span class="pl-en">numpy</span>()
<span class="pl-s1">out2</span> <span class="pl-c1">=</span> <span class="pl-s1">conv</span>.<span class="pl-en">double</span>()(<span class="pl-v">X</span>).<span class="pl-en">detach</span>().<span class="pl-en">numpy</span>()
<span class="pl-s1">w2</span> <span class="pl-c1">=</span> <span class="pl-s1">conv</span>.<span class="pl-s1">weight</span>.<span class="pl-en">detach</span>().<span class="pl-en">numpy</span>()
<span class="pl-s1">out3</span> <span class="pl-c1">=</span> <span class="pl-s1">conv</span>.<span class="pl-en">float</span>()(<span class="pl-v">X</span>.<span class="pl-en">float</span>()).<span class="pl-en">detach</span>().<span class="pl-en">numpy</span>()
<span class="pl-s1">w3</span> <span class="pl-c1">=</span> <span class="pl-s1">conv</span>.<span class="pl-s1">weight</span>.<span class="pl-en">detach</span>().<span class="pl-en">numpy</span>()
<span class="pl-en">print</span>(<span class="pl-s">'w == w1?'</span>)
<span class="pl-en">print</span>(<span class="pl-s1">np</span>.<span class="pl-en">allclose</span>(<span class="pl-s1">w</span>.<span class="pl-en">detach</span>().<span class="pl-en">numpy</span>(), <span class="pl-s1">w1</span>))
<span class="pl-en">print</span>(<span class="pl-s">'w1 == w2?'</span>)
<span class="pl-en">print</span>(<span class="pl-s1">np</span>.<span class="pl-en">allclose</span>(<span class="pl-s1">w1</span>, <span class="pl-s1">w2</span>))
<span class="pl-en">print</span>(<span class="pl-s">'w2 == w3?'</span>)
<span class="pl-en">print</span>(<span class="pl-s1">np</span>.<span class="pl-en">allclose</span>(<span class="pl-s1">w2</span>, <span class="pl-s1">w3</span>))
<span class="pl-en">print</span>(<span class="pl-s">'out1 == out2?'</span>)
<span class="pl-en">print</span>(<span class="pl-s1">np</span>.<span class="pl-en">allclose</span>(<span class="pl-s1">out1</span>, <span class="pl-s1">out2</span>, <span class="pl-s1">atol</span><span class="pl-c1">=</span><span class="pl-c1">1e-6</span>))
<span class="pl-en">print</span>(<span class="pl-s">'out2 == out3?'</span>)
<span class="pl-en">print</span>(<span class="pl-s1">np</span>.<span class="pl-en">allclose</span>(<span class="pl-s1">out2</span>, <span class="pl-s1">out3</span>, <span class="pl-s1">atol</span><span class="pl-c1">=</span><span class="pl-c1">1e-6</span>))</pre></div>
<p dir="auto">Result:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="w == w1?
True
w1 == w2?
True
w2 == w3?
True
out1 == out2?
False
out2 == out3?
True"><pre class="notranslate"><code class="notranslate">w == w1?
True
w1 == w2?
True
w2 == w3?
True
out1 == out2?
False
out2 == out3?
True
</code></pre></div>
<p dir="auto">Some notes:</p>
<ol dir="auto">
<li>The type for the bias doesn't seem to matter.</li>
<li><code class="notranslate">out2</code> and <code class="notranslate">out3</code> match the result for a Keras convolution layer with the same weights.</li>
</ol>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto"><code class="notranslate">out1</code> should be within machine precision of <code class="notranslate">out2</code> and <code class="notranslate">out3</code>.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">PyTorch version: 1.0.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: None</p>
<p dir="auto">OS: Mac OSX 10.14.2<br>
GCC version: Could not collect<br>
CMake version: Could not collect</p>
<p dir="auto">Python version: 3.7<br>
Is CUDA available: No<br>
CUDA runtime version: No CUDA<br>
GPU models and configuration: No CUDA<br>
Nvidia driver version: No CUDA<br>
cuDNN version: No CUDA</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] Could not collect<br>
[conda] blas                      1.0                         mkl<br>
[conda] mkl                       2019.1                      144<br>
[conda] mkl-service               1.1.2            py37hfbe908c_5<br>
[conda] mkl_fft                   1.0.6            py37h27c97d8_0<br>
[conda] mkl_random                1.0.2            py37h27c97d8_0<br>
[conda] pytorch                   1.0.0                   py3.7_1    pytorch<br>
[conda] torchvision               0.2.1                      py_2    pytorch<br>
[conda] numpy                    1.15.4</p>