<h2 dir="auto">Issue description</h2>
<p dir="auto">I was training an autoencoder with MSELoss, and the loss values on the training data were huge but the loss values on the validation data were small. It appeared as if the loss was not being averaged on the training pass, but it was on the validation pass. A little poking around in the debugger revealed this to be the case.</p>
<p dir="auto">The problem is in the <code class="notranslate">_pointwise_loss</code> loss function in <code class="notranslate">torch/nn/functional.py</code>:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def _pointwise_loss(lambd, lambd_optimized, input, target, reduction='elementwise_mean'):
    if target.requires_grad:
        d = lambd(input, target)
        if reduction == 'none':
            return d
        return torch.mean(d) if reduction == 'elementwise_mean' else torch.sum(d)
    else:
        return lambd_optimized(input, target, _Reduction.get_enum(reduction))"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">_pointwise_loss</span>(<span class="pl-s1">lambd</span>, <span class="pl-s1">lambd_optimized</span>, <span class="pl-s1">input</span>, <span class="pl-s1">target</span>, <span class="pl-s1">reduction</span><span class="pl-c1">=</span><span class="pl-s">'elementwise_mean'</span>):
    <span class="pl-k">if</span> <span class="pl-s1">target</span>.<span class="pl-s1">requires_grad</span>:
        <span class="pl-s1">d</span> <span class="pl-c1">=</span> <span class="pl-en">lambd</span>(<span class="pl-s1">input</span>, <span class="pl-s1">target</span>)
        <span class="pl-k">if</span> <span class="pl-s1">reduction</span> <span class="pl-c1">==</span> <span class="pl-s">'none'</span>:
            <span class="pl-k">return</span> <span class="pl-s1">d</span>
        <span class="pl-k">return</span> <span class="pl-s1">torch</span>.<span class="pl-en">mean</span>(<span class="pl-s1">d</span>) <span class="pl-k">if</span> <span class="pl-s1">reduction</span> <span class="pl-c1">==</span> <span class="pl-s">'elementwise_mean'</span> <span class="pl-k">else</span> <span class="pl-s1">torch</span>.<span class="pl-en">sum</span>(<span class="pl-s1">d</span>)
    <span class="pl-k">else</span>:
        <span class="pl-k">return</span> <span class="pl-en">lambd_optimized</span>(<span class="pl-s1">input</span>, <span class="pl-s1">target</span>, <span class="pl-s1">_Reduction</span>.<span class="pl-en">get_enum</span>(<span class="pl-s1">reduction</span>))</pre></div>
<p dir="auto">In the line <code class="notranslate">return torch.mean(d) if reduction == 'elementwise_mean' else torch.sum(d)</code>, the variable <code class="notranslate">reduction</code> is actually an integer with value <code class="notranslate">1</code>, not a string with value <code class="notranslate">'elementwise_mean'</code>, and so the loss is summed instead of averaged, even though the 'elementwise_mean` option was chosen.</p>
<h2 dir="auto">Code example</h2>
<p dir="auto">This code reproduces the problem:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
import torchvision as tv


class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.c1 = torch.nn.Conv2d(1, 16, 3, padding=1)
        self.c2 = torch.nn.Conv2d(16, 1, 3, padding=1)
        self.relu = torch.nn.ReLU(inplace=True)
    
    def forward(self, x):
        x = self.c2(self.relu(self.c1(x)))
        return x

net = Net()
data = torchvision.datasets.MNIST('MNIST', True, download=True, transform=tv.transforms.ToTensor())
mse = torch.nn.MSELoss(reduction='elementwise_mean')

img = data[0][0].view(1, 1, 28, 28).requires_grad_()
pred = net(img)

loss1 = mse(pred, img).item()

img.requires_grad = False
loss2 = mse(pred, img).item()

print(loss1, loss2)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">import</span> <span class="pl-s1">torchvision</span> <span class="pl-k">as</span> <span class="pl-s1">tv</span>


<span class="pl-k">class</span> <span class="pl-v">Net</span>(<span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>):
        <span class="pl-en">super</span>(<span class="pl-v">Net</span>, <span class="pl-s1">self</span>).<span class="pl-en">__init__</span>()
        <span class="pl-s1">self</span>.<span class="pl-s1">c1</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">Conv2d</span>(<span class="pl-c1">1</span>, <span class="pl-c1">16</span>, <span class="pl-c1">3</span>, <span class="pl-s1">padding</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">c2</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">Conv2d</span>(<span class="pl-c1">16</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>, <span class="pl-s1">padding</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">relu</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">ReLU</span>(<span class="pl-s1">inplace</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
    
    <span class="pl-k">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">x</span>):
        <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">c2</span>(<span class="pl-s1">self</span>.<span class="pl-en">relu</span>(<span class="pl-s1">self</span>.<span class="pl-en">c1</span>(<span class="pl-s1">x</span>)))
        <span class="pl-k">return</span> <span class="pl-s1">x</span>

<span class="pl-s1">net</span> <span class="pl-c1">=</span> <span class="pl-v">Net</span>()
<span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">torchvision</span>.<span class="pl-s1">datasets</span>.<span class="pl-v">MNIST</span>(<span class="pl-s">'MNIST'</span>, <span class="pl-c1">True</span>, <span class="pl-s1">download</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">transform</span><span class="pl-c1">=</span><span class="pl-s1">tv</span>.<span class="pl-s1">transforms</span>.<span class="pl-v">ToTensor</span>())
<span class="pl-s1">mse</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">MSELoss</span>(<span class="pl-s1">reduction</span><span class="pl-c1">=</span><span class="pl-s">'elementwise_mean'</span>)

<span class="pl-s1">img</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span>[<span class="pl-c1">0</span>][<span class="pl-c1">0</span>].<span class="pl-en">view</span>(<span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">28</span>, <span class="pl-c1">28</span>).<span class="pl-en">requires_grad_</span>()
<span class="pl-s1">pred</span> <span class="pl-c1">=</span> <span class="pl-en">net</span>(<span class="pl-s1">img</span>)

<span class="pl-s1">loss1</span> <span class="pl-c1">=</span> <span class="pl-en">mse</span>(<span class="pl-s1">pred</span>, <span class="pl-s1">img</span>).<span class="pl-en">item</span>()

<span class="pl-s1">img</span>.<span class="pl-s1">requires_grad</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
<span class="pl-s1">loss2</span> <span class="pl-c1">=</span> <span class="pl-en">mse</span>(<span class="pl-s1">pred</span>, <span class="pl-s1">img</span>).<span class="pl-en">item</span>()

<span class="pl-en">print</span>(<span class="pl-s1">loss1</span>, <span class="pl-s1">loss2</span>)</pre></div>
<p dir="auto">And the output is:<br>
<code class="notranslate">155.7478  0.1987</code></p>
<h2 dir="auto">System Info</h2>
<p dir="auto">PyTorch version: 0.4.1<br>
Is debug build: No<br>
CUDA used to build PyTorch: 9.0.176</p>
<p dir="auto">OS: Ubuntu 16.04.2 LTS<br>
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609<br>
CMake version: version 3.10.2</p>
<p dir="auto">Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: 7.5.17<br>
GPU models and configuration:<br>
GPU 0: GeForce GTX 1080 Ti<br>
GPU 1: GeForce GTX 1080 Ti<br>
GPU 2: GeForce GTX 1080 Ti<br>
GPU 3: GeForce GTX 1080 Ti<br>
GPU 4: GeForce GTX 1080 Ti<br>
GPU 5: GeForce GTX 1080 Ti<br>
GPU 6: GeForce GTX 1080 Ti<br>
GPU 7: GeForce GTX 1080 Ti</p>
<p dir="auto">Nvidia driver version: 387.34<br>
cuDNN version: Probably one of the following:<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5.0.5<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn_static.a<br>
/usr/local/cuda-9.0/lib64/libcudnn.so<br>
/usr/local/cuda-9.0/lib64/libcudnn.so.7<br>
/usr/local/cuda-9.0/lib64/libcudnn.so.7.0.3<br>
/usr/local/cuda-9.0/lib64/libcudnn_static.a</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] Could not collect<br>
[conda] Could not collect</p>
<h2 dir="auto">Issue description</h2>
<p dir="auto">If a tensor with requires_grad=True is passed to mse_loss, then the loss is reduced even if reduction is none.<br>
Appeared in Pytorch 0.4.1.</p>
<h2 dir="auto">Code example</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
x = torch.zeros((4, 5, 2))
print('Good', torch.nn.functional.mse_loss(x, x, reduction='none').shape)
x.requires_grad = True
print('Bad', torch.nn.functional.mse_loss(x, x, reduction='none').shape)"><pre lang="(python)" class="notranslate"><code class="notranslate">import torch
x = torch.zeros((4, 5, 2))
print('Good', torch.nn.functional.mse_loss(x, x, reduction='none').shape)
x.requires_grad = True
print('Bad', torch.nn.functional.mse_loss(x, x, reduction='none').shape)
</code></pre></div>
<p dir="auto">Outputs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Good torch.Size([4, 5, 2])
Bad torch.Size([])"><pre class="notranslate"><code class="notranslate">Good torch.Size([4, 5, 2])
Bad torch.Size([])
</code></pre></div>
<h2 dir="auto">System Info</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.3 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.6"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.3 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.6
</code></pre></div>