<p dir="auto">The function divides by the entire number of elements and not just batch sample size.<br>
For example, using</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="loss_f1 = nn.MSELoss()
loss_f2 = nn.MSELoss(size_average=False)

x = autograd.Variable(torch.FloatTensor([    [1,2,3], [1,2,3]   ]))

y = autograd.Variable(torch.FloatTensor([    [2,4,6], [3,4,7]   ]))

# sample size of current batch = 2
# dimension of features = 3"><pre class="notranslate"><code class="notranslate">loss_f1 = nn.MSELoss()
loss_f2 = nn.MSELoss(size_average=False)

x = autograd.Variable(torch.FloatTensor([    [1,2,3], [1,2,3]   ]))

y = autograd.Variable(torch.FloatTensor([    [2,4,6], [3,4,7]   ]))

# sample size of current batch = 2
# dimension of features = 3
</code></pre></div>
<p dir="auto">shows the problem.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
# loss no averaging/no division...just plain summing up
loss_2 = loss_f2(x, y) # = 38

# averaged loss over batch sample size
loss_1 = loss_f1(x, y) # = 6.3333

# Apparently division by 6 was done in the second averaging case."><pre class="notranslate"><code class="notranslate">
# loss no averaging/no division...just plain summing up
loss_2 = loss_f2(x, y) # = 38

# averaged loss over batch sample size
loss_1 = loss_f1(x, y) # = 6.3333

# Apparently division by 6 was done in the second averaging case.
</code></pre></div>
<p dir="auto">The documentation describes the implemented MSE loss as</p>
<blockquote>
<p dir="auto">loss(x,y)=1/n (sum(||x-y||**2)</p>
</blockquote>
<p dir="auto">where n is the sample size of the current batch.<br>
However, apparently the expression does not get divided by n (which would be 2 in our example), but by 2*3=6, i.e. the entire number of elements per training batch.</p>
<p dir="auto">Is this intended? From a theoretical standpoint it should be 2, shouldn't it?</p>
<p dir="auto">Setting <code class="notranslate">size_average=False</code> behaves differently for <code class="notranslate">mse_loss</code> than for <code class="notranslate">cross_entropy</code>. For <code class="notranslate">cross_entropy</code>, it multiplies the result by the batch size. For <code class="notranslate">mse_loss</code> it multiplies the result by the overall size of the matrix.</p>
<p dir="auto">I believe the <code class="notranslate">cross_entropy</code> behavior is correct and <code class="notranslate">mse_loss</code> should be changed to behave similarly. Alternatively, the documentation should be changed to clarify the difference.</p>
<p dir="auto">Actually, <code class="notranslate">mse_loss</code> behaves similarly to <code class="notranslate">kl_div</code>, but at least the behavior is documented in <code class="notranslate">kl_div</code>. (But before reading the documentation, I expected also <code class="notranslate">kl_div</code> to behave similarly to <code class="notranslate">cross_entropy</code>.) I haven't checked how other loss functions behave.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch.version
from torch.nn import functional as F
from torch import FloatTensor
from torch.autograd import Variable

def v(array):
    return Variable(FloatTensor(array))

def kl(logit, target, size_average=True):
    return F.kl_div(F.log_softmax(logit), target, size_average=size_average)

def xe(logit, target, size_average=True):
    return F.cross_entropy(logit, target.max(1)[1], size_average=size_average)

def mse(logit, target, size_average=True):
    return F.mse_loss(F.softmax(logit), target, size_average=size_average)

target = v(np.eye(5)[:3])
logit = v(np.zeros((3, 5)))

print(&quot;Torch version:&quot;, torch.__version__)
print()
print(&quot;How each loss function scales the result with size_average:&quot;)
for loss in [kl, mse, xe]:
    with_size_average = loss(logit, target, size_average=True)
    without_size_average = loss(logit, target, size_average=False)
    factor = (without_size_average / with_size_average).data[0]
    print(&quot;- {loss.__name__}: {factor}&quot;.format(**locals()))```"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>.<span class="pl-s1">version</span>
<span class="pl-k">from</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span> <span class="pl-k">import</span> <span class="pl-s1">functional</span> <span class="pl-k">as</span> <span class="pl-v">F</span>
<span class="pl-k">from</span> <span class="pl-s1">torch</span> <span class="pl-k">import</span> <span class="pl-v">FloatTensor</span>
<span class="pl-k">from</span> <span class="pl-s1">torch</span>.<span class="pl-s1">autograd</span> <span class="pl-k">import</span> <span class="pl-v">Variable</span>

<span class="pl-k">def</span> <span class="pl-en">v</span>(<span class="pl-s1">array</span>):
    <span class="pl-k">return</span> <span class="pl-v">Variable</span>(<span class="pl-v">FloatTensor</span>(<span class="pl-s1">array</span>))

<span class="pl-k">def</span> <span class="pl-en">kl</span>(<span class="pl-s1">logit</span>, <span class="pl-s1">target</span>, <span class="pl-s1">size_average</span><span class="pl-c1">=</span><span class="pl-c1">True</span>):
    <span class="pl-k">return</span> <span class="pl-v">F</span>.<span class="pl-en">kl_div</span>(<span class="pl-v">F</span>.<span class="pl-en">log_softmax</span>(<span class="pl-s1">logit</span>), <span class="pl-s1">target</span>, <span class="pl-s1">size_average</span><span class="pl-c1">=</span><span class="pl-s1">size_average</span>)

<span class="pl-k">def</span> <span class="pl-en">xe</span>(<span class="pl-s1">logit</span>, <span class="pl-s1">target</span>, <span class="pl-s1">size_average</span><span class="pl-c1">=</span><span class="pl-c1">True</span>):
    <span class="pl-k">return</span> <span class="pl-v">F</span>.<span class="pl-en">cross_entropy</span>(<span class="pl-s1">logit</span>, <span class="pl-s1">target</span>.<span class="pl-en">max</span>(<span class="pl-c1">1</span>)[<span class="pl-c1">1</span>], <span class="pl-s1">size_average</span><span class="pl-c1">=</span><span class="pl-s1">size_average</span>)

<span class="pl-k">def</span> <span class="pl-en">mse</span>(<span class="pl-s1">logit</span>, <span class="pl-s1">target</span>, <span class="pl-s1">size_average</span><span class="pl-c1">=</span><span class="pl-c1">True</span>):
    <span class="pl-k">return</span> <span class="pl-v">F</span>.<span class="pl-en">mse_loss</span>(<span class="pl-v">F</span>.<span class="pl-en">softmax</span>(<span class="pl-s1">logit</span>), <span class="pl-s1">target</span>, <span class="pl-s1">size_average</span><span class="pl-c1">=</span><span class="pl-s1">size_average</span>)

<span class="pl-s1">target</span> <span class="pl-c1">=</span> <span class="pl-en">v</span>(<span class="pl-s1">np</span>.<span class="pl-en">eye</span>(<span class="pl-c1">5</span>)[:<span class="pl-c1">3</span>])
<span class="pl-s1">logit</span> <span class="pl-c1">=</span> <span class="pl-en">v</span>(<span class="pl-s1">np</span>.<span class="pl-en">zeros</span>((<span class="pl-c1">3</span>, <span class="pl-c1">5</span>)))

<span class="pl-en">print</span>(<span class="pl-s">"Torch version:"</span>, <span class="pl-s1">torch</span>.<span class="pl-s1">__version__</span>)
<span class="pl-en">print</span>()
<span class="pl-en">print</span>(<span class="pl-s">"How each loss function scales the result with size_average:"</span>)
<span class="pl-k">for</span> <span class="pl-s1">loss</span> <span class="pl-c1">in</span> [<span class="pl-s1">kl</span>, <span class="pl-s1">mse</span>, <span class="pl-s1">xe</span>]:
    <span class="pl-s1">with_size_average</span> <span class="pl-c1">=</span> <span class="pl-en">loss</span>(<span class="pl-s1">logit</span>, <span class="pl-s1">target</span>, <span class="pl-s1">size_average</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
    <span class="pl-s1">without_size_average</span> <span class="pl-c1">=</span> <span class="pl-en">loss</span>(<span class="pl-s1">logit</span>, <span class="pl-s1">target</span>, <span class="pl-s1">size_average</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)
    <span class="pl-s1">factor</span> <span class="pl-c1">=</span> (<span class="pl-s1">without_size_average</span> <span class="pl-c1">/</span> <span class="pl-s1">with_size_average</span>).<span class="pl-s1">data</span>[<span class="pl-c1">0</span>]
    <span class="pl-en">print</span>(<span class="pl-s">"- {loss.__name__}: {factor}"</span>.<span class="pl-en">format</span>(<span class="pl-c1">**</span><span class="pl-en">locals</span>()))<span class="pl-s">``</span>`</pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Torch version: 0.2.0+40ca356

How each loss function scales the result with size_average:
- kl: 15.0
- mse: 15.0
- xe: 3.0"><pre class="notranslate"><code class="notranslate">Torch version: 0.2.0+40ca356

How each loss function scales the result with size_average:
- kl: 15.0
- mse: 15.0
- xe: 3.0
</code></pre></div>