title: loss calculation over different batch sizes in keras
body: <p>I know that in theory, the loss of a network over a batch is just the sum of all the individual losses. This is reflected in the <a href="https://github.com/keras-team/keras/blob/58fd1f0589d33aeb33c4129cfedfb7737495efc0/keras/engine/training.py#L309" rel="noreferrer">Keras code</a> for calculating total loss. Relevantly:</p>
  
  <pre><code>            for i in range(len(self.outputs)):
              if i in skip_target_indices:
                  continue
              y_true = self.targets[i]
              y_pred = self.outputs[i]
              weighted_loss = weighted_losses[i]
              sample_weight = sample_weights[i]
              mask = masks[i]
              loss_weight = loss_weights_list[i]
              with K.name_scope(self.output_names[i] + '_loss'):
                  output_loss = weighted_loss(y_true, y_pred,
                                              sample_weight, mask)
              if len(self.outputs) &gt; 1:
                  self.metrics_tensors.append(output_loss)
                  self.metrics_names.append(self.output_names[i] + '_loss')
              if total_loss is None:
                  total_loss = loss_weight * output_loss
              else:
                  total_loss += loss_weight * output_loss
  </code></pre>
  
  <p>However, I noticed that when I train a network with a <code>batch_size=32</code> and a <code>batch_size=64</code>, the loss value for every epoch still comes out to more or less the same with only a <code>~0.05%</code> difference. However, the accuracy for both networks remained the exact same. So essentially, the batch size didn't have too much effect on the network.</p>
  
  <p>My question is when I double the batch size, assuming the loss is really being summed, shouldn't the loss in fact be double the value it was previously, or at least greater? The excuse that the network  probably learned better with the bigger batch size is negated by the fact the accuracy has stayed exactly the same.</p>
  
  <p>The fact that the loss stays more or less the same regardless of the batch size makes me think it's being averaged.</p>