Reduction parameter of the negative log likelihood
What is the intuitive explanation of the reduction parameter in negative log likelihood loss function in PyTorch? The parameter can take values such as 'mean' or 'sum'. Is it summing over the elements of the batch?

torch.nn.functional.nll_loss(outputs.mean(0), target, reduction="sum")


From the documentation: 

Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'

If you use none, the output will be the same as batch size, 

If you use mean, it will be the mean (sum divided by batch)

If you use sum, it will be the sum of all elements. 

You can also verify this with following code:

import torch 
logit = torch.rand(100,10)
target = torch.randint(10, size=(100,)) 
m = torch.nn.functional.nll_loss(logit, target)
s = torch.nn.functional.nll_loss(logit, target, reduction="sum") 
l = torch.nn.functional.nll_loss(logit, target, reduction="none")
print(torch.abs(m-s/100))
print(torch.abs(l.mean()-m))


The output should be 0 or very close to 0. 
