Computing loss along batch dimension in PyTorch
I have tensors of BxCxHxW where B is the batch. I am interested in how is the loss implemented in PyTorch when batch size is more than 1. The below is what we usually do in PyTorch,
l1_loss = torch.nn.L1Loss()
loss = l1_loss(pred,gt)

Is it that the loss is the average loss over the batch?
If this is indeed the case, then is the following codes equivalent to what is we usually do in PyTorch (the code above)?
l1_loss = torch.nn.L1Loss()

for idx in range(B):
  if idx==0:
    loss = l1_loss(pred[idx,:,:,:],gt[idx,:,:,:])
  else:
    loss = l1_loss(pred[idx,:,:,:],gt[idx,:,:,:]) + loss

loss = loss/B


The documentation describes the behavior of L1loss : it is indeed (by default) the mean over the whole batch. You can change it easily to the sum instead :
l1_loss = torch.nn.L1Loss(reduction='sum')

Yes your code is equivalent to what Pytorch does. A version without the call to L1loss would be :
# Assuming your output is a vector of shape (B, F)
loss = torch.abs(pred - gt).sum(dim=1).mean(dim=0)

