Adding my custom loss messes up autograd in Pytorch?
I'm trying to use two different losses, MSELoss for some of my labels and a custom loss for the other labels. I'm then trying to sum these losses together before backprop. My model prints out the same loss after every epoch so I must be doing something wrong. Any help is appreciated! I suspect my implementation is messing up Pytorch's autograd. See code below:
mse_loss = torch.nn.MSELoss()
...
loss1 = mse_loss(preds[:,(0,1,3)], label[:,(0,1,3)])
print(&quot;loss1&quot;, loss1)
loss2 = my_custom_loss(preds[:,2], label[:,2])
print(&quot;loss2&quot;, loss2)
print(&quot;summing losses&quot;)
loss = sum([loss1, loss2]) # tensor + float = tensor
print(&quot;loss sum&quot;, loss)
loss = torch.autograd.Variable(loss, requires_grad=True)
print(&quot;loss after Variable(loss, requires_grad=True)&quot;, loss)

These print statements yield:
loss1 tensor(4946.1221, device='cuda:0', grad_fn=&lt;MseLossBackward0&gt;)
loss2 34.6672
summing losses
loss sum tensor(4980.7891, device='cuda:0', grad_fn=&lt;AddBackward0&gt;)
loss after Variable() tensor(4980.7891, device='cuda:0', requires_grad=True)

My custom loss function is below:
def my_custom_loss(preds, label):
    angle_diff = preds - label
    # /2 to bring angle diff between -180&lt;theta&lt;180
    half_angle_diff = angle_diff.detach().cpu().numpy()/2
    sine_diff = np.sin(half_angle_diff)
    square_sum = np.nansum(sine_diff**2)
    return square_sum


The reason why you are not backpropagating through your second loss is that you haven't defined it as a differentiable operator. You should stick with PyTorch operators without switching to NumPy.
Something like this will work:
def my_custom_loss(preds, label):
    half_angle_diff = (preds - label)/2
    sine_diff = torch.sin(half_angle_diff)
    square_sum = torch.nansum(sine_diff**2)
    return square_sum


You can check that your custom loss is differentiable with dummy inputs:
&gt;&gt;&gt; preds = torch.rand(1,3,10,10, requires_grad=True)
&gt;&gt;&gt; label = torch.rand(1,3,10,10)
&gt;&gt;&gt; my_custom_loss(preds, label)
tensor(11.7584, grad_fn=&lt;NansumBackward0&gt;)

Notice the grad_fn attribute on it which shows the output tensor is indeed attached to a computational graph, and you can therefore perform back propagation from it.
Additionally, you should not use Variable as it is now deprecated.
