modification from classification loss to regression loss
General
I am following this repo for object detection
https://github.com/yhenon/pytorch-retinanet
Motivation
Object detection networks usually perform 2 tasks.For every object in the image output a class confidence score and bounding box regression score.For my task along with these 2 outputs, for every object i want to output one more regression score which will be between 0-5.
Problem statement
The approach i have taken is that since the network already does classifcation i thought i would modify some of the parts and make it a regression loss.The loss used in the repo mentioned above is focal loss.Part of what the classification loss looks like is described below, i would like to modify this to be a regression loss.
     targets = torch.ones(classification.shape) * -1#Here classification shape is = torch.Size([114048, 1])
    targets = targets.cuda()
    targets[torch.lt(IoU_max, 0.4), :] = 0#Iou_max is a tensor of shape = torch.Size([114048]) which looks like tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:0')
    positive_indices = torch.ge(IoU_max, 0.5)
    num_positive_anchors = positive_indices.sum()
    assigned_annotations = bbox_annotation[IoU_argmax, :]#Here bbox_annotation has the shape torch.Size([6, 6]) and IoU_argmax has the shape torch.Size([114048])
    targets[positive_indices, :] = 0#Here positive indices has the shape torch.Size([114048])
    targets[positive_indices, assigned_annotations[positive_indices, 4].long()] = 1
    alpha_factor = torch.ones(targets.shape).cuda() * alpha#here alpha is 0.25
    alpha_factor = torch.where(torch.eq(targets, 1.), alpha_factor, 1. - alpha_factor)
    focal_weight = torch.where(torch.eq(targets, 1.), 1. - classification, classification)
    focal_weight = alpha_factor * torch.pow(focal_weight, gamma)#here gamma is 2.0

    bce = -(targets * torch.log(classification) + (1.0 - targets) * torch.log(1.0 - classification))
    cls_loss = focal_weight * bce
    cls_loss = torch.where(torch.ne(targets, -1.0), cls_loss,torch.zeros(cls_loss.shape).cuda())
  classification_losses.append(cls_loss.sum()/torch.clamp(num_positive_anchors.float(), min=1.0))

So i tried modifying some of the bits described above.But even though the code does not break This does not learn anything.Here erosion is just a tensor but erosion score is something we want to regress.
    targets = torch.ones(erosion.shape) * -1#Here erosion shape is torch.Size([114048, 1])
    targets = targets.cuda()
    targets[torch.lt(IoU_max, 0.4), :] = 0#This part remains the same as above
    positive_indices = torch.ge(IoU_max, 0.5)
    num_positive_anchors = positive_indices.sum()
    assigned_annotations = bbox_annotation[IoU_argmax, :]
    targets[positive_indices, :] = 0
    targets[assigned_annotations[positive_indices, 5].long()] = 1#Here i have indexed along the 5th dimension because the 5th dimension contains annotations for erosion.
    criterion = nn.MSELoss()#This part is where i use the mean squared loss for regression. 
    loss = torch.sqrt(criterion(targets, erosion))
    erosion_losses.append(loss)

This will not throw an error , but it does not learn anything either.
Could someone please help me with the correct formulation of this loss to perform regression.
More info
        assigned_annotations_erosion = assigned_annotations[positive_indices, 5]
        erosion_pred =  erosion[positive_indices].view(-1)
        criterion = nn.MSELoss()#
        loss = torch.sqrt(criterion(assigned_annotations_erosion, erosion_pred))
        loss *torch.tensor(5.0).cuda()
        # import pdb;pdb.set_trace()
        erosion_losses.append(loss)


If I understand it correctly you want to add another regression parameter which should have values in [0-5].

Instead of trying to change the classification part you can just add it to the box regression part. So you add one more parameter to predict for every anchor.

Here is the loss calculation for the regression parameters. If you added one parameter to every anchor (probably by increasing the filter count in the last layer) you have to "extract" it here like with the other parameters. You also probably want to think about the activation for the value e.g. sigmoid so you get something between 0 and 1, you can rescale to 0-5 if you want.

After line 107 you want to add something like this

anchor_erosion_pi = anchor_erosion[positive_indices]


Then you have to get the gt somehow, maybe rescale it to 0-1, so you can just use the sigmoid activated output for this value.
Then you have to set the target. This is just:

targets_erosion = gt_erosion


If you are not dependent on the anchor which seems like it is the case.
After that the targets are stacked, where you have to add your new target value and a weighting is added, you have to decide and try out if you need a weighting here. I would probably add start by adding 0.2 or something to the tensor. Maybe have a look inside the paper how these other weightings are chosen.

targets = targets/torch.Tensor([[0.1, 0.1, 0.2, 0.2, 0.2]]).cuda()


After that the Smooth L1 loss is calculated where you probably don't need to change anything.

there is probably more you have to do. Just look at the code, see where the other parameters are and try to add your parameter to it.
