How should I define the loss function for a multi-object detection problem?
Okay so your CNN model is taking an Image and outputting the bounding boxes for them. That means the last layer of CNN model must be having four outputs which are generating real numbers. This is a regression problem.
In that case, you can take L1 loss (mean absolute error) or L2 loss (mean square error) as your loss function. I have created a similar project, I have used L1 loss.
Suppose your input image is x and predicted_bb is the model output and real_bb is your original bounding box for image x. Then you should proceed as follows
predicted_bb = model(x)

# CALCULATE LOSS BETWEEN THE ACTUAL &amp; PREDICTED BB COORDINATES

loss_bb = torch.nn.functional.l1_loss(predicted_bb, real_bb, reduction=&quot;none&quot;).sum(1)

# SET GRADIENTS TO ZERO

optimizer.zero_grad()

# BACKPROPOGATE THE LOSS

loss.backward()

For tensorflow keras
predicted_bb = Dense(4, activation='relu')(x)

model = Model(inputs = image_input, outputs = [predicted_bb])
model.compile(loss=['mae'], optimizer='adam', metrics =['accuracy'])