Object detection is not improving although loss is decreasing
So the question asks why you are seeing a decrease in the loss function (for both training and validation?), but you are also observing decreasing generalisation performance over iterations.
One first thought could be due to the loss function that you have chosen might not be appropriate for your task.