Regression and Classification in one Neural network
Yes, these type of loss functions can be optimized using backpropagation, also in Tensorflow.
The value of the loss is a scalar (same as just the cross entropy, or the MSE, otherwise you wouldn't be able to add them), which means that it doesn't really work any different from just optimizing for any other scalar loss function. As long as the operations involved in calculating the loss function are differentiable (&quot;smooth&quot;) enough, Tensorflow (or any other framework that does automatic differentation) doesn't care.
Think of it this way: calculating any interesting loss function involves summing up a bunch of terms living in a higher-dimensional vector space. In this case, for some of the directions in this vector space, you apply a different function on them than for others in that vector space. Doesn't really matter to Tensorflow.
What does matter though is how you sum them: since the cross entropy and the MSE are not working on the same type of units (think dimensional analysis), you have to determine some sort of scale between them. You can view this as a hyperparameter, where you can chose how important it is to have the classification correct vs how correct your bounding boxes are.