How to optimize the lambdas of a hybrid loss in a deep learning model I am using a generative adversarial deep learning model (GAN) with a hybrid loss represented by a linear combination of four losses with three $\lambda$'s, something like:

$total\_loss = loss_1 + \lambda_1\times loss_2 + \lambda_2\times loss_3 + \lambda_3\times loss_4$.

Is there a way to optimize these $\lambda$'s towards the best performance? provided that the computational complexity is not trivial. If my objective is for $loss_4$ to be minimum, should I use a high value for $\lambda_3$ compared to the other ones? But, how would that affect the overall performance, as all the models used in the GAN might play an important role in the final result.

NB. $loss_1$ is based on MSE measure, while the other losses are based on $l_1$ norm.
When there is an optimization problem involving more than one objective function to be optimized simultaneously, it is called multi-objective optimization.

Gradient descent methods can still be used to minimize the overall objective function. A paper entitled "Gradient-Based Multiobjective Optimization with
Uncertainties" goes into greater detail.

In PyTorch, autograd.backward can handle multiple objectives. You need to provide a list of tensors for backpropagation.