Will it cause gradient vanishing by adding a constant positive and constant negative loss?
I want to ask a question about using two losses to train one model.
I am going to generate some specific kinds of question sentences.
To achieve it, I use (1) a normal GAN to generate the normal question space. Then  (2) an auxiliary classifier to let the generator focus on generating that kind of questions.
By pre-experiment, as I use BCELoss() on Generator-Discriminator loss (loss_G_D), the loss value is around 3.
And as I use -Entropy on Generator-Classifier loss (loss_G_C), the loss value is always negative, and very big, e.g. -300.
To not let them affect each other's training procedure, I used this training method.
optimizer.zero_grad()
loss_G_D = BCELoss(discriminator(generated_data))
loss_G_D.backward()
optimizer.step()

optimizer.zero_grad()
loss_G_C = -Entropy(classifier(generated_data))
loss_G_C.backward()
optimizer.step()

However, this training procedure is a little bit slow and seems like they put the network training 'back and forth'.
So a pal suggested me this method:
optimizer.zero_grad()
loss_G_D = BCELoss(discriminator(generated_data))
loss_G_C = -Entropy(classifier(generated_data))
loss = loss_G_D+loss_G_C
# if you worry about the scale, give some weight, like
# loss = loss_G_D+0.01*loss_G_C
loss.backward()
optimizer.step()

I thought it makes sense. However, like the loss_G_D is 3, loss_G_C is -300, wouldn't it cause gradient vanishing by add them up? As using loss = loss_G_D+0.01*loss_G_C=0 ? Or can I say because they are different type of loss considering always positive and negative, we should not add them up?
(PS. I think maybe map the -Entropy loss into an always positive activation function, then add the two loss up will work?)

No it won't. In your case it is the total loss that is &quot;vanishing&quot;, not the gradient. The gradient is simply the sum of the two separate gradients calculated from two losses. Since one loss is only enforced on classifier and the other one is only enforced on discriminator, back propagating the former should only assign gradients to classifier and back propagating the latter should only assign gradients to discriminator. So they won't affect each other.
Let me also explain with a simple example. Say you have only two parameters x and y. And you want to make x as small as possible by enforcing a loss L_x = abs(x). Meanwhile, you also want to make y as large as possible by enforcing L_y=-abs(y). So the total loss is actually L=abs(x)-abs(y). Suppose initially we have x=y=1. Then the gradient is (dL/dx,dL/dy)=(1,-1), while the loss is L=0.
More specifically, at (x,y)=(1,1), L_x=1 gives a gradient of (d(L_x)/dx,d(L_x)/y)=(1,0) and L_y=-1 gives a gradient of (d(L_y)/dx,d(L_y)/y)=(0,-1). You see that even though L_x and L_y cancel each other, their gradients don't.
