Multi-task learning, finding a loss function that "ignores" certain samples I'm training a convolutional neural network with 1 input, and 3 outputs: a classification and 2 regression outputs.

I initially used 3 separate models, but I found that combining the 1st and 2nd loss functions into 1 model improves accuracy/rmse for both. I'm applying multi task learning.

Now I'm experimenting with incorporating the 3rd loss function into the same model with the first 2.

My challenge is that the 3rd loss function is only valid when the 1st classifier identifies a positive sample (it's a regression output that measures the identified sample). These labels are known at training time.

I attempted:


Standardizing the labels and impute a 0 value for the 3rd regression for all negative samples. That approach biases towards 0 in some cases, so I threw it out.
Alternate training between all samples on the 1st and 2nd loss, and only positive samples using the sum of all 3 loss functions. Mixed results here, but the 3rd loss function seems to perform worse than it does in its own separate model.


Question: 

I'm considering how I might re-structure the loss function (currently L2 loss) so that it treats a 0-value as irrelevant (e.g. no gradient is propagated when the label is 0).

Any thoughts?
I seem to have a workable solution using this minor tweak to square loss:

$L=(y-\hat{y})^2 \cdot \left(\frac{y}{y + \epsilon}\right)$

Where $y$ is the label and $\hat{y}$ is the prediction, and $\epsilon=1e-6$ is to avoid a divide by zero error (it's ~ the smallest number within 32 bit precision)

In tensorflow:

epsilon = 1e-6
loss = tf.reduce_sum(
          tf.mul(tf.square(prediction - label),
                 tf.div(label, label + epsilon))
)


The second term is 0 when the label is 0 (e.g. loss is 0 and no gradient is added for that sample), and it'll round to 1, or near enough, when the label is non zero. As long as the range of labels does not cross 0 this should work, and that's a simple matter of scaling the labels if it's not the case by default.