Issue while predicting multiple values which possess different order of magnitude (regression) I have a system that take electric signal and output parameters (regression). However I run into an issue. The parameters I want to predict are not on the same magnitude. I would like also to use only one neural networks if possible.

Parameter 1:
Mean    = 12.53673
Minimum = 10.00461
Maximum = 14.98899

Parameter 2:
Mean    = 148656955394038
Minimum = 75029133522564
Maximum = 224934092847235

Parameter 3:
Mean    = 1.475720134278e+17
Minimum = 7.506184799345e+16
Maximum = 2.249190781380e+17

...


If I use any loss function such as RMSE, MSE, MAE, MSLE the losses of the big parameters takeover all the rest. The network therefore randomly guess all the smaller parameters

I am searching a good way to find those parameters using one network. I tried using "mean absolute percentage error". But the learning is exceptionally slow, and it often get stuck into extremely low value and extremely high value.

I am maybe thinking about creating one loss per target. Freeze all the network except the output layer and each epoch let one of the loss train the whole network. But I am not sure it is a good idea.

Currently I am normalizing the target before predicting it, but I don't like this solution very much. I would prefer having a slightly more complicated architecture than normalizing and denormalizing (afraid of data leakage) and it seems more robust to directly output the right value.

Moreover In the future I will have an issue similar except I will need to predict series that possesses different order of magnitude (and not only some distinct value). So I hope being able to reuse the technique I am learning

Any help would be very appreciated!
Thanks :)

EDIT:
For those who happens to have the same issue, this is the code I went with (keras)

class Model:
    def __init__(self, shape_features, shape_targets, mean_targets, std_targets):
        self.shape_features = shape_features
        self.shape_targets = shape_targets
        self.mean_targets = mean_targets
        self.std_targets = std_targets

    def build(self):
        inputs = Input(shape=self.shape_features, name='Input')
        x = Conv1D(filters=16, kernel_size=3, activation='relu', name='Convolution1D_1')(inputs)
        x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_1')(x)
        x = Conv1D(filters=32, kernel_size=3, activation='relu', name='Convolution1D_2')(x)
        x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_2')(x)
        x = Conv1D(filters=64, kernel_size=3, activation='relu', name='Convolution1D_3')(x)
        x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_3')(x)
        x = Flatten(name='Flatten')(x)
        x = Dense(128, activation='relu', name='Dense_1')(x)
        x = Dense(64, activation='relu', name='Dense_2')(x)
        x = Dropout(0.2, name='Dropout')(x)
        outputs = Dense(self.shape_targets, activation='linear')(x)
        model = tf.keras.Model(inputs=inputs, outputs=outputs)

        def de_normalizing(tensor):
            return tensor * self.std_targets + self.mean_targets

        predictions = Lambda(de_normalizing)(outputs)
        model_prediction = tf.keras.Model(inputs=inputs, outputs=predictions)

        def custom_loss(y_true, y_pred):
            y_true = (y_true-self.mean_targets)/self.std_targets
            return K.mean(K.square(y_pred - y_true), axis=-1)

        return model, model_prediction, custom_loss


One model is used for training, the other is used for predictions
I am maybe thinking about creating one loss per target. Freeze all the
  network except the output layer and each epoch let one of the loss
  train the whole network. But I am not sure it is a good idea.


This is one option. It should work but it may be slower to converge as you only make use of only a third of your losses each epoch. Also, it is not completely unlikely that your losses may work "against each other". Meaning that they could somewhat cancel each other out, which can make convergence take longer.


  Currently I am normalizing the target before predicting it, but I don't like this solution very much. I would prefer having a slightly more complicated architecture than normalizing and denormalizing (afraid of data leakage) and it seems more robust to directly output the right value.


That's what I would do. I don't think you should be afraid of data leakage. Computers have enough precision in them that normalizing/denormalizing shouldn't create any issues. Also, keep in mind that computers have greater precision for floats close to 0. So normalizing your data may actually be good.

And when you say, I'd rather directly output the right value. You can do both! Your network could directly output the denormalized value, but the loss used to train the network can be computed on any layer before that.