"Optimal range for loss function": Myth or Truth? I am currently working on a regression problem using a deep neural network that given a volume 32x32x256 in input need to generate a second volume of the same dimensions in output, this is not a segmentation problem and the volume generated by the net is formed by floats (to be more specific, im tryng to remove noise from a noisy x-ray beam).
As a loss function i am using the MSE beetwen the volume generated by the net and a ground volume already calculated without noise.
The volumes have been normalized on values between 0 and 1, before giving them to the net.
When we started the first trainings we were getting very low values in MSE (about 3*10^-4), but the deep-net was converging (on values near 5*10^6).
One of my co-worker stated that those values (of the loss) were far to low to get an optimal result with a framework as Keras adn Adam as optimizer, and proposed to change the loss function to get result within a range of 100 - 0.01, he proposed to do the sum of the square difference instead of the mean.
At first i was really confused by his statement, but a did a test:
Instead of normalizing the volumes on values between 0 and 1 i normalized them between values 0 and 1000, before giving them to the net.
In this test the MSE went from values of about 30'000 and coverged of values near 600, after denormalizing the data this new neural network gave result much more accurate of the first network, and just by changing the starting normalization!
Do you have any idea of the cause of this behaviour, was my coworker right?
Very interesting statement you got there, i was not aware of this 'optimal loss value'.
the only way i could explain it is with the gradient computation : gradient is computed using :
$Grad = loss*lr$
So to me, changing the loss range is the same as changing the learning rate of your optimizer, it could be a good experiment to increase your lr instead of increasing the loss function range and see what happens to the training. I'm not aware of the subtilties of Adam optimizer so my explanation may be completly wrong.