Techniques to manage high dynamic range in neural network regression?
You may try cycle GANs.
In normal Generative Adversarial Networks (GANs), a generative network -the generator- is trained to generate images with an auxiliary network, the discriminator. The discriminator learns to tell apart data generated by the generator (i.e. fake data) from real data, while the generator learns to generate data that fools the discriminator. GANs are known to generate good quality data where MSE and the likes generate &quot;blurry&quot; data that lack detail.
Cycle GANs are two coupled GANs, where the generators learn to &quot;convert&quot; the input data from domain A to domain B, and vice versa, plus an extra loss term to ensure cyclic consistency (i.e. generator1(generator2(a)) = a).
The results are very impressive. Here you can see cycle GANs trained to convert images, changing horses into zebras and vice versa, winter landscape into summer, etc:

If your data is continuous and your architecture is fully convolutional, then cycleGANs may actually work in your case.