regression network that will be optimised on a subset of the data I am trying to optimise my network that is trying to perform regression. Currently, the dataset is dominated by values in a certain range. It is very good at predicting these values. I want to try to find a way to make it focus on the values outside of this bulk, these values outside the bulk are much higher. We thought about splitting the dataset into parts, and training multiple networks, but we really want the regression to be done by a single network.
An idea that we had was to adapt how the loss function calculated loss as we are training. So, after a certain number of epochs passed, or after a certain accuracy was achieved, we would only pass the loss function values that were being very poorly predicted. The motivation was that this may improve the performance of the network as training would have to focus on these points.
I have not found much literature, or examples, trying something like this out. Although, it seems like it could work. Naively, it seems like it's in the same vein as weighting a loss function, or when you perform transfer learning and retrain just the final few layers of the network to make it specific to your problem.
My questions:

Generally, how does this sound to people? Does anyone have any opinions on it or ideas on how to improve it?
Is there any work been done doing something similar that people know of?
Your reasoning sounds similar to curriculum learning: the idea is to pass training data to the network not randomly, but based on some scoring function $f$. It was proven that a network learns faster and better, even though the process itself is not well understood. I think it is a better approach than the one you suggested: I am afraid that, if you pass only poorly predicted data after a while, the network will forget what learned before.
What you can also do is to perform data augmentation to compensate your unbalanced data (if I understood well, higher ranges in your dataset are not well represented compared to a certain range).