How to stop a neural network from regressing to the mean I'm very familiar with neural networks for classification, but I'm trying a regression task for the first time. I'm finding that the network tends to go towards guessing a mean for the whole dataset rather than making case specific predictions.
What are the potential reasons for this behaviour and how can I stop it?
Hint
In my specific case this behaviour is present in both the training and validation set. The training set is very small, and it's questionable as to whether the inputs have a strong correlation with the target.
Also might be worth noting that I'm training on images. So I'm using a CNN.
Update
I did some inspection of my model and found that it drives the output of my CNN backbone to 0 and simply uses the single bias on my fully connected layer to produce the output. I also found that if I freeze the bias during training, I can increase the learning rate indefinitely without the loss blowing up...
The regression tasks are not very different from classification and the behavior you faced with is probably due to a bug in the code.
If your training set is small and the network is comparably large, it should overfit on it (correlation with inputs doesn't matter, there was a paper https://arxiv.org/pdf/1611.03530.pdf that shows you can randomly shuffle imagenet labels and train one of the version of resnet on it and get perfect accuracy on that training set). Sample agnostic predictions mean underfitting. You can also try to overfit a small batch and get a zero loss. If you won't be unable to do this, there's a bug, definitely. Possibly your gradients affect the latter layers much more than the first layers, which means after some epochs of training you still have random outputs from the first layer and it's no wonder the net is trying to predict the mean of the dataset. This can happen if your net is very deep and you're not using residual connections. But it's not the case for a net of 10 (for instance) or less layers. Maybe your learning rate is too high and your first layers can't be tuned correctly. To check whether your first layers extract some meaningful information, try to look at the outputs of some of your first layers. If those outputs look like a complete noise, double check the code for optimization. You can find more interesting cases of bugs and the behavior of a buggy code in Andrej Karpathy's blog http://karpathy.github.io/2019/04/25/recipe/