How to stop a neural network from regressing to the mean
The regression tasks are not very different from classification and the behavior you faced with is probably due to a bug in the code.
If your training set is small and the network is comparably large, it should overfit on it (correlation with inputs doesn't matter, there was a paper https://arxiv.org/pdf/1611.03530.pdf that shows you can randomly shuffle imagenet labels and train one of the version of resnet on it and get perfect accuracy on that training set). Sample agnostic predictions mean underfitting. You can also try to overfit a small batch and get a zero loss. If you won't be unable to do this, there's a bug, definitely. Possibly your gradients affect the latter layers much more than the first layers, which means after some epochs of training you still have random outputs from the first layer and it's no wonder the net is trying to predict the mean of the dataset. This can happen if your net is very deep and you're not using residual connections. But it's not the case for a net of 10 (for instance) or less layers. Maybe your learning rate is too high and your first layers can't be tuned correctly. To check whether your first layers extract some meaningful information, try to look at the outputs of some of your first layers. If those outputs look like a complete noise, double check the code for optimization. You can find more interesting cases of bugs and the behavior of a buggy code in Andrej Karpathy's blog http://karpathy.github.io/2019/04/25/recipe/