Poor performance of regression model for imbalanced data I am trying to train a neural network model to solve a regression problem. The specificity of my dataset is that it has something like an exponential distribution of target values (imbalanced). Therefor, the model seems to output just values less than 2 (if the range is [0,6]), for instance, and it absolutely ignores bigger target's values, which have a smaller performance in the dataset. How is it possible to improve the model's results in such a case?
For example, when it comes to a multiclass classification, we can weigh penalties for errors on a smaller class to enhance performance with imbalanced data. Are there any tricks in terms of regression? Which loss-functions could be useful? It seems, that MSE loss function is more preferable than RMSE. Is it more powerful loss functions for this problem?
There is a paper about such a problem of imbalanced regression (http://proceedings.mlr.press/v74/branco17a/branco17a.pdf) which might be helpful to someone. However, I'm more interested in special tricks for a neural network, not pre-processing approaches (I can't generate more data for example).
Neural Networks can in general be interpreted as a regression problem and as such, you could apply well known ways of dealing with this. This paper gives you a good introduction to different approaches. For instance you can upsample the minority class, or you could do loss weight balancing during training.

For instance, consider the data point $x_i$ that can belong to one of two classes $a$ and $b$. For instance, class $a$ is here the minority. Then you would, during training, multiply the calculated loss with the weight. $$L_i=\begin{cases}l_a\cdot x_i,&amp;\text{if datapoint $x_i$ is in the minority class}\\l_b\cdot x_i,&amp;\text{if datapoint $x_i$ is in the majority class}\end{cases}$$ with $l_a$ > $l_b$ which would be natural for this example.

You could also look into continous performance measures that could work nicely with imbalanced dataset. For instance the generalized dice (F1) score.