Balancing data-sets for regression problems Unbalanced data-sets are a well described problem for classification-problems.

However, for regression similar problems can arise. An example is the data-set where target variable has a very irregular histogram.



When fitting a model on such data, the main focus will be on the peak around the value 120. However, I want to create a model which works over the entire interval from 40 to 160. 


How would you typically approach these problems?
Are you aware of any literature regarding this topic?
Is balancing the data-set a good approach and how can this be implemented?
First, the histogram you plotted is that of the marginal distribution. The residuals might look perfectly fine once you regress on the variables. Linear regression is an estimation of Y conditional on the variables X; thus, in order to determine whether the distribution is a good fit, you examine the residuals which are assumed to be gaussian.

If you have data that is bounded between an interval, one approach would be modeling it using a beta regression. If the data is constrained to be positive, assuming a gamma distribution could work as well. You must, however, check the diagnostic plots afterwards to ensure that the distribution assumptions that you are making are correct.

See Beta Regression by Ferrari, A Better Lemon Squeezer by Smithson '06, and Extending the General Linear Model by Faraway. Also, for Bayesian implementations of regression models, you could look at Gelman's book.