How to find optimal sample weights in binary classification/regression problem I'm training a model (NN) that gets some data as input and outputs a single value in the range of $[0, 1]$. Right now, the average of the outputs in my dataset is around 0.5, but I know that future data will largely consist of 0.0s, and thus there will eventually be a strong data imbalance towards 0.0. I want the training procedure to be future-proof and scalable, so I'm trying to find a way to automatically rebalance the dataset. My library (Keras) supports sample weights in training, which seems like a straightforward way to do this without losing any information.

Basically, I think what I'm looking for is a function $w(y_i) \rightarrow w_i$ that given a training example $y_i$ gives me a weight $w_i$, so that the weighted average of all training examples $Y$ with the weights $W$ is $0.5$. I know there are many configurations of weights that have this property, but of course the weights should be as close to 1 as possible and definitely $&gt; 0$. I also realize that this is not possible for cases where e.g. all numbers are the same, or all are $&lt; 0.5$. But let's assume my data is diverse enough.

I'm sure I'm not the first one to think of this, but I can't find any solutions / best practices for this case. I guess I could treat it as a little optimization problem of its own, but I hope there is something simpler.
There are a few things which are unclear so I am going to have to make some assumptions. 

You say it is binary classification/regression. Are you trying to find the probability of each binary class?

When you say "future data", do you mean future training data or future test data? If it is test data then you do not need to worry at all about his effect. If it is future training data then yes you may have a problem. You do not want to do model building on a training set that is different from the training set you would actually use in the productionized model. I would suggest you produce a number of validation plots for this case.

If you really are doing binary classification. Most models in python have a method (eg predict_proba() ) that will produce a probability and will let you decide on class based on that probability. The default being 50%. For this reason I do not think that you want to have the weighted average be 0.5 but the "probability" represent the true probability. Please refer to this tutorial http://scikit-learn.org/stable/modules/calibration.html

As to your original question. You want the weights to be $M/N_i$ where $N_i$ is the number of samples in each class $i$ and $M$ the total number of samples. This is a way to do some sort of pseudo-upsampleing and have the classes be weighted by the number of samples.