Classification: Dealing with unknown class balance in the prediction dataset I'm working on a classification problem that predicts if a grant application will be accepted. The data I'm training on is from 2005 to 2008. I'de like to predict any time after 2008.

The issue I'm running into is that the ratio of successful grant applications differs over time. For example, the success rate in the training set is 20%. But depending on the period I use after 2008, the success rate could be anywhere from 10 to 30%. This results in under or over predicting. 

Since the success rate of grant applications is not constant, how do I account for this in my model?
This phenomenon is called Covariate Shift and it can affect features,target, you name it.

This happens when the test data distribution is different from the training data. Or your problem is a time-series one where time is the biggest independent variable.

First case : No model can adapt to the target shifting in its behavior. You have to either wait until you collect enough data from the current times, and retrain a fairly satisfying model, or do online learning which means that every datapoint that you predict for , is fed into the model to be trained on. That way; if changes in the target occur, your model will get updated with every new datapoint.

Second case : Lets say your target variable has a tendency to go up every 2 years, and then goes down. This is an assumption i'm making. Your task is to detect that shift in time that affects your target and model it. You create a model that takes into account that every 2 years, your target goes down. and so on. It's what we call a time-series problem where time is the biggest contributor to the target's changes.

If the second case doesn't make sense to you, and doesn't relate to your problem, then look at first case.