How to train and evaluate machine learning models with growing/changing datasets over time Assume that you have a classification machine learning model, and you start with an initial dataset that contains 3 classes. You split the initial dataset into training/testing spits, you train the initial model and evaluate it, and then overtime, you collect more data for your dataset. Now you have more data that you want to add to you initial training dataset. The question is: How do you you organize you dataset and model training regiment so you can effectively quantify possible improvement between the initial model and the new model?
One possible solutions: if you split the initial training dataset into training/testing data, once you have more data, you just split that data and then add the respective new training/testing splits into the original splits. But this approach feels a little too simplistic, and it could potentially lead to drift in the dataset itself.
This is similar to doing regression testing but for machine learning models, but I am not sure im using the correct technical jargon for it.
The solution you described is definitely an option.
The other motivation for retraining is that the distribution of new data has drifted away from that of the original dataset and that warrants retraining to maintain the same level of model performance.
In such situations, It is possible that you may actually throw away some of your older training data because they may have become stale and include a part of newly available data for training. The remainder new data can then be used for out of sample testing and validation.
You also want to be able to figure out when to trigger the retraining. This is usually done by an independent drift detection process, which keeps track of some metric related to your Business and triggers retraining if it dips below a predefined threshold.