Machine Learning: Balanced training set but highly unbalanced prediction set? How to adjust? I am trying to train a model to detect gender in a dataset of CEO speeches. Here are the datasets that I have:


Final Dataset: 20K CEO voices analyzed (around 95% male) 
Testing dataset (?): 1K CEO voices analyzed from the final dataset, less unbalanced because I added more females on purpose (80% male)
Training dataset: 6K voices analyzed from audiobooks and TED talks (55% male).


For now, I have been trying different models by training and splitting dataset #3 (70%, 30%). I get good accuracy (95%) using this method. However, when I apply the trained model to dataset #2, I get an accuracy of 85%.

I am not sure what to do. Should I undersample women in the training dataset so that its distribution is more similar to the final dataset?
Over- or undersampling should be your second choice. Currently, the best method to deal with class-imbalances is to use the weights argument which sklearn- and keras-classifiers support (see for example the DecisionTreeClassifier).

On a general sidenote, I would recommend focusing on the f1-score,  AUC, and the confusion matrix to evaluate the model's performance, as accuracy might be not as informative in this case.