How do you effectively predict the top 20% most likely customers to churn from a dataset?
If you use k-fold cross validation, then every customer in your dataset will lie in the test set of one of the k folds. That might help alleviate the issue at hand. Finally you can train your model against your entire train set and run inferences against net new customers.