How to cross-validate a deep learning model for highly imbalanced datasets? I am working with a multi-modality classification problem (with Keras). I have 1000, 5000 and 10000 samples for three different classes. I would like to do a five fold cross-validation to select the best pre-trained deep learning model for deployment. I'm including class weights during model training to give more weightage for less-pronounced classes. For a given fold, I would be validating with 200, 1000 and 2000 samples from these three classes. Is accuracy a good prediction measure to be used in this case? Or do I have to measure the F1-score and Matthews correlation coefficients as well? Am I doing it right?
Accuracy is not a good indicator of success with imbalanced data. The accepted answer is correct: F1 score is commonly used. Other options include roc_auc_score (see here) and average_precision_score (see here), both defined through scikit-learn.

If you're using Keras, I would recommend using class_weights (note this will not work well if you have a multi-label problem, although there are some workarounds, for example here).