issue with early-stopping on f1 score with imbalanced data I have a highly imbalanced dataset with less than 0.5% of the minor class. Using Keras, I'm training DNN on the training set and evaluate performance on validation set. Loss function is binary_crossentropy. 

I'm setting my early-stopping on f1 score, instead of validation loss. What I observe during training is f1 score fluctuate wildly up and down while validation loss is decreasing. I actually end up with a very low f1 score with early-stopping, although f1 score was many epochs back...

I'm confused about it. Am I supposed to do early-stopping on the performance metrics? Should we alway use validation loss for early-stopping criterion? thanks.
F1 is based on hard classification; if the probability scores are hovering near the threshold, then the classifications may be flopping a lot, leading to unstable F1 scores.

A low F1 score is not too surprising in the presence of such imbalance; the default cutoff of 0.5 will often lead to high recall but low precision.