How to cross-validate a deep learning model for highly imbalanced datasets?
Accuracy is not a good indicator of success with imbalanced data. The accepted answer is correct: F1 score is commonly used. Other options include roc_auc_score (see here) and average_precision_score (see here), both defined through scikit-learn.

If you're using Keras, I would recommend using class_weights (note this will not work well if you have a multi-label problem, although there are some workarounds, for example here).