Multi class text classification with imbalanced data
It is straightforward to combine k-fold cross-validation with a technique like oversampling or undersampling. 

First, apply the balance-restoration technique to your training data. Then parametrize a model using k-fold cross-validation on the re-balanced training data. In Scikit learn, I believe you can even bundle these actions together into a single 'pipeline' object to make it easier to manipulate.

Precision/recall is probably a fine starting place for measuring performance.