ZeroR as baseline for balanced / imbalanced datasets I have a dataset containing thousands of text posts. I am building a binary classifier that will classify posts as safe (0) or  risky (1). I randomly picked some of them and manually labeled. Label 1 is minority.

Imbalanced data results in skewed performance. To overcome this, I undersample to obtain a 50%/50% distribution for both train and test set.


If I use stratified strategy (in scikit-learn) for ZeroR, it acts like random guesser. Is this a good baseline?
If I use "guess 1" all the time, then its recall is 100% all the time. However recall is the most important metric for me. So other algorithms' predictions look unsuccessful. Should I ignore recall when using ZeroR and only compare accuracy?
I also would like to apply 10-fold cross validation, should I apply undersampling before splitting as well?
Or instead should I go with F1-Score without undersampling? This time ZeroR has high F1 score due to 100% Recall.
You should only undersample the data that is used for training. Test data should represent the true distribution.

Then you have at least two possible baselines. One is ZeroR (your second point), which always predicts the majority class. Given a true distribution of (0) 10% and (1) 90%, this would give you a recall of 100% for (1) and 0% for (0) and 90% accuracy. 

If rows represent the true labels and the columns the predictions, the confusion matrix looks something like this:

$\begin{bmatrix}90 &amp; 0\\10 &amp; 0\end{bmatrix}$

The other would be to flip a coin with probability 90% for class (1) (your first point but without undersampling), which gives you a recall of 90% for (1) and 10% for (0), and 82% accuracy, which is a little more balanced:

$\begin{bmatrix}81 &amp; 9\\9 &amp; 1\end{bmatrix}$

If your classes are strongly imbalanced then recall and accuracy have the same problems as you noted above. F1 might be a better choice, but the best metric depends on your application.