Binary classification problem with imbalanced dataset, how to compare to random classifier We have a very imbalanced dataset (2% of class 1). To the best of our knowledge, there is no baseline in the literature to the problem we want to solve - so we thought of comparing our performance to a random classifier. We evaluate our model as a combination of precision and recall - we vary the threshold at which data points are classified as 1 and compute the rolling threshold and recall. We could use F1-score as well.

What would be an acceptable way to define a random predictor that we can compare to our model such that the comparison is as fair as possible?
Since you are interested in different decision thresholds, your random model should produce scores.  In that case, a reasonable base-line model assigns a score uniformly at random in $[0,1]$.  Such a model will, at threshold $t$, have 

$$\begin{align*}
\operatorname{precision} &amp;= \frac{2\%\cdot N\cdot (1-t)}{N(1-t)} = 0.02,\\[1em]
\operatorname{recall} &amp;= \frac{2\%\cdot N\cdot (1-t)}{2\%\cdot N} = 1-t.
\end{align*}
$$ 

(Perhaps a very simple model will serve as a better baseline.)