Measuring performance of different classifiers with different sample sizes I'm currently using several different classifiers on various entities extracted from text, and using precision/recall as a summary of how well each separate classifier performs across a given dataset.

I'm wondering if there's a meaningful way of comparing the performance of these classifiers in a similar way, but which also takes into account the total numbers of each entity in the test data that's being classified?

Currently, I'm using precision/recall as a measure of performance, so might have something like:

                    Precision Recall
Person classifier   65%       40%
Company classifier  98%       90%
Cheese classifier   10%       50%
Egg classifier      100%      100%


However, the dataset I'm running these on might contain 100k people, 5k companies, 500 cheeses, and 1 egg.

So is there a summary statistic I can add to the above table which also takes into account the total number of each item? Or is there some way of measuring the fact that e.g. 100% prec/rec on the Egg classifier might not be meaningful with only 1 data item?

Let's say we had hundreds of such classifiers, I guess I'm looking for a good way to answer questions like "Which classifiers are underperforming? Which classifiers lack sufficient test data to tell whether they're underperforming?".
You need to look at the confidence interval of the statistic.  This helps measure how much uncertainty in the statistic, which is largely a function of sample size.