Kappa From Combined Confusion Matrices I am trying to evaluate and compare several different machine learning models built with different parameters (i.e. downsampling, outlier removal) and different classifiers (i.e. Bayes Net, SVM, Decision Tree).

I am performing a type of cross validation where I randomly select 67% of the data for use in the training set and 33% of the data for use in the testing set. I perform this for several iterations, say, 20.

Now, from each iteration I am able to generate a confusion matrix and compute a kappa. My question is, what are some ways to aggregate these across the iterations? I am also interested in aggregating accuracy and expected accuracy, among other things.

For the kappa, accuracy, and expected accuracy, I have just been taking the average up to this point. One of the problems is that when I recompute kappa with the aggregated average and expected average, it is not the same with the aggregated kappa.

For the confusion matrix, I have been first normalizing the confusion matrix from each iteration and then averaging them, in an attempt to avoid an issue of confusion matrices with different numbers of total cases (which is possible with my cross validation scheme).

When I recompute the kappa from this aggregated confusion matrix, it is also different from the previous two.

Which one is most correct? Is there another way of computing an average kappa that is more correct?

Thanks, and if more concrete examples are needed in order to illustrate my question please let me know.
I think most people would simply average a statistic like accuracy or kappa over the several runs, rather than compute one overall statistic.

While it's not necessarily true that an average (weighted by the test set size in each case) of these statistics equals the statistic computed over all of the runs at once, it happens to be true for accuracy and kappa. Hence it shouldn't matter.

I think the reason you may be getting a difference is that you're normalizing away the test set size. I don't think you should.