Kappa From Combined Confusion Matrices
I think most people would simply average a statistic like accuracy or kappa over the several runs, rather than compute one overall statistic.

While it's not necessarily true that an average (weighted by the test set size in each case) of these statistics equals the statistic computed over all of the runs at once, it happens to be true for accuracy and kappa. Hence it shouldn't matter.

I think the reason you may be getting a difference is that you're normalizing away the test set size. I don't think you should.