What is the appropriate statistical significance test for multi-class classification? I have a multi-class classification problem. I am primarily using macro-average F1 measure to evaluate the performance of models and want to verify if the results are statistically significant. I have the results of two classifiers on the same train/test-set (paired observations).
Some sources suggest to use McNemar’s test for binary classification task. However, is there any generalization of McNemar’s test for multi-class classification problem? If so, what would be the appropriate procedure to carry out these tests?
Generalisation of Mcnemars is called Cochran–Mantel–Haenszel test.
There is an implementation in R, but I suppose porting to Python should not be too hard. You can find the R version here.