how can i compare two classification algorithms? all,
i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. clearly they are very close! how can i assess if they are statistically different?
i have used mcnemars test:
from mlxtend.evaluate import mcnemar_table
from mlxtend.evaluate import mcnemar
lgbm_pred = second_best.predict(x_test)
xg_pred = chosen_model.predict(x_test)

tb = mcnemar_table(y_target=y_test, 
                   y_model1=lgbm_pred, 
                   y_model2=xg_pred)

chi2, p = mcnemar(ary=tb, corrected=True)
print('chi-squared:', chi2)
print('p-value:', p)

output is: chi-squared: 2.25 p-value: 0.13361440253771584, so i would not reject null that models peformance are equal. (hopefully i am using this correctly so pls let me know if i am not.)
i have seen some threads on using 'permuation tests' etc.. but i am unsure how to interpret these and also i thought these tests were only if you have small sample size which i don't.
https://stackoverflow.com/questions/52373318/how-to-compare-roc-auc-scores-of-different-binary-classifiers-and-assess-statist
basically how can i assess which classifier being better? when in an ideal world i would want a classifer which is able to predict has cancer. can i compare precisions of model to model? what is best approach - does anyone also know how i can use permutation tests?
Performance of a model can be judged along several dimensions:

Accuracy - Is the train-test validation performing well?

Overfitting - Is the difference between training score and the validation score minimal?

Efficiency - Is the model light-weight, does it compute and calculate fast?

Complexity - Is the model easy to explain, does it use minimal transformations?

Deployability - Will I be able to deploy and use the model easily?


Rather than from a pure statistics perspective (which is just dimension 1 &amp; 2) judge the models from all perspecties. If they still perform equally well than they are in fact equal (not surprising for two similiar models performed on the same data set).