How to evaluate performance of a new feature in a model? I am working on a binary classification where I have 4712 records with Label 1 being 1554 records and Label 0 being 3558 records.

When I tried multiple models based on 6,7 and 8 features, I see the below results. Based on the newly added 7th or (7th &amp; 8th) feature, I see an AUC improvement only in one of the models (LR scikit and Xgboost).

I also come across articles online that says AUC or F1-score aren't strict scoring rules. We could use log-loss metric but it's only applicable for logistic regression. but we can't use log-loss metric for Xgboost or RF or SVM right? So, is there any common metric which I can use to compare. Am I missing something here?



Does this mean that new feature is helping us improve the performance? But it decreases the performance in other models?

Please note that I split the data into train and test and did 10 fold CV on train data.

So, how do I know that this newly added 7th feature is really helping in improving the model performance?

update based on answer

from statsmodels.stats.contingency_tables import mcnemar
# define contingency table
table = [[808,138],    # here I added confusion matrix of two models together (I mean based on TP in model 1 is added with TP in model 2 etc)
         [52, 416]]
# calculate mcnemar test
result = mcnemar(table, exact=True)
# summarize the finding
print('statistic=%.3f, p-value=%.3f' % (result.statistic, result.pvalue))
# interpret the p-value
alpha = 0.05
if result.pvalue &gt; alpha:
    print('Same proportions of errors (fail to reject H0)')
else:
    print('Different proportions of errors (reject H0)')
Informally democracy.

So how many classifiers did it improve, only 1 than dont add it.

Formally there are a couple of statistical tests.

Cochran's Q test

Is a generalisation of the McNemars test for comparing Machine Learning models.

or read this formal paper where they discuss it.