More features hurts when underfitting? I was training a binary classifier using XGBClassifier (basically boosted decision trees if I understand it correctly). I have 10K training examples. I have two distinct set of features (but they could be dependent), one contains 26 features (call the set A) and the other contains 96 features (call the set B). I tried to train 3 classifiers, each w/ a different combination of the feature sets, namely A, B and A+B. The result is that using only A is clearly better than using both A and B. At this point I thought it might be overfitting, so that using less number of features actually avoids the overfitting.

The # of trees used in the above trainings was 100. So I used 10-fold cross validation to find the optimal # of trees for each feature set combination, and they are all beyond 100 (like 300-500). So it seems to me the models were learning w/ less-than-desired degree of freedom and they were underfitting if you will.

So this confuses me: why providing additional features make things worse (i.e. using only A is better than using both A and B) when the models are on the under-fitting side (as opposed to overfitting)?

Or maybe a more general question: how do I find out the real problem in this situation?

Note: I have read this question about overfitting/underfitting and I still think my models should be underfitting, because apparently increasing model complexity helps (e.g. from 100 trees to 300-500 trees).
Theoretically speaking, more data leads to better model. However, in practice, more features often leads to the difficulty of model training.

Assuming there are 30 "main" features of your dataset. Feature set A contains 20 "main" features, so it might be easy (20 out of 26) for one "main" feature to be "chosen" and "trained", under certain hyperparameters (in your case, 100 trees). When it comes to feature set B, which contains all "main" features, it's hard (30 out of 96) for one "main" feature to be chosen, and it's harder when there's only 100 trees (cause there are 66 "minor" features which should not be trained, relatively). That's what we called "under-fitting".

Back to your question, when the model is under-fitting, if we're lucky that model trained on feature set B (namely model B) contains all these 30 "main" features, the model will be good, and we might not able to find out it's under-fitting. But in most cases, we're not that lucky, model can be ruined with all 66 "minor" features, relatively.

In my ML practice, I'll try more training iterations when more samples come, and try a more complex model when more features come, with the control of over-fitting/under-fitting.