Why is XGBClassifier in Python outputting different feature importance values with the same data across different repetitions? I am fitting an XGBClassifier to a small dataset (32 subjects) and find that if I loop through the code 10 times the feature importances (gain) assigned to the features in the model varies slightly.
I am using the same hyperparameter values between each iteration, and have subsample and colsample set to the default of 1 to prevent any random variation between executions. I am using the scikit learn feature_importance_ function to extract the values from the fitted model.
Any ideas as to why this variation in feature importance could be occurring? Does this mean that some of my features may be correlated and is there a way to ensure the XGBoost outputs the same importance values each time it is called? Note that the prediction and the predicted probabilities are all constant across iterations: it is just the feature importances.
Thanks in advance!
Look at the 'max_features' parameter in the GradientBoosting Classifier in scikit learn, here: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier
That means that the algorithm is fitting different trees to try and account for the residual information after each sequence of trees, and that the features for each tree can be randomly permuted based on this parameter's selection.
Scikit learn does say that this can be made deterministic by setting random_state...