Is Recursive Feature Elimination finding best features subset? On a set of 9 features I have applied Recursive Feature Elimination (RFE) algorithm using SVM estimator, following approach from (1). When requesting a subset of size 1 to be found, then RFE returned feature X.
However, when I trained SVM over each feature individually, I found another feature Y to have higher accuracy than SVM trained over X.
I thought that RFE finds features with the highest accuracy.
Is my understanding of RFE wrong?
(1): Gene selection for cancer classification using support vector machines
No, RFE cannot guarantee that it finds the feature subset with optimal score.
As with most greedy processes, the point of RFE is to reduce the computational cost (fitting a model for each of the $2^m$ feature subsets), at the cost of perhaps not finding the actual optimum (but hopefully &quot;close enough&quot;).
See also https://stats.stackexchange.com/questions/232461/question-about-recursive-feature-elimination