Feature selection and classification accuracy relation One of the methodology to select a subset of your available features for your classifier is to rank them according to a criterion (such as information gain) and then calculate the accuracy using your classifier and a subset of the ranked features.

For example, if your features are A, B, C, D, E, and if they are ranked as follow D,B,C,E,A, then you calculate the accuracy using D, then D, B then D, B, C, then D, B, C, E... until your accuracy starts decreasing. Once it starts decreasing, you stop adding features.



In example1 (above), you would pick features F, C, D, A and drop the other features as they decrease your accuracy.

That methodology assumes that adding more features to your model increases the accuracy of your classifier until a certain point after which adding additional features decreases the accuracy (as seen in example 1)

However, my situation is different. I have applied the methodology described above and I found that adding more features decreased the accuracy up until a point after which it increases.



In a scenario such as this one, how do you pick your features? Do you only pick F and drop the rest? Do you have any idea why the accuracy would decrease and then increase?
feature selection involves several approaches just like methods for machine learning. Idea is to keep most relevant but not redundant feature for predictive model that can yield optimal accuracy.

In your case, I can not see which method you are using for feature selection but assuming that you are not taking account of multivariate nature of feature dependency. Say you have N features, likely reason that your model accuracy drops after n top feature(s) but improves by adding n+k (where n &lt; k &lt; N when features are in descending order based on information gain) is due to inter-dependency (more relevance and less redundancy) of top n and k features. Univariate feature selection does not necessarily get optimal model accuracy when features are inter-dependent and not mutually exclusive. From philosophical point of view, set of optimal features is analogous to a quote by Aristotle: "The whole is greater than the sum of its parts"!

For optimal feature selection, I often is Caret package in R language where one may do feature selection using recursive feature elimination (RFE) among several other approaches. There is also a package called mRMRe to do feature selection based on maximum relevance, minimal redundancy.

Best,
Samir