Determine useful features for machine learning model I am working with a dataset with hundreds of features. I wish to create a simple machine learning model using 7-10 features from the original dataset. My question is this:

What quantitative metrics can I use to determine that a feature will be useful to the learning model?

I have been comparing the distribution of the target mean over the feature groups, to the target mean of the dataset as a whole. For example, take a binary feature X and a binary target. Let's say the target has a mean of 0.10 when taken over the entire dataset.

To analyze the feature X, I take the target mean for each group within feature X.

mean (X=0) = 0.07
mean (X=1) = 1.15


In this way, I can observe the effect of a feature on the target.

I know that there must be some stronger metrics which people use to determine the strength of a feature. In school I used p tests to determine the statistical significance of a variable. Is there an analog in DS/ML?
I suggest taking a look at this page for some more ideas:

Feature Selection

That being said a couple of ideas that come to mind quickly, is to:


use a tree based method (like Random Forest) and look at your feature importances. Scikit Learn has a handy class for doing just that see the link above.
Use some sort of regularization/penalty like L1 or L2 regularization. That will force non-useful features to have parameters close to zero.
Recursively remove variables and see what the resulting output is and cross-validate. Again sklearn has a method for this.


Generally, these methods will be "expensive" as you are fitting multiple models to get you where you need to go.