have huge influence on how models perform on given dataset this is particularly true for less complex models like linear models and naive bayes models tree-based modelson the other handare often able to discover important interactions themselvesand don' require transforming the data explicitly most of the time other modelslike svmsnearest neighborsand neural networksmight sometimes benefit from using binninginteractionsor polynomialsbut the implications there are usually much less clear than in the case of linear models automatic feature selection with so many ways to create new featuresyou might get tempted to increase the dimensionality of the data way beyond the number of original features howeveradding more features makes all models more complexand so increases the chance of overfitting when adding new featuresor with high-dimensional datasets in generalit can be good idea to reduce the number of features to only the most useful onesand discard the rest this can lead to simpler models that generalize better but how can you know how good each feature isthere are three basic strategiesunivariate statisticsmodel-based selectionand iterative selection we will discuss all three of them in detail all of these methods are supervised methodsmeaning they need the target for fitting the model this means we need to split the data into training and test setsand fit the feature selection only on the training part of the data univariate statistics in univariate statisticswe compute whether there is statistically significant relationship between each feature and the target then the features that are related with the highest confidence are selected in the case of classificationthis is also known as analysis of variance (anovaa key property of these tests is that they are univariatemeaning that they only consider each feature individually consequentlya feature will be discarded if it is only informative when combined with another feature univariate tests are often very fast to computeand don' require building model on the other handthey are completely independent of the model that you might want to apply after the feature selection to use univariate feature selection in scikit-learnyou need to choose testusually either f_classif (the defaultfor classification or f_regression for regressionand method to discard features based on the -values determined in the test all methods for discarding parameters use threshold to discard all features with too high -value (which means they are unlikely to be related to the targetthe methods differ in how they compute this thresholdwith the simplest ones being selectkb estwhich selects fixed number of featuresand selectpercentilewhich selects fixed percentage of features let' apply the feature selection for classification on the representing data and engineering features