title: How to perform feature selection with gridsearchcv in sklearn in python
body: <p>I am using <code>recursive feature elimination with cross validation (rfecv)</code> as a feature selector for <code>randomforest classifier</code> as follows. </p>
  
  <pre><code>X = df[[my_features]] #all my features
  y = df['gold_standard'] #labels
  
  clf = RandomForestClassifier(random_state = 42, class_weight="balanced")
  rfecv = RFECV(estimator=clf, step=1, cv=StratifiedKFold(10), scoring='roc_auc')
  rfecv.fit(X,y)
  
  print("Optimal number of features : %d" % rfecv.n_features_)
  features=list(X.columns[rfecv.support_])
  </code></pre>
  
  <p>I am also performing <code>GridSearchCV</code> as follows to tune the hyperparameters of <code>RandomForestClassifier</code> as follows.</p>
  
  <pre><code>X = df[[my_features]] #all my features
  y = df['gold_standard'] #labels
  
  x_train, x_test, y_train, y_test = train_test_split(X, y, random_state=0)
  
  rfc = RandomForestClassifier(random_state=42, class_weight = 'balanced')
  param_grid = { 
      'n_estimators': [200, 500],
      'max_features': ['auto', 'sqrt', 'log2'],
      'max_depth' : [4,5,6,7,8],
      'criterion' :['gini', 'entropy']
  }
  k_fold = StratifiedKFold(n_splits=10, shuffle=True, random_state=0)
  CV_rfc = GridSearchCV(estimator=rfc, param_grid=param_grid, cv= k_fold, scoring = 'roc_auc')
  CV_rfc.fit(x_train, y_train)
  print(CV_rfc.best_params_)
  print(CV_rfc.best_score_)
  print(CV_rfc.best_estimator_)
  
  pred = CV_rfc.predict_proba(x_test)[:,1]
  print(roc_auc_score(y_test, pred))
  </code></pre>
  
  <p>However, I am not clear how to merge feature selection (<code>rfecv</code>) with <code>GridSearchCV</code>.</p>
  
  <p><strong>EDIT:</strong> </p>
  
  <p>When I run the answer suggested by @Gambit I got the following error: </p>
  
  <pre><code>ValueError: Invalid parameter criterion for estimator RFECV(cv=StratifiedKFold(n_splits=10, random_state=None, shuffle=False),
     estimator=RandomForestClassifier(bootstrap=True, class_weight='balanced',
              criterion='gini', max_depth=None, max_features='auto',
              max_leaf_nodes=None, min_impurity_decrease=0.0,
              min_impurity_split=None, min_samples_leaf=1,
              min_samples_split=2, min_weight_fraction_leaf=0.0,
              n_estimators='warn', n_jobs=None, oob_score=False,
              random_state=42, verbose=0, warm_start=False),
     min_features_to_select=1, n_jobs=None, scoring='roc_auc', step=1,
     verbose=0). Check the list of available parameters with `estimator.get_params().keys()`.
  </code></pre>
  
  <p>I could resolve the above issue by using <code>estimator__</code> in the <code>param_grid</code> parameter list.</p>
  
  <hr>
  
  <p>My question now is <strong>How to use the selected features and parameters in <code>x_test</code> to verify if the model works fine with unseen data. How can I obtain the <code>best features</code> and train it with the <code>optimal hyperparameters</code>?</strong></p>
  
  <p>I am happy to provide more details if needed.</p>