title: Using make_scorer in GridSearchCV to score based on recall on postive class
body: <p>I am trying to use <code>Sklearn</code>'s <code>GridSearchCV</code> for a binary classifcation problem (originally a multi-class problem, but I was having issues as asked about <a href="https://stackoverflow.com/questions/71847809/scoring-gridsearchcv-based-on-the-recall-of-one-or-more-target-classes-but-not-o">here</a>, so I decided to simplify to a binary target). In short, I have observations that are either &quot;cancer&quot; or &quot;not cancer&quot;. The goal is to be able to correctly classify some observations as certainly not cancer so as to reduce the number of cases professionals would have to look over. It is clearly more important to make sure all &quot;cancer&quot; cases are caught than to worry about if some &quot;not cancer&quot; cases are accidentally classed as &quot;cancer&quot;, so I am passing <code>GridSearchCV</code> a scorer that has <code>score=recall_score</code>, <code>average='binary'</code>, and <code>pos_label='cancer'</code>. In my understanding, this will make it choose the best model (e.g find the best parameter set) based on the percent of cancerous observations that are caught. However, running the model with the best parameters as chosen by <code>GridSearchCV</code> produces recall on the postive class that is actually the same or even worse that that found with the defualt parameters. This is true for multiple models. Here is the code:</p>
  <pre><code>## Model 1: logistic regression
  
  print('\nLogistic Regression')                                                                                     
  mod = LogisticRegression(penalty='l2', max_iter=10000, solver='lbfgs')
  mod.fit(X_train, y_train)
  y_predict = mod.predict(X_test)
  test_precision, test_recall, test_fscore, _ = precision_recall_fscore_support(y_test, y_predict, average='binary', pos_label='unhealthy') 
  print('With default params, test recall on the postive label: ', test_recall) 
  
  kf = KFold(shuffle=True, random_state=72018, n_splits=3);                                                       
  mod = LogisticRegression(penalty = 'l2', max_iter=10000, solver='lbfgs')                                                                  
  estimator = Pipeline([(&quot;mod&quot;, mod)])
  params = {'mod__C' : np.linspace(0.1, 0.5, 10)}
  recall_scorer = make_scorer(recall_score, average='binary', pos_label='unhealthy')
  best_model = GridSearchCV(estimator, params, cv=kf, scoring=recall_scorer)                                                                                                                                               
  best_model.fit(X_train,y_train)                                                                          
  best_final_score = best_model.best_score_ ; best_final_params = best_model.best_params_ 
  print('Using GS, best params are: ', best_final_params)                                    
  y_predict = best_model.predict(X_test)  
  test_precision, test_recall, test_fscore, _ = precision_recall_fscore_support(y_test, y_predict, average='binary', pos_label=&quot;unhealthy&quot;) 
  print('Using these params, test recall on the postive label: ', test_recall)
  
  ## Model 2: KNN
  
  print('\nKNN')
  mod = KNeighborsClassifier()
  mod.fit(X_train, y_train)
  y_predict = mod.predict(X_test)
  test_precision, test_recall, test_fscore, _ = precision_recall_fscore_support(y_test, y_predict, average='binary', pos_label='unhealthy') 
  print('With default params, test recall on the postive label: ', test_recall)
  
  kf = KFold(shuffle=True, random_state=72018, n_splits=3);                                                       
  mod = KNeighborsClassifier()                                             
  estimator = Pipeline([(&quot;mod&quot;, mod)])
  params = {'mod__n_neighbors': np.linspace(1, 30, 30, dtype=int)} 
  recall_scorer = make_scorer(recall_score, average='binary', pos_label='unhealthy')
  best_model = GridSearchCV(estimator, params, cv=kf, scoring=recall_scorer)                                                                                                                                            
  best_model.fit(X_train,y_train)                                                                          
  best_final_score = best_model.best_score_ ; best_final_params = best_model.best_params_ 
  print('Using GS, best params are: ', best_final_params)                                    
  y_predict = best_model.predict(X_test)  
  test_precision, test_recall, test_fscore, _ = precision_recall_fscore_support(y_test, y_predict, average='binary', pos_label='unhealthy') 
  print('Using these params, test recall on the postive label: ', test_recall) 
  </code></pre>
  <p>This gives the following output:</p>
  <pre><code>Logistic Regression
  With default params, test recall on the postive label: 0.8297872340425532
  Using GS, best params are:  {'mod__C': 0.27777777777777}
  Using these params, test recall on the postive label: 0.8297872340425532
  
  KNN
  With default params, test recall on the postive label: 0.7588652482269503
  Using GS, best params are:  {'mod__n_neighbors': 1}
  Using these params, test recall on the postive label: 0.75177304964539
  </code></pre>
  <p>Am I messing up in how I define the scorer?</p>