title: Best_params in GridSearch
body: <p>I was using grid_search in order to find the best combination of parameters and i made a plot to see how score is score changing when the parameters are changed.
  When i run gs_clf.best_params_ i get this as the best combination of params: 
  {'learning_rate': 0.01, 'n_estimators': 200}
  I don't understand why then the valdiation plot doesn't show the best score for this combination of parameters?</p>
  
  <p>My code is provided below.</p>
  
  <pre><code>from sklearn.ensemble import GradientBoostingClassifier
  from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_score
  from sklearn.metrics import  accuracy_score, average_precision_score, recall_score, f1_score, precision_recall_curve, auc, confusion_matrix, classification_report
  import matplotlib.pyplot as plt
  import numpy as np
  
  
  clf = GradientBoostingClassifier(min_samples_split=300, max_depth=4, random_state=0)
  
  kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=0) 
  
  number_of_estimators= [20,200]
  LR=[0.01,1]
  
  grid = GridSearchCV(clf, param_grid = dict(n_estimators=number_of_estimators,learning_rate=LR), cv=kfold, return_train_score=True, scoring = 'accuracy', pre_dispatch='1*n_jobs',n_jobs=1)
  
  gs_clf = grid.fit(X_train, Y_train.values.ravel()) # Fit the Grid Search on Train dataset
  
  scores = [x for x in gs_clf.cv_results_['mean_train_score']]
  scores = np.array(scores).reshape(len(number_of_estimators), len(LR))
  
  for ind, i in enumerate(number_of_estimators):
      plt.plot(LR, scores[ind], label='Number_of_estimators: ' + str(i))
  plt.legend()
  plt.xlabel('Learning rate')
  plt.ylabel('Mean score')
  plt.title('Train score')
  plt.show()
  
  scores = [x for x in gs_clf.cv_results_['mean_test_score']]
  scores = np.array(scores).reshape(len(number_of_estimators), len(LR))
  
  for ind, i in enumerate(number_of_estimators):
      plt.plot(LR, scores[ind], label='Number_of_estimators: ' + str(i))
  plt.legend()
  plt.xlabel('Learning rate')
  plt.ylabel('Mean score')
  plt.title('Validation score')
  plt.show()
  
  gs_clf.best_params
  </code></pre>
  
  <p>Images of plots that i get:</p>
  
  <p><a href="https://i.stack.imgur.com/kYb8A.png" rel="nofollow noreferrer">Train score plot</a></p>
  
  <p><a href="https://i.stack.imgur.com/KfNue.png" rel="nofollow noreferrer">Validation score plot</a></p>