title: Why is my mean test score at parameter tuning (cv) lower than on hold out test set (RandomForestClassifier)?
body: <p>I'm doing hyperparameter tuning using <code>RandomizedSearchCV</code> (sklearn) with a 3 fold cross validation on my training set. After that I'm checking my score (accuracy, recall_weighted, cohen_kappa) on the test set. Surprisingly its always a bit higher than the best_score attribute of my <code>RandomizedSearchCV</code>.</p>
  
  <p>In the beginning I'm performing a stratified data split into 70/30 training and testing sets.</p>
  
  <p>My dataset includes 12 classes, 12 features and is imbalanced. I have ~3k datapoints. </p>
  
  <p>Is it normal (or not too surprising) when I compare the cross validation training score of the parameter tuning with the score on a hold out test set?</p>
  
  <p>I already tried different random seeds for the initial split and different scoring methods (accuracy, recall_macro, recall_weighted, cohen_kappa).</p>
  
  <p>Here is my code:</p>
  
  <pre><code>#Split data in training and test set (70/30 stratified split)
  x_train, x_test, y_train, y_test = train_test_split(X_Distances, Y, test_size=0.3, random_state=42, stratify=Y, shuffle=True)
  
  #Scorings used for parameter tuning evaluation
  scoring = {'Accuracy' : make_scorer(accuracy_score), 'Recall' : 'recall_weighted', 'Kappa' : make_scorer(cohen_kappa_score)}
  
  #Initializing of parameter ranges
  params_randomSearch = {"min_samples_leaf": np.arange(1,30,2),
                "min_samples_split": np.arange(2,20,2),
                "max_depth": np.arange(2, 20, 2),
                "min_weight_fraction_leaf": np.arange(0. ,0.4, 0.1),
                "n_estimators": np.arange(10, 1000, 100),
                "max_features" : ['auto', 'sqrt', 'log2', None],
                "criterion" : ['entropy', 'gini']}
  
  #Perform RandomSearchCV over a wide range of possible parameters
  if __name__ == '__main__':
      rs = RandomizedSearchCV(RandomForestClassifier(random_state=42), param_distributions=params_randomSearch, scoring = scoring, cv = 3, refit = 'Recall', n_iter=60, n_jobs=-1, random_state=42)
      rs.fit(x_train, y_train)
      print('Best Score: ', rs.best_score_, '\nBest parameters: ', rs.best_params_)
      y_predict = rs.best_estimator_.predict(x_test)
      acc = recall_score(y_test, y_predict, average='weighted')
  </code></pre>
  
  <p>Results for recall_weighted:</p>
  
  <pre><code># RandomizedSearchCV:
  best_params_ = {dict} {'n_estimators': 310, 'min_weight_fraction_leaf': 0.0, 'min_samples_split': 12, 'min_samples_leaf': 5, 'max_features': 'auto', 'max_depth': 14, 'criterion': 'entropy'}
  best_score_ = {float64} 0.5103216514642342
  
  # Hold out test set:
  0.5666293393057111
  </code></pre>
  
  <p>I want to use the hold-out test set to compare how different algorithms work on this data set.</p>
  
  <p><strong>Question</strong>: Is there an error in my approach leading to this difference in score or can I ignore it and how should I interpret it?</p>