title: GridSearchCV issue with sklearn
body: <p>I am currently working on a text classifier and using GridsearchCV from sklearn to get the best hyper-parameters for my classifiers. However, there is something I don't understand in the "best_score" returned by gridsearch:</p>
  
  <pre><code>f=open('cleaned_data.pkl','rb')
  X=pickle.load(f)
  f.close()
  
  f=open('cleaned_targets.pkl','rb')
  Y=pickle.load(f)
  f.close()
  
  X_train,X_test,Y_train,Y_test=train_test_split(X,Y,test_size=0.01,random_state=1,stratify=Y)
  
  test_param_gamma=[i for i in np.arange(0.1,0.6,0.1)]
  test_param_C=[i for i in np.arange(4,4.5,0.1)]
  
  count_vect = CountVectorizer(stop_words='english')
  tfidf_transformer = TfidfTransformer()
  
  parameters = {'clf2__gamma': test_param_gamma,'clf2__C':test_param_C}
  nb=Pipeline([['cv',count_vect],['tfidf',tfidf_transformer],['clf2',SVC()]]) 
  gs_clf2 = GridSearchCV(nb, parameters,verbose=10)
  gs_clf2 = gs_clf2.fit(X_train, Y_train)
  print(gs_clf2.best_score_)
  print(gs_clf2.best_params_)
  </code></pre>
  
  <p>If I fit my gridsearch using X_train and Y_train that are smaller data sets than X and Y (but just a little because I used a test_size of 0.01), I get a 10 points higher best_score than if I train it with the entire data sets, that is to say:</p>
  
  <pre><code>gs_clf2 = gs_clf2.fit(X, Y)
  </code></pre>
  
  <p>My questions are:</p>
  
  <ol>
  <li>Why is my classifier better with a smaller data set ?</li>
  <li>Why is there such a big difference of efficiency for a data set that has only got something like 20 samples more ?</li>
  </ol>
  
  <p>NB: I observe the same evolutions using Naive Bayes classifiers...
  I have tried several value for test_size and it seems it has not a significant impact on best_score, there is something I don't understand.</p>
  
  <p>Thank you in advance !</p>
  
  <p>Antoine</p>