title: Grid Search parameter and cross-validated data set in KNN classifier in Scikit-learn
body: <p>I'm trying to perform my first KNN Classifier using SciKit-Learn. I've been following the User Guide and other online examples but there are a few things I am unsure about. For this post lets use the following</p>
  
  <p>X = data
  Y = target</p>
  
  <p>1) In most introduction to machine learning pages that I've read it seems to say you want a training set, a validation set, and a test set. From what I understand, cross validation allows you to combine the training and validations sets to train the model, and then you should test it on the test set to get a score. However, I have seen in papers that in a lot of cases you can just cross validate on the entire data set and then report the CV score as the accuracy. I understand in an ideal world you would want to test on separate data but if this is legitimate I would like to cross-validate on my entire dataset and report those scores</p>
  
  <p>2) So starting the process</p>
  
  <p>I define my KNN Classifier as follows</p>
  
  <pre><code>knn = KNeighborsClassifier(algorithm = 'brute')
  </code></pre>
  
  <p>I search for best n_neighbors using</p>
  
  <pre><code>clf = GridSearchCV(knn, parameters, cv=5)
  </code></pre>
  
  <p>Now if I say</p>
  
  <pre><code>clf.fit(X,Y)
  </code></pre>
  
  <p>I can check the best parameter using</p>
  
  <pre><code>clf.best_params_
  </code></pre>
  
  <p>and then I can get a score</p>
  
  <pre><code>clf.score(X,Y)
  </code></pre>
  
  <p>But - as I understand it, this hasn't cross validated the model, as it only gives 1 score?</p>
  
  <p>If I have seen clf.best_params_ = 14 now could I go on</p>
  
  <pre><code>knn2 = KNeighborsClassifier(n_neighbors = 14, algorithm='brute')
  cross_val_score(knn2, X, Y, cv=5)
  </code></pre>
  
  <p>Now I know the data has been cross validated but I don't know if it is legitimate to use clf.fit to find the best parameter and then use cross_val_score with a new knn model?</p>
  
  <p>3) I understand that the 'proper' way to do it would be as follows</p>
  
  <p>Split to X_train, X_test, Y_train, Y_test, 
  Scale train sets -> apply transform to test sets</p>
  
  <pre><code>knn = KNeighborsClassifier(algorithm = 'brute')
  clf = GridSearchCV(knn, parameters, cv=5)
  clf.fit(X_train,Y_train)
  clf.best_params_
  </code></pre>
  
  <p>and then I can get a score</p>
  
  <pre><code>clf.score(X_test,Y_test)
  </code></pre>
  
  <p>In this case, is the score calculated using the best parameter?</p>
  
  <hr>
  
  <p>I hope that this makes sense. I've been trying to find as much as I can without posting but I have come to the point where I think it would be easier to get some direct answers. </p>
  
  <p>In my head I am trying to get some cross-validated scores using the whole dataset but also use a gridsearch (or something similar) to fine tune the parameters.</p>
  
  <p>Thanks in advance</p>