KNN pipeline w/ cross_validation_scores Using the wine quality dataset, I'm attempting to perform a simple KNN classification (w/ a scaler, and the classifier in a pipeline). It works, but I've never used cross_val_scores this way and I wanted to be sure that there isn't a better way. For me, it became a little less straightforward using the cross_val_scores inside the pipeline, including the n range

I've included my entire code, as it's fairly short.

import pandas as pd

import pylab as pl

from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline, Pipeline, FeatureUnion
from sklearn.cross_validation import cross_val_score

X = pd.read_csv("https://s3.amazonaws.com/demo-datasets/wine.csv")
y = X.pop('high_quality')

le = LabelEncoder()
X.color = le.fit_transform(X.color)

results = []

for n in range(1, 50, 2):
   pipe = make_pipeline(StandardScaler(),
   KNeighborsClassifier(n_neighbors=n))
   c_val = cross_val_score(pipe, X, y, cv=5, scoring='accuracy').mean()
   results.append([n, c_val])


print results

results = pd.DataFrame(results, columns=["n", "accuracy"])

pl.plot(results.n, results.accuracy)
pl.title("Accuracy with Increasing K")
pl.show()
Your usage (and your code) is actually fine, which means there isn't much to review.

The only tip I would give is that having only the mean of the cross validation scores is not enough to determine if your model did well. Imagine, for instance, that you have 4 cv that gave the following accuracy scores : [0.01, 0.90, 0.90, 0.90] / 4 = 0.677 vs another set of hyper-parameters that gave [0.77, 0.79, 0.76, 0.80] / 4 = 0.78. That's a fairly crazy example, because such a big gap in the first cv scores is unrealistic, but it's only an example. The mean isn't very robust to outliers, that's what I mean. In order to have more information, it's great to also keep tracking of the standard deviation. You can also plot it using errorbar in matplotlib.