SKLearn - Principal Component Analysis leads to horrible results in knn predictions
Question: by adding PCA to the algorithm, I'm working to improve %96.5 SKlearn kNN
prediction score for kaggle digit recognition tutorial, yet new kNN
predictions based on PCA output are horrible like 23%.

below is the full code and i appreciate if you point out where i am mistaken.

    
    
    import pandas as pd
    import numpy as np
    import pylab as pl
    import os as os
    from sklearn import metrics
    %pylab inline
    os.chdir("/users/******/desktop/python")
    
    traindata=pd.read_csv("train.csv")
    traindata=np.array(traindata)
    traindata=traindata.astype(float)
    X,y=traindata[:,1:],traindata[:,0]
    
    from sklearn.cross_validation import train_test_split
    X_train, X_test, y_train, y_test= train_test_split(X,y,test_size=0.25, random_state=33)
    
    #scale & PCA train data
    from sklearn import preprocessing
    from sklearn.decomposition import PCA
    X_train_scaled = preprocessing.scale(X_train)
    estimator = PCA(n_components=350)
    X_train_pca = estimator.fit_transform(X_train_scaled)
    
    # sum(estimator.explained_variance_ratio_) = 0.96
    
    from sklearn.neighbors import KNeighborsClassifier
    neigh = KNeighborsClassifier(n_neighbors=6)
    neigh.fit(X_train_pca,y_train)
    
    # scale & PCA test data
    X_test_scaled=preprocessing.scale(X_test)
    X_test_pca=estimator.fit_transform(X_test_scaled)
    
    y_test_pred=neigh.predict(X_test_pca)
    # print metrics.accuracy_score(y_test, y_test_pred) = 0.23
    # print metrics.classification_report(y_test, y_test_pred)
    


Answer: When you are processing the test data, you used `fit_transform(X_test)` which
actually recomputes another PCA transformation on the test data. You should be
using `transform(X_test)`, so that the test data undergoes the same
transformation as the training data.

The portion of code will look something like (thanks ogrisel for the `whiten`
tip):

    
    
    estimator = PCA(n_components=350, whiten=True)
    X_train_pca = estimator.fit_transform(X_train)
    X_test_pca = estimator.transform(X_test)
    

Try and see if it helps?

