title: Selecting kernel and hyperparameters for kernel PCA reduction
body: <p>I'm reading <a href="https://rads.stackoverflow.com/amzn/click/com/1491962291" rel="nofollow noreferrer" rel="nofollow noreferrer">Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems</a></p>
  
  <p>I'm trying to optimize an unsupervised kernel PCA algorithm.
  Here is some context:</p>
  
  <blockquote>
    <p>Another approach, this time entirely unsupervised, is to select the kernel and hyperparameters that yield
    the lowest reconstruction error. However, reconstruction is not as easy as with linear PCA</p>
  </blockquote>
  
  <p>....</p>
  
  <blockquote>
    <p>Fortunately, it is possible to find a point in the original space that
    would map close to the reconstructed point. This is called the
    reconstruction pre-image. Once you have this pre-image, you can
    measure its squared distance to the original instance. You can then
    select the kernel and hyperparameters that minimize this
    reconstruction pre-image error.</p>
    
    <p>One solution is to train a supervised regression model, with the
    projected instances as the training set and the original instances as
    the targets.</p>
    
    <p>Now you can use grid search with cross-validation to find the kernel
    and hyperparameters that minimize this pre-image reconstruction error.</p>
  </blockquote>
  
  <p>The code provided in the book to perfom the reconstruction without cross validation is:</p>
  
  <pre><code>rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.0433,fit_inverse_transform=True)
  X_reduced = rbf_pca.fit_transform(X)
  X_preimage = rbf_pca.inverse_transform(X_reduced)
  
  &gt;&gt;&gt; from sklearn.metrics import mean_squared_error
  &gt;&gt;&gt; mean_squared_error(X, X_preimage)
  32.786308795766132
  </code></pre>
  
  <p>My question is, how do i go about implementing cross validation to tune the kernel and hyperparameters to minimze the pre-image reconstruction error?</p>
  
  <p>Here is my go at it so far:</p>
  
  <pre><code>from sklearn.metrics import mean_squared_error
  from sklearn.decomposition import KernelPCA
  
  mean_squared_error(X, X_preimage)
  
  kpca=KernelPCA(fit_inverse_transform=True, n_jobs=-1) 
  
  from sklearn.model_selection import GridSearchCV
  
  param_grid = [{
          "kpca__gamma": np.linspace(0.03, 0.05, 10),
          "kpca__kernel": ["rbf", "sigmoid", "linear", "poly"]
      }]
  
  grid_search = GridSearchCV(clf, param_grid, cv=3, scoring='mean_squared_error')
  X_reduced = kpca.fit_transform(X)
  X_preimage = kpca.inverse_transform(X_reduced)
  grid_search.fit(X,X_preimage)
  </code></pre>
  
  <p>Thank you </p>