title: Interpreting OLS Weights after PCA (in Python)
body: <p>I want to interpret the regression model weights in a model where the input data has been pre-processed with <strong>PCA</strong>. In reality, I have 100s of input dimensions which are highly correlated, so I know that PCA is useful. However, for the sake of illustration I will use the <strong>Iris</strong> dataset.</p>
  
  <p>The <strong>sklearn</strong> code below illustrates my question:</p>
  
  <pre><code>import numpy as np
  import sklearn.datasets, sklearn.decomposition
  from sklearn.linear_model import LinearRegression
  
  # load data
  X = sklearn.datasets.load_iris().data
  w = np.array([0.3, 10, -0.1, -0.01])
  Y = np.dot(X, w)
  
  # set number of components to keep from PCA
  n_components = 4
  
  # reconstruct w
  reg = LinearRegression().fit(X, Y)
  w_hat = reg.coef_
  print(w_hat)
  
  # apply PCA
  pca = sklearn.decomposition.PCA(n_components=n_components)
  pca.fit(X)
  X_trans = pca.transform(X)
  
  # reconstruct w
  reg_trans = LinearRegression().fit(X_trans, Y)
  w_trans_hat = np.dot(reg_trans.coef_, pca.components_)
  print(w_trans_hat)
  </code></pre>
  
  <p>Running this code, one can see that the weights are reproduced fine.</p>
  
  <p>However, if I set the number of components to 3 (i.e. <code>n_components = 3</code>) then then weights printed out deviate substantially from the true ones.</p>
  
  <p>Am I misunderstanding how I can transform back these weights? Or is it because of PCA's information loss moving from 4 to 3 components?</p>