title: Ridge Regression: Scikit-learn vs. direct calculation does not match for alpha > 0
body: <p>In Ridge Regression, we are solving <code>Ax=b</code> with <code>L2</code> Regularization. The direct calculation is given by:</p>
  
  <blockquote>
    <p>x = (A<sup>T</sup>A + alpha * I)<sup>-1</sup>A<sup>T</sup>b</p>
  </blockquote>
  
  <p>I have looked at the scikit-learn code and they do implement the same calculation. But, I can't seem to get the same results for <code>alpha &gt; 0</code></p>
  
  <p>The minimal code to reproduce this.</p>
  
  <pre><code>import numpy as np
  A = np.asmatrix(np.c_[np.ones((10,1)),np.random.rand(10,3)])
  b = np.asmatrix(np.random.rand(10,1))
  I = np.identity(A.shape[1])
  alpha = 1
  x = np.linalg.inv(A.T*A + alpha * I)*A.T*b
  print(x.T)
  &gt;&gt;&gt; [[ 0.37371021  0.19558433  0.06065241  0.17030177]]
  
  from sklearn.linear_model import Ridge
  model = Ridge(alpha = alpha).fit(A[:,1:],b)
  print(np.c_[model.intercept_, model.coef_])
  &gt;&gt;&gt; [[ 0.61241566  0.02727579 -0.06363385  0.05303027]]
  </code></pre>
  
  <p>Any suggestions on what I can do to resolve this discrepancy?</p>