title: Principal Component Analysis in Python: Analytical Mistake
body: <p>I'm implementing a <code>Principal Component Analysis</code> for face recognition in <code>Python</code> without making use of the already defined <code>PCA</code> methods in <code>numpy</code> or <code>OpenCV</code>. But my results are literally rubbish.</p>
  
  <p>I read the documentation and algorithmic description at <code>OpenCV</code>'s <a href="http://docs.opencv.org/modules/contrib/doc/facerec/facerec_tutorial.html" rel="nofollow noreferrer">doc</a>. But some things aren't clear.</p>
  
  <ul>
  <li>If <code>X = {x1, x2, ..., xn}</code> then I believe it isn't the X matrix used when calculating the Covariance matrix? But that you've to subtract the mean etc. as explained in the first two steps.</li>
  <li>The eigenvectors must be sorted corresponding to the eigenvalues in descending order. Do you have to sort while looking at the absolute value of the corresponding eigenvalue or not? Anyway the order is not the main issue as I plot all the eigenvectors, so I can rearrange them accordingly. I think I made an analytical mistake.</li>
  </ul>
  
  <p>I implemented the following:</p>
  
  <pre><code>MU = X.mean(axis=0)
  for i in range(n):
      X[i,:] -= MU
  
  S = (np.dot(X, X.T) / float(n))
  #Hermitian (or symmetric) matrix.
  eigenvalues, eigenvectors = numpy.linalg.eigh(S)
  eigenvectors = numpy.dot(X.T, eigenvectors)
  for i in range(n):
      eigenvectors[:,i] = normalize_vector(eigenvectors[:,i])
  </code></pre>
  
  <p>Note that the sample values are stored in the rows and not the columns. So <code>X</code> is of shape <code>nxd</code> with <code>n</code> the number of samples and <code>d</code> the dimension of the samples.</p>
  
  <p><img src="https://i.stack.imgur.com/oqUv0.png" alt="enter image description here"></p>
  
  <p>The above image is the reference. The first is the mean, the following three are the largest eigenvectors.
  The below image is my result. The first is the mean, the following are all the eigenvectors in some order. But they seem not to match the result.</p>
  
  <p>The <code>cv2.PCACompute(X, 6)</code> still produces better results.</p>