title: Matlab and Support Vector Machines: Why doesn't the implementation of PCA give good prediction results?
body: <p>I have a training dataset with 60,000 images and a testing dataset with 10,000 images. Each image represents an integer number from 0 to 9. My goal was to use libsvm which is a library for Support Vector Machines in order to learn the numbers from the training dataset and use the classification produced to predict the images of the testing dataset.</p>
  
  <p>Each image is <code>28x28</code> which means that it has <code>784</code> pixels or <em>features</em>. While the features seem to be too many it took only 5-10 minutes to run the SVM application and learn the training dataset. The testing results were very good giving me <code>93% success rate</code>.</p>
  
  <p>I decided to try and use PCA from matlab in order to reduce the amount of features while at the same time not losing too much information. </p>
  
  <pre><code>[coeff scores latent] = princomp(train_images,'econ');
  </code></pre>
  
  <p>I played with the latent a little bit and found out that the first 90 features would have as a result <code>10%</code> information loss so I decided to use only the first 90.</p>
  
  <p>in the above code train_images is an array of size <code>[60000x784]</code></p>
  
  <p>from this code I get the scores and from the scores I simply took the number of features I wanted, so finally I had for the training images an array of <code>[60000x90]</code></p>
  
  <p><strong>Question 1: What's the correct way to project the testing dataset to the coefficients => coeff?</strong></p>
  
  <p>I tried using the following:</p>
  
  <pre><code>test_images = test_images' * coeff;
  </code></pre>
  
  <p>Note that the test_images accordingly is an array of size <code>[784x10000]</code> while the coeff an array of size <code>[784x784]</code></p>
  
  <p>Then from that again I took only the 90 features by doing the following:</p>
  
  <pre><code>test_images = test_images(:,(1:number_of_features))';
  </code></pre>
  
  <p>which seemed to be correct. However after running the training and then the prediction, I got a <code>60% success rate</code> which is way lower than the success rate I got when I didn't use any PCA at all.</p>
  
  <p><strong>Question 2: Why did I get such low results?</strong></p>
  
  <p>After PCA I scaled the data as always which is the correct thing to do I guess. Not scaling is generally not a good idea according to the libsvm website so I don't think that's an issue here.</p>
  
  <p>Thank you in advance</p>