title: Choosing between the SVM and the MLP classifier
body: <p>I have to train a classifier that would be able to discern 6 possible classes of the input samples. I also have a <em>Cost Matrix</em> to estimate the classifier's performance with and without considering the reject option.</p>
  
  <p>So far, using the <em>Cross-validation</em> (<em>leave-one-out</em>), I split the dataset in training set and validation set, so I can measure the classifier's performance. I've reached these results in terms of <strong>accuracy</strong>:</p>
  
  <ul>
  <li><strong>MultiLayer Perceptron</strong>: 57,69% without the reject option, 48,26% with the reject option</li>
  <li><strong>Support Vector Machine</strong>: 61,99% without the reject option, 35,09% with the reject option</li>
  </ul>
  
  <p>While in terms of <strong>costs</strong> (these are estimates obtained using the <em>Minimum Risk Classification</em> rule):</p>
  
  <ul>
  <li><strong>MLP</strong>: 2,0028 without the reject option, 1,4965 with the reject option</li>
  <li><strong>SVM</strong>: 1,6089 without the reject option, 0,8502 with the reject option</li>
  </ul>
  
  <p>So I've reached a point where I don't know which classifier is better.</p>
  
  <p>Of course the SVM has ridiculous low costs, but when you consider the reject option you suddenly notice that its accuracy is pretty bad (-13% lower than MLP).</p>
  
  <p>In terms of <strong>accuracy</strong>, I'd say that the MLP is <em>better</em> than the SVM because of its average accuracy (considering both with/without the reject option): 52,97% (MLP) vs 48,54% (SVM).</p>
  
  <p>But the SVM is <em>better</em> in terms of average cost: 1,74965 (MLP) vs 1,22955 (SVM).</p>
  
  <p>Are there any guide lines to facilitate this decision?</p>
  
  <p><strong>Edit (more information as requested):</strong> the dataset has ~700 samples with ~1250 features. However with the Feature Selection I reduced the features to 81.</p>
  
  <p>The test set (which I don't have) will be of ~700 samples.</p>