title: Meta filtered classifier and manually filtered classifiers give different results
body: <p>I get contradicting results with two methods that in my mind should produce the same output. Could someone point out what the differences are because I can't get my head around it :S</p>
  
  <p>I am working on Drexel_Stats.arff. I use a 1-NN classifier with 10 fold cross validation.
  Without any preprocessing this is the confusion matrix I get:</p>
  
  <pre><code>  a  b   &lt;-- classified as
   14  3 |  a = Win
    5  1 |  b = Loss
  </code></pre>
  
  <p>To get better results I used:</p>
  
  <pre><code> weka.attributeSelection.InfoGainAttributeEval
   weka.attributeSelection.Ranker -T -1.0 -N 5
  </code></pre>
  
  <p>to get the 5 most discriminating features of the data set. Then I manually got rid of all the other features and re-run my 1-NN and I got these results:</p>
  
  <pre><code>  a  b   &lt;-- classified as
   16  1 |  a = Win
    1  5 |  b = Loss
  </code></pre>
  
  <p>Now that's where it gets confusing (to me at least). I tried to use a meta filtered classifier to save the hassle of manually discarding features. Here is what I used (copied from the GUI):</p>
  
  <pre><code> weka.classifiers.meta.FilteredClassifier
  -F "weka.filters.supervised.attribute.AttributeSelection
  -E \"weka.attributeSelection.InfoGainAttributeEval \"
  -S \"weka.attributeSelection.Ranker -T -1.0 -N 5\""
  -W weka.classifiers.lazy.IB1 -D
  </code></pre>
  
  <p>I understand this as being an automation of the previous operation but in fact the results I get this time are different:</p>
  
  <pre><code>  a  b   &lt;-- classified as
   15  2 |  a = Win
    4  2 |  b = Loss
  </code></pre>
  
  <p>What did I get wrong?</p>
  
  <p>Thanks</p>
  
  <p><strong>edit:</strong> Here is part of the WEKA output:</p>
  
  <pre><code>    === Attribute Selection on all input data ===
  
  Search Method:
      Attribute ranking.
  
  Attribute Evaluator (supervised, Class (nominal): 39 Outcome):
      Information Gain Ranking Filter
  
  Ranked attributes:
   0.828    1 Opponent
   0.469   38 Opp_Steals
   0.42    24 Opp_Field_Goal_Pct
   0.331   15 Def_Rebounds
   0.306   28 Opp_Free_Throws_Made
  
  Selected attributes: 1,38,24,15,28 : 5
  
  
  Header of reduced data:
  @relation 'Basketball_Statistics-weka.filters.unsupervised.attribute.Remove-V-R1,38,24,15,28,39'
  
  @attribute Opponent {Florida_Gulf_Coast,Vermont,Penn,Rider,Toledo,Saint_Joseph,Fairleigh_Dickinson,Villanova,Syracuse,Temple,George_Mason,Georgia_State,UNC_Wilmington,James_Madison,Hofstra,Old_Dominion,Northeastern,Delaware,VCU,Towson}
  @attribute Opp_Steals numeric
  @attribute Opp_Field_Goal_Pct numeric
  @attribute Def_Rebounds numeric
  @attribute Opp_Free_Throws_Made numeric
  @attribute Outcome {Win,Loss}
  
  @data
  </code></pre>
  
  <p>Are these the same features selected at each fold of the cross validation? Can different features be selected depending on the instances split?</p>