title: Using cross validation to decrease variation in prediction outcomes
body: <p>Im trying to predict the won category of some soccer matches. I have the following df:</p>
  
  <pre><code>head(df)
  won               TEAM1              TEAM2  EXPG1   EXPG2 VAR1  VAR2
  41   1            Bordeaux             Bastia 1.4200 0.93285  0.33  0.32
  42   0                Caen             Rennes 1.3105 1.11580  0.44  0.43
  43   2                Lens              Reims 1.3678 0.90433  0.34  0.43
  44   2             Lorient           Guingamp 1.1773 0.96671  0.43  0.54
  45   2 Olympique Marseille               Nice 1.3541 0.89154  0.53  0.54
  46   2                Metz Olympique Lyonnais 1.1768 0.99026  0.53  0.61
  </code></pre>
  
  <p>No I would like to the following:</p>
  
  <ul>
  <li>Test whether my prediction gets better if I include VAR1 OR VAR2 besides EXPG1 and EXPG2</li>
  <li>(If my prediction get bettter) - test whether OR VAR1 or VAR2 should be included (which var gets a better accuracy.</li>
  </ul>
  
  <p>I did the following</p>
  
  <pre><code>#library(caret)
  inTrain &lt;- createDataPartition(y=df$won, p=0.7, list=FALSE)
  training &lt;- df[inTrain, ]
  testing &lt;- df[-inTrain, ]
  
  #perform rf for EXPG1 and EXPG2
  fit1 &lt;- randomForest(won ~ EXPG1 + EXPG2, data=training, importance=TRUE, ntree=2000) 
  Prediction1 &lt;- predict(fit1, testing)
  conf1 &lt;- confusionMatrix(Prediction, testing$won)
  #get accuracy
  
     #perform rf for EXPG1, EXPG2 and VAR1
  fit1 &lt;- randomForest(won ~ EXPG1 + EXPG2 + VAR1 , data=training, importance=TRUE, ntree=2000) 
  Prediction1 &lt;- predict(fit1, testing)
  conf1 &lt;- confusionMatrix(Prediction, testing$won)
  #get accuracy
  
  
  #perform rf for EXPG1, EXPG2 and VAR1
  fit1 &lt;- randomForest(won ~ EXPG1 + EXPG2 + VAR2 , data=training, importance=TRUE, ntree=2000) 
  Prediction1 &lt;- predict(fit1, testing)
  conf1 &lt;- confusionMatrix(Prediction, testing$won)
  #get accuracy
  </code></pre>
  
  <p>To get an overview of whether VAR1/VAR2 have impact and which factor might have better impact. </p>
  
  <p>Im not sure whether this is the right approach however because when I run this statement twice (WITHOUT setting seed):</p>
  
  <pre><code>     #perform rf for EXPG1 and EXPG2
  fit1 &lt;- randomForest(won ~ EXPG1 + EXPG2, data=training, importance=TRUE, ntree=2000) 
  Prediction1 &lt;- predict(fit1, testing)
  conf1 &lt;- confusionMatrix(Prediction, testing$won)
  #get accuracy
  </code></pre>
  
  <p>I get two different accuracy values (41% en 43%). </p>
  
  <p>So if im getting the result that adding VAR1 increases the accuracy by 1% that might be due to chance (as running the algoritm twice with only EXPG1 and EXPG2 can already differ 2%).</p>
  
  <p>Any thoughts on how I can do a solid test to see what the added value of VAR1 and VAR2 might be?</p>