title: How to stack machine learning models in R
body: <p>I am new to machine learning and R.</p>
  
  <p>I know that there is an R package called <a href="http://github.com/zachmayer/caretEnsemble" rel="nofollow noreferrer">caretEnsemble</a>, which could conveniently stack the models in R. However, this package looks has <a href="http://github.com/zachmayer/caretEnsemble/issues/8" rel="nofollow noreferrer">some problems when deals with multi-classes classification tasks</a>. </p>
  
  <p>Temporarily, I wrote some codes to try to stack the models manually and here is the example I worked on:</p>
  
  <pre><code>    library(caret)
      set.seed(123)
      library(AppliedPredictiveModeling)
      data(AlzheimerDisease)
      adData = data.frame(diagnosis, predictors)
      inTrain = createDataPartition(adData$diagnosis, p = 3 / 4)[[1]]
      training = adData[inTrain,]
      testing = adData[-inTrain,]
  
      set.seed(62433)
      modelFitRF &lt;- train(diagnosis ~ ., data = training, method = "rf")
      modelFitGBM &lt;- train(diagnosis ~ ., data = training, method = "gbm",verbose=F)
      modelFitLDA &lt;- train(diagnosis ~ ., data = training, method = "lda")
  
      predRF &lt;- predict(modelFitRF,newdata=testing)
      predGBM &lt;- predict(modelFitGBM, newdata = testing)
      prefLDA &lt;- predict(modelFitLDA, newdata = testing)
  
      confusionMatrix(predRF, testing$diagnosis)$overall[1]
      #Accuracy 
      #0.7682927 
  
      confusionMatrix(predGBM, testing$diagnosis)$overall[1]
      #Accuracy 
      #0.7926829 
  
      confusionMatrix(prefLDA, testing$diagnosis)$overall[1]
      #Accuracy 
      #0.7682927
  </code></pre>
  
  <p>Now I've got three models: <code>modelFitRF</code>, <code>modelFitGBM</code> and <code>modelFitLDA</code>, and three predicted vectors corresponding to such three models based on the <code>test set</code>. </p>
  
  <p>Then I will create a data frame to contain these predicted vectors and the original dependent variable in the <code>test set</code>:</p>
  
  <pre><code>   predDF &lt;- data.frame(predRF, predGBM, prefLDA, diagnosis = testing$diagnosis, stringsAsFactors = F)
  </code></pre>
  
  <p>And then, I just used such data frame as a new <code>train set</code> to create a stacked model:</p>
  
  <pre><code>   modelStack &lt;- train(diagnosis ~ ., data = predDF, method = "rf")
     combPred &lt;- predict(modelStack, predDF)
     confusionMatrix(combPred, testing$diagnosis)$overall[1] 
     #Accuracy 
     #0.804878
  </code></pre>
  
  <p>Considering that stacking models usually should improve the accuracy of the predictions, I'de like to believe this might be a right to stack the models. However, I also doubt that here I used the <code>predDF</code> which is created by the predictions from three models with the <code>test set</code>. </p>
  
  <p>I am not sure <strong>whether I should use the results from the <code>test set</code> and then apply them back to the <code>test set</code> to get final predictions</strong>? 
  (I am referring to this block below:)</p>
  
  <pre><code>   predDF &lt;- data.frame(predRF, predGBM, prefLDA, diagnosis = testing$diagnosis, stringsAsFactors = F)
     modelStack &lt;- train(diagnosis ~ ., data = predDF, method = "rf")
     combPred &lt;- predict(modelStack, predDF)
     confusionMatrix(combPred, testing$diagnosis)$overall[1] 
  </code></pre>