Post processing with Random Forest
Here's a solution using the caret package for R. A Random Forest is first trained on the data. All observations for which the probability (from the voting) is less than 99% are then passed to model 2, linear discriminant analysis. Only the probabilities from unseen resampling observations are used, since the Random Forest will otherwise fit the training data perfectly. That is what caret is needed for.

The accuracy is a little higher for the uncertain cases, but this is probably overfitting since I have tried several different models and the data set is small.

I'd like to know if this really improves the performance in your application for the out of sample test data. Are there any papers that recommend this approach? This approach seems to resemble boosting. I tried it on some of my data but could not improve the out of sample performance I got from model 1 (a Random Forest).

data(iris)
library(caret)
myTrainControl &lt;- trainControl(method = "cv", number = 10,
                               savePredictions = T,
                               classProbs = TRUE)
set.seed(4213) # To get the same resamples every time
M1 &lt;- train(y = iris$Species,
            x = iris[, !(names(iris) == "Species")],
            method = "rf",
            trControl = myTrainControl)

M1pred &lt;- M1$pred[M1$pred$mtry == 2, ]
confusionMatrix(M1pred$pred, M1pred$obs)
# Accuracy 96%

# Inspect the predicted class probabilities:
probs &lt;- cbind(M1pred$setosa,
                   M1pred$versicolor,
               M1pred$virginica)
colnames(probs) &lt;- c("setosa", "versicolor", "virginica")
maxCol &lt;- max.col(probs)
probs &lt;- cbind(probs, maxCol)
maxProbs &lt;- apply(probs, 1, function(x) x[x["maxCol"]])
summary(maxProbs)
# Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
# 0.5120  0.9805  0.9980  0.9645  1.0000  1.0000

# Let's define anything below 99% as uncertain:
uncertain &lt;- which(maxProbs &lt; 0.99)
length(uncertain) # 45
# Find rows in iris data that belong to 'uncertain' cases:
uncertainIndex &lt;- M1pred$rowIndex[uncertain]
    confusionMatrix(M1pred$pred[uncertain],
                reference = M1pred$obs[uncertain])
# M1 Performance in uncertain cases:
#              Reference
# Prediction   setosa versicolor virginica
# setosa          5          0         0
# versicolor      0         17         3
# virginica       0          3        17
#
# Overall Statistics
# Accuracy : 0.8667

# Train new model on uncertain data only:
irisUncertain &lt;- iris[uncertainIndex, ]
set.seed(4213) # To get the same resamples every time
M2 &lt;- train(y = irisUncertain$Species,
            x = irisUncertain[, !(names(irisUncertain) == "Species")],
            method = "lda",
            trControl = myTrainControl)

M2pred &lt;- M2$pred

confusionMatrix(M2pred$pred, reference = M2pred$obs)
#              Reference
# Prediction   setosa versicolor virginica
# setosa          4          0         0
# versicolor      1         16         0
# virginica       0          4        20
# 
# Overall Statistics
# Accuracy : 0.8889 
# (Small discrepancy: Why does caret report an accuracy of 88,5% for M2?)

# For new data predictions can be made as follows:
# (just as an example from the original data again)
# Some 'uncertain' cases are 'certain' now using the full M2 model
newdat &lt;- irisUncertain[30:34, -5]
M1maxProbs &lt;- apply(predict(M1, newdat, type = "prob"), 1, max)
ifelse(M1maxProbs &lt; 0.99,
       paste("M2:", predict(M2, newdat)),
       paste("M1:", predict(M1, newdat)))
# 85               94              107              126 
# "M2: versicolor" "M1: versicolor"  "M2: virginica"  "M1: virginica" 
# 71 
# "M2: virginica"