Ensemble model overfitting?
Yes I suspect you are overfitting. When you build your first stage of models (nearest neighbors, random forest, gradient boosting, etc...) is the process like this?

random_forest.fit(train_data,target)
random_forest_probabilities = random_forest.predict_proba(train_data)
...


If so you are plugging in the same data you used to train the models into your models to get probabilities, which will lead to better probabilities than should be expected in real life and thus much better than should be expected results on your 2nd stage model (i.e.: overfitting). To remedy this you must do everything within a cross validation loop, that looks like this in pseudo code:

for &lt;fold&gt; in &lt;crossvalidation&gt;:
     random_forest.fit(&lt;train_data_fold&gt;,&lt;target_fold&gt;)
     &lt;fold predictions&gt; = random_forest.fit(&lt;test_data_fold&gt;)
     ...


Doing things this way will emulate real life since your predictions will be gathered using models that did not include that data when training them. You can build your final model using all of your prediction variables as input. If you struggle with actually executing on this either edit your question with code or post a new question on how to run everything with a CV loop.