Does ensemble (bagging, boosting, stacking, etc) always at least increase performance? Ensembling is getting more and more popular. I understand that there are in general three big fields of ensembling, bagging, boosting and stacking. 

My question is that does the ensembling always at least increase the performance in practice? I guess mathematically, it is not true. I am jsut asking a real life situation.

For example, I could train 10 base learner, and then stack them with another learner, which is at 2nd level. Does this 2nd-level learner always outperform the best of the base learners in practice?
The short answer is no.

I have worked on several projects that evaluated an ensemble of several classifiers versus the classifies themselves.  In some cases the precision and recall was better with the ensemble, but more often, it was not.  That's not to say it's not worth investigating.  But sometimes, there is one model that does a reasonable job of classifying data, but it can get drowned out in an ensemble.  Perhaps a weighted ensemble might improve the results, but its not a clear cut approach to improving the performance.

In practice, I would try several models and then try an ensemble of the models.  If the ensemble is the best, however you define best, go with it.  But sometimes it is easier to just pick the best base model and then figure out how to tune that model.