How to do boosting in model-ensembling? Boosting is a sequential technique in which, the first algorithm is trained on the entire dataset and the subsequent algorithms are built by fitting the residuals of the first algorithm, thus giving higher weight to those observations that were poorly predicted by the previous model. Examples are adaboost and GBM, etc. My question is that how to perform boosting when ensembling the base learner? Especially how to get the residual if it is a classification problem?

I know how to bagging the base learners and stacking the base learners. I just have no idea how to boosting the base learners.

Thanks!
You can use any base learner for boosting (Adaboost requires sample weighting though). Keep in mind however that the original idea is to use weak learners with strong bias and reducing that bias through boosting.

If it is a classification problem, usually logarithmic loss is used to calculating the residual/gradient for boosting.

For Python, there is a nice AdaBoost wrapper in scikit-learn (AdaBoostClassifier) which can take for example a Random Forest as base learner.