An update on statistical boosting in biomedicine
Andreas Mayr Benjamin Hofner Elisabeth Waldmann Tobias Hepp Olaf Gefeller Matthias Schmid








An update on statistical boosting in biomedicine 





Andreas Mayr[1] Address for correspondence:
Andreas Mayr, Institut für Medizininformatik, Biometrie und
Epidemiologie, Friedrich-Alexander Universität
Erlangen-Nürnberg, Waldstr. 6, 91054 Erlangen.
 Email:
mailto:andreas.mayratfau.deandreas.mayr@fau.de^1,3,
Benjamin Hofner^2, Elisabeth Waldmann^1,
 Tobias Hepp^1, Olaf Gefeller^1, Matthias Schmid^3 




 [1] Institut für Medizininformatik, Biometrie und Epidemiologie,

Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany

[2] Section Biostatistics,
Paul-Ehrlich-Institut, Langen, Germany 

[3] Institut für medizinische Biometrie, Informatik und Epidemiologie, 

Rheinische Friedrich-Wilhelms-Universität Bonn, Germany











Statistical boosting algorithms have triggered a lot of research during the last decade. They combine a powerful machine-learning approach with classical statistical modelling, offering various practical advantages like automated variable selection and implicit regularization of effect estimates. They are extremely flexible, as the underlying base-learners (regression functions defining the type of effect for the explanatory variables) can be combined with any kind of loss function (target function to be optimized, defining the type of regression setting). In this review article, we highlight the most recent methodological developments on statistical boosting regarding variable selection, functional regression and advanced time-to-event modelling. Additionally, we provide a short overview on relevant applications of statistical boosting in biomedicine.









§ INTRODUCTION


Statistical boosting algorithms are one of the advanced methods in the toolbox of a modern statistician or data scientist <cit.>. They offer multiple advantages in the presence of high-dimensional data as they can deal with more potential candidate variables than observations (p > n situations) while still yielding classical statistical models with well-known interpretability <cit.>. Key features in this context are automated variable selection and model choice <cit.>.

The field of research is methodologically situated between the world of statistics and computer science. They bridge the gap between two rather different point of views on how to gather information from data <cit.>: on the one hand, there is the classical statistical modelling community that focuses on models describing and explaining the outcome in order to find an approximation to the underlying stochastic data generation process. On the other hand, there is the machine learning community that focuses primarily on algorithmic models predicting the outcome while treating the nature of the underlying process as unknown. Statistical boosting algorithms have their roots in machine learning <cit.> but were later adapted in order to estimate classical statistical models <cit.>. A pivotal aspect of these algorithms is that they incorporate data-driven variable selection and shrinkage of effect estimates similar to the one of classical penalized regression <cit.>. 

In a review some years ago <cit.>, we highlighted this evolution of boosting from machine-learning to statistical modelling. Furthermore, we emphasized the similarity of two boosting approaches – gradient boosting <cit.> and likelihood-based boosting <cit.> –  introducing statistical boosting as a generic term for these kind of algorithms. Throughout this article, we will use this term to reflect both approaches. 

The earlier review <cit.> was accompanied by a second article <cit.>  highlighting the multiple variants the basic algorithms have been extended towards (i) enhanced variable selection properties, (ii) new types of predictor effects, and (iii) new regression settings. The substantial new methodological developments on statistical boosting algorithms throughout the last years (e.g., stability selection <cit.>), opening the door for the growing community to new model classes and frameworks (e.g., joint models <cit.> and functional data <cit.>), make it necessary to provide an update on the available extensions. 

This article is structured as follows: In Section 2 we shortly highlight both basic structure and properties of statistical boosting algorithms and point to their connections to classical penalization approaches like the lasso. In Section 3 we focus on new developments regarding variable selection, which can be also combined with boosted functional regression models presented in Section 4. Section 5 focuses on advanced survival models before we briefly summarize in Section 6 what other relevant developments and applications have been proposed for the framework of statistical boosting.




§ STATISTICAL BOOSTING





 §.§ From machine learning to statistical models


The original boosting concept by Schapire <cit.> and Freund <cit.> emerged from the field of supervised learning, focusing on boosting the accuracy of weak classifiers (base-learners) by iteratively applying them to re-weighted data to get stronger results.  

Even if the base-learners individually only slightly outperform random guessing the combined ensemble solution can often be boosted to a perfect classification <cit.>. The introduction of AdaBoost <cit.> was the breakthrough for boosting in the field of supervised machine learning, allegedly leading Leo Breiman to praise its performance: Boosting is the best off-the-shelf classifier in the world <cit.>.

The main target of classical machine-learning approaches is predicting observations y_new of the outcome Y given one or more input variables X = {X_1, …,X_p}. The estimation of the prediction or generalization function is based on an observed sample (y_1, x_1), …, (y_n, x_n). However, since the underlying nature of the data generating process is treated as unknown, the focus is not on quantifying or describing this process, but solely on predicting ŷ_new for new observations x_new as accurately as possible.

As a consequence, many machine-learning approaches (also including the original AdaBoost with trees or stumps) should mainly be seen as black box prediction schemes. Although typically yielding accurate predictions <cit.>, they do not offer much insight into the structure of the relationship between explanatory variables X and the outcome Y.

Statistical regression models on the other hand, particularly aim at describing and explaining the underlying relationship in a structured way. The impact of single explanatory variables can not only be quantified in terms of variable importance measures <cit.>, but the actual effect of these variables is interpretable. The work of Friedman et al. <cit.> laid the groundwork to understand the concept of boosting from a statistical perspective and to adapt the general idea in order to estimate statistical models. 



 §.§ General model structure


The aim of statistical boosting algorithms is to estimate and select the effects in structured additive regression models. The most important model class are generalized additive models ('GAM', <cit.>), where the conditional distribution of the response variable is assumed to follow an exponential family distribution. Then, the expected response is modeled given the observed value x of one or more  explanatory variables using a link-function g as 

    g(E(Y|X=x)) =  f(x).


In the typical case of multiple explanatory variables,  the function f(·) is called additive predictor and consists of the additive effects of the single predictors,

    f(x) = β_0 +  f_1(x_1) + ⋯ + f_p(x_p)

where β_0 represents a common intercept and the functions f_j(x_j), j = 1, … p are the partial effects of the variables x_j. The generic notation f_j(x_j) may comprise different types of predictor effects such as classical linear effects x_jβ_j, smooth non-linear effects constructed via regression splines, spatial effects or random effects of the explanatory variable x_j, to name but a few. 
In statistical boosting algorithms, the different partial effects are estimated by separate base-learners h_1(·), ..., h_p(·) (component-wise boosting, <cit.>) which are typically simple regression-type prediction functions.







 §.§ Gradient boosting


Gradient boosting <cit.> is one of the two important approaches in the context of statistical boosting. For a generic overview on the structure of statistical boosting algorithms see Box <ref>. 

In gradient boosting, the iterative procedure fits the base-learners h_1(x_1),..., h_p(x_p) one-by-one to the negative gradient of the loss function ρ(y,f(·)), evaluated at the previous iteration:

    u =  (-  . ∂/∂ fρ(y_i,f(·) )|_ f = f̂^[m-1](·))_i = 1,...,n.


The loss function describes the discrepancy between the observed outcome y and the additive predictor f(·) and is the target function that should be minimized to get an optimal fit for f(·). In case of GAMs, the loss function is typically the negative log-likelihood of the corresponding exponential family. For Gaussian distributed outcomes, this reduces to the L_2 loss ρ(y, f(·)) = (y - f(·))^2, where the gradient vector u is simply the vector of residuals y - f(·) from the previous iteration and boosting hence corresponds to refitting of residuals. 

In each boosting iteration, only the best-fitting base-learner h_j^* is selected based on the residual sum of squares of the base-learner fit

    j^* = 1 ≤ j ≤ pargmin∑_i=1^n (u_i - ĥ_j(x_j))^2  .

Only this base-learner h_j^* is added to the current additive predictor f(·). In order to ensure small updates, only a small proportion of the base-learner fit (typically the step length is ν = 0.1 <cit.>) is actually added. Note that the base-learner h_j(·) can be selected and updated various times; the partial effect of variable x_j is the sum of all corresponding base-learner that had been selected: 

    f̂_j(x_j) = ∑_m  ν·ĥ_j(x_j) I_j = j* .



This component-wise procedure of fitting the base-learners one by one to the current gradient of the loss function can be described as gradient descent in function space <cit.>, where the function space is spanned by the base-learners. The algorithm effectively optimizes the loss function step-by-step, eventually converging to the minimum. In order to avoid overfitting and to ensure variable selection, the algorithm is typically stopped before convergence (based on predictive performance evaluated via cross-validation or resampling <cit.>), which leads to an implicit penalization <cit.>. 

Gradient boosting is implemented in the add-on package mboost <cit.> for the  open source programming environment  <cit.>, providing a large number of pre-implemented loss functions for various regression settings, as well as different base-learners to represent various types of effects (see <cit.> for an overview). Recent changes in the software, which were introduced after the comprehensive mboost tutorial <cit.> are provided as Appendix <ref>.



 §.§ Likelihood-based boosting


Likelihood-based boosting <cit.> is the other general approach in the framework of statistical boosting algorithms. It follows a very similar structure as gradient boosting (see Box <ref>), although both approaches only coincide in special cases such as classical Gaussian regression via the L_2 loss <cit.>. In contrast to gradient boosting, the base-learners are directly estimated via optimizing the overall likelihood, using the additive predictor from the previous iteration as offset. In case of the L_2 loss, this has a similar effect than refitting the residuals. 


In every step, the algorithm hence optimizes regression models as base-learners one-by-one by maximizing the likelihood (using one step Fisher scoring), selecting only the base-learner j^* which leads to the largest increase in the likelihood. In order to obtain small boosting steps, a quadratic penalty term is attached to this likelihood. This has a similar effect as multiplying the fitted base-learner by a small step-length factor as in gradient boosting.

Likelihood-based boosting for generalized linear and additive regression models is provided by the  add-on package GAMBoost <cit.>, and an adapted version for boosting Cox regression is provided with CoxBoost <cit.>. For a comparison of both statistical boosting approaches, i.e., likelihood-based and gradient boosting in case of Cox proportional hazard models, we refer to <cit.>. 



 §.§ Connections to L_1-regularization


Statistical boosting algorithms result in regularized models with shrinked effect estimates although they only apply implicit penalization <cit.> by stopping the algorithm before convergence. By performing regularization without the use of an explicit penalty term, boosting algorithms clearly differ from other direct regularization techniques such as the lasso <cit.>. However, both approaches sometimes result in very similar models after being tuned to a comparable degree of regularization <cit.>. 

This close connection has been first noted between the lasso and forward stagewise regression, which can be viewed as special case of the gradient boosting algorithm (Box <ref>), and led, along with the development of least angle regression (LARS), to the formulation of the positive cone condition (PCC) <cit.>.


If this condition holds, LARS, lasso and forward stagewise regression coincide.
Figuratively speaking, the PCC requires that all coefficient estimates monotonically increase or decrease with relaxing degree of regularization and applies, for example, to the case of low-dimensional settings with orthogonal X. 
It should be noted that the PCC is connected to the diagonal dominance condition for the inverse covariance matrix of X, which allows for a more convenient way to investigate the equivalence of these approaches in practice <cit.>.

Given that the solution of the lasso is optimal with respect to the L_1-norm of the coefficient vector, these findings led to the notion of boosting as some “sort of L_1-sparse” regularization technique <cit.>, but it remained unclear which optimality constraints possibly apply to forward stagewise regression if the PCC is violated.

By further extending X with a negative version of each variable and enforcing only positive updates in each iteration, Hastie et al. <cit.> demonstrated that forward stagewise regression always approximates the solution path of a similarly modified version of the lasso.
From this perspective, they showed that forward stagewise regression minimizes the loss function subject to the L_1-arc-length

    ∑_j=1^p ∫^t_0 | ∂β_j(s)/∂ s| ds ≤ t.


This means that the travelled path of the coefficients is penalized (allowing as little overall changes in the coefficients as possible, regardless of their direction), whereas the L_1-norm considers only the absolute sum of the current set of estimates. In the same article, Hastie et al. <cit.> further showed that these properties hold for general convex loss functions and therefore apply not only to forward stagewise regression but for the more general gradient boosting method (in case of logistic regression models as well as for many other generalized linear regression settings). 

The consequence of these differing optimization constraints can be observed in the presence of strong collinearity, where the lasso estimates tend to be very unstable regarding different degrees of regularization while boosting approaches avoid too many changes in the coefficients as they consider the overall travelled path <cit.>.

It has to be acknowledged, however, that direct regularization approaches as the lasso are applied more often in practice <cit.>. Statistical boosting, on the other hand, is far more flexible due to its modular nature allowing to combine any base-learner with any type of loss-function <cit.>. 



§ ENHANCED VARIABLE SELECTION



Early stopping of statistical boosting algorithms via  cross-validation approaches plays a vital role to ensure a sparse model with optimal prediction performance on new data. Resampling, i.e., random sampling of the data drawn without replacement, tends to result in sparser models compared to other sampling schemes <cit.>, including the popular bootstrap <cit.>. By using base-learners of comparable complexity (in terms of degrees of freedom) selection bias can be strongly reduced <cit.>. The resulting models have optimal prediction accuracy on the test data. Yet, despite regularization the final models are often relatively rich <cit.>. 



 §.§ Stability selection

Meinshausen and Bühlmann <cit.> proposed a generic 
approach called stability selection to further refine the models and enhance sparsity. 
This approach was then transferred to boosting <cit.>. 
In general, stability selection can be combined with any variable selection approach 
and is especially useful for high-dimensional data with many potential predictors. To
assess how stable the selection of a variable is, B random subsets that comprise half of the data are drawn. On each of these subsets, the model is fitted until q base-learners are  selected. Usually, B = 100 subsets are sufficient. Computing the relative frequencies of random subsamples in which specific base-learners were  selected give a notion of how stable the selection is with respect to perturbations of the data. Base-learners are considered to be of importance if the selection frequency exceeds a pre-specified threshold level π_thr∈ [0.5, 1]. 

Meinshausen and Bühlmann <cit.> showed that this approach controls 
the per-family error rate (PFER), i.e., it provides an upper bound for the expected number of false positive selections (V):

    E(V) ≤q^2/(2π_thr - 1) p,

where p is the number of base-learners. This upper bound is rather conservative
and hence was further refined by Shah and Samworth <cit.> for
specific assumptions on the distribution of the selection frequencies. Stability
selection with all available error bounds is implemented for a variety of modelling
techniques in the  package stabs <cit.>.

An important issue is the choice of the hyper-parameters of stability selection. 
The choice of a fix value of q should be made such that it is large enough 
to select all hypothetically influential variables <cit.>. A sensible value for q should
usually be smaller or equal to the number of base-learners selected via early stopping with cross-validation. 

In general, the size of q is of minor importance if it is in a sensible range. 
With a fixed q the threshold for 
stable effects either the threshold for π_thr can be chosen additionally or, as can be seen from Equation (<ref>)
using equality, the upper bound for the PFER can be pre-specified and 
the threshold can be derived accordingly. The latter would be the preferred choice if error 
control is of major importance, the former if error control is just considered
a by-product (see e.g., <cit.>). For an interpretation of the 
PFER, particularly with regard to standard error rates such as the per-comparison
error rate or the family-wise error rate, we refer to Hofner et al. <cit.>. 
Note that for a fixed q, it is computationally very easy to change any of the 
other two parameters (π_thr or the upper bound for the PFER) as the 
resampling results can be reused <cit.>.

Please note that base-learners selected via stability selection might not reflect any model which can be derived with a specific penalty parameter using the original modelling approach. This means that for boosting, no m_stop value might exist that results in a model with the stably selected base-learners; the provided set of stable base-learners is a fundamentally new solution. 



 §.§ Extension and application of boosting with stability selection


Variable selection is especially important in high-dimensional gene expression data and other large scale biomedical data sources. 
Recently, stability selection with boosting was successfully applied to select a small number of informative biomarkers for survival of breast cancer patients <cit.>.
The model was derived based on a novel boosting approach that optimizes the concordance index <cit.>.
Hence, the resulting prediction rule was optimal with respect to its ability to discriminate between patients with longer and shorter survival, i.e., its discriminatory power. 

Thomas et al. <cit.> derived a modified algorithm for boosted
generalized additive models for location, scale and shape (GAMLSS, <cit.>) to allow a combination of this very flexible model class with stability selection. 
The basic idea of GAMLSS is to model all parameters of the conditional distribution by their own additive predictor and associated link function.
Extensive simulation studies showed that the new fitting algorithm leads to comparable models as the previous algorithm <cit.> but is superior regarding the computational speed, especially in combination with cross-validation approaches. Furthermore, simulations showed that this algorithm can be successfully combined with stability selection to select sparser models identifying a smaller subset of truly informative variables from high-dimensional data. The current algorithm is implemented in the  add-on package gamboostLSS <cit.>, the modified version is currently available on GitHub <cit.>.



 §.§ Further approaches for sparse models


In order to construct risk prediction signatures on molecular data, such as DNA methylation, Sariyar et al. <cit.> proposed an adaptive likelihood-based boosting algorithm. The authors included a step size modification factor c_f which represents an additional tuning parameter, adaptively controlling the size of the updates. In case of sparse settings, the approach decreases shrinkage of effect estimates (by using a larger step-length) leading to a smaller bias. In settings with larger numbers of informative variables, the approach allows to fit models with lower degree of sparsity when necessary by smaller updates. The modification factor c_f has to be selected together with m_stop via cross-validation or resampling on a two-dimensional grid. 

Zhang et al. <cit.> argue that variable ranking in practice is more favourable than variable selection, as ranking allows to easily apply a thresholding rule in order to identify a subset of informative variables. The authors implemented a pseudo-boosting approach, which is technically not based on statistical boosting but is adapted to rank and select variables for statistical models.
Note that also stability selection can be seen as a variable ranking scheme based on their selection frequency, as its selection feature is only triggered by implementing the threshold π_thr.

Following a gradient based approach, Huang et al. <cit.> adapted the sparse boosting approach by Bühlmann and Yu <cit.> in order to promote similarity of model sparsity structures in the integrative analysis of multiple data sets, which surely is an important topic regarding the trend toward big data. 




§ FUNCTIONAL REGRESSION


Due to technological developments, more and more data is measured continuously over time. Over the last years, a lot of methodological research focused on regression methods for this type of functional data. A groundbreaking work in this new and evolving field of statistics is provided by Ramsay and Silverman <cit.>. 

Functional regression models can either contain functional responses (defined on a continuous domain), functional covariates or both. This leads basically to three different classes of functional regression models, i.e., function-on-scalar (response is functional), scalar-on-function (functional explanatory variable) and function-on-function regression. For a recent review on general methodological developments on functional regression, see Morris <cit.>.




 §.§ Boosting functional data


The first statistical boosting algorithm for functional regression, allowing for data-driven variable selection, was proposed by Brockhaus et al. <cit.>. The authors' approach focused on linear array models <cit.> providing a unified framework for all three settings outlined above. Since the general structure of their gradient boosting algorithm is similar to the one in Box <ref>, the resulting models still have the same form as in (<ref>), only that the response Y and the covariates may be functions. The underlying functional partial effects h_j(x_j, t) can be represented using tensor product basis 

    h_j(x_j)(t) = (b_j(x_j)^⊤⊗ b_Y(t)^⊤)θ_j   ,

where θ_j is the vector of coefficients, b_j and b_Y are basis functions, and ⊗ denotes the Kronecker product. 

This functional array model is limited in two ways: (i) the functional responses need to be measured on a common grid and (ii) covariates need to be constant over the domain of the response.  As particularly the second assumption might often not be fulfilled in practice, Brockhaus et al. <cit.> soon after proposed a general framework for boosting functional regression models avoiding this assumption and dropping the linear array structure.  

This newer framework <cit.> comprises also all three model classes outlined above and particularly focuses on historical effects, where functional response and functional covariates are observed over the same time interval. The underlying assumption is that observations of the covariate affect the response only up to the corresponding time point t

    E(Y(t) | X = x) = ∑_j = 1^J ∫_t_1^t x_j(s) β_j(s,t) ds  ,

where s represents the time points the covariate was observed for.
In other words, only the part of the covariate function lying in the past (not the future) can affect the present response.
However, this is a sensible restriction in most practical applications and thus not a strong restriction.

Both approaches for boosting functional regression are implemented in the  add-on package FDboost <cit.>, which relies on the fitting methods and infrastructure of mboost.




 §.§ Extensions of boosting functional regression


Boosting functional data can be combined with stability selection (see Section <ref>) in order to enhance the variable selection properties of the algorithm <cit.>.

The boosting approach for functional data was already extended towards the model class of generalized additive models for location, scale and shape (GAMLSS) for a scalar-on-function setting by Brockhaus et al. <cit.>.  The functional approach was named signal regression models for location, scale and shape <cit.>. The estimation via gradient boosting is based on the corresponding gamboostLSS algorithm for boosting GAMLSS <cit.>. 

In an approach to analyse the functional relationship between bioelectrical signals like electroencephalography (EEG) and facial electromyography (EMG), Rügamer et al. <cit.> focused on extending the framework of boosting functional regression by incorporating factor specific historical effects, similar to (<ref>). 


Although functional data analysis triggered a lot of methodological research, a recent systematic review by Ullah and Finch <cit.> revealed that the number of actual biomedical applications of functional data analysis in general and functional regression in particular is rather small. The authors argued that the potential benefits of these flexible models (like richer interpretation and more flexible structures) are not yet well understood by practitioners and that further efforts are necessary to promote the actual usage of these novel techniques. 



§ BOOSTING ADVANCED SURVIVAL MODELS


While Cox regression is still the dominant model class for boosting time-to-event data (see <cit.> for a comparison of two different boosting algorithms, and <cit.> for different general approaches to estimate Cox models in the presence of high-dimensional data), over the last years several alternatives emerged <cit.>.  

In this section we will particularly focus on boosting joint models of time-to-event outcomes and longitudinal markers  but will also briefly refer to other recent extensions.



 §.§ Boosting joint models


The concept of joint modelling of longitudinal and time-to-event data has found its way into the statistical literature in the last few years as it gives a very complete answer to questions on continous data recorded over time and event times related to this continous data. Modelling those to processes independently as done up to the suggestion of the joint modelling idea <cit.> leads to misspecified models prone to bias. There are various joint modelling approaches and thus also various different model equations. The type we are going to refer to in this review are of the following type:

    y_ij   =    η_l(x_ij) + η_ls(x_i,t_ij) + ε_ij
    λ(t|α,η_s(x_i, t),η_ls(x_i, t))    =   λ_0(t)exp(η_s(x_i, t) + αη_ls(x_i, t) ),

where y_ij is the j-th observation of the i-th individual with i = 1,…,n and j=1,…,n_i and λ(t|α,η_s(x_i, t),η_ls(x_i, t)) is the hazard function for individual i at time point t. Both outcomes, the longitudinal measurement as well as the event time are modeled based on two sub-predictors each: one that is supposed to have an impact on only one of them (the longitudinal sub-predictor η_l(x_ij) and the survival sub-predictor η_s(x_ij,t)) and one of them being shared by both parts of the model (the shared sub-predictor η_ls(x_ij,t)). All those sub-predictors are functions of different, possibly time dependent variables x_i. In many cases the shared sub-predictor consists of or at least includes some type of random effects. The function λ_0(t) is the baseline hazard. Most approaches for joint models are based on likelihood or Bayesian inference using the joint likelihood resulting as a product from the above likelihoods <cit.>. Those approaches are, however, unable to conduct variable selection and cannot deal with high-dimensional data. 

Waldmann et al. <cit.> suggested a boosting algorithm tackling these challenges. The model used in this paper was a reduced version of (<ref>) in which no survival sub-predictor was considered and a fixed baseline hazard λ_0 was used. The algorithm is a version of the classical boosting algorithm as represented in Box <ref>, which is adapted to the special case of having to estimate a set of different sub-predictors (similar to <cit.>). The algorithm is therefore composed of three steps which are performed circularly.
In the first step a regular boosting step to update the longitudinal sub-predictor η_l(x_ij) is performed and the parameters of the shared sub-predictor are treated as fixed.
In the second step, the parameters of the longitudinal sub-predictor are fixed and a boosting step for the shared sub-predictor η_ls(x_ij) is conducted.
The third step is a simple optimization step: based on the current values of the parameters in both sub-predictors the likelihoods are optimized with respect to λ_0, σ^2 and α (cf., <cit.>). The number of iterations now depends on two stopping iterations m_stop, l and m_stop, ls which have to be optimized on a two dimensional grid via cross validation. Waldmann et al. <cit.> showed that the benefits of boosting algorithm (automated variable selection and handling of p>n situations) can be transfered to joint modelling and hence lay the groundwork to further extended joint modelling approaches.

The code for the approach presented here is available in the  add-on package JMboost <cit.>, currently on GitHub.



 §.§ Other new approaches on boosting survival data


Reulen and Kneib <cit.> extended the framework of statistical boosting towards multi-state models for patients exposed to competing risks (e.g., adverse events, recovery, death or relapse). The approach is implemented in the gamboostMSM package <cit.>,  relying on the infrastructure of mboost.
Möst and Hothorn <cit.> focused on boosting patient-specific survivor function based on conditional transformation models <cit.> incorporating inverse probability of censoring weights <cit.>.

When statistical boosting algorithms are used to estimate survival models, the motivation is most often the presence of high-dimensional data. De Bin et al. <cit.> investigated several approaches (including gradient boosting and likelihood-based boosting) to incorporate both clinical and high-dimensional omics data to build prediction models. 

Guo et al. <cit.> proposed a new adaptive likelihood-based boosting algorithm to fit Cox models, incorporating a direct lasso-type L_1 penalization in the fitting process in order to avoid the inclusion of variables with small effect. This general motivation is similar to the one of the boosting algorithm with step-length modification factor proposed by Sariyar et al. <cit.>. In another approach, Sariyar et al. <cit.> combined a likelihood-based boosting approach for the Cox model with random forest in order to screen for interaction effects in high-dimensional data. Hieke et al. <cit.> combined likelihood-based boosting with resampling in an approach to identify prognostic SNPs in potentially small clinical cohorts.



§ NEW FRONTIERS AND APPLICATIONS


There were even more new topics that have been incorporated into the framework of statistical boosting, but not all of them can be presented in detail here. However, we want to give a short overview of the most relevant developments, notably many of those were actually motivated by biomedical applications. 

Weinhold et al. <cit.> proposed to analyse DNA methylation data (signal intensities M and U), via a “ratio of correlated gammas” model. Based on a bivariate gamma distribution for M and U values, the authors derived the density for the ratio M/M+U and optimized it via gradient boosting.

A boosting algortihm for differential item functioning in Rasch models was developed by  Schauberger and Tutz <cit.> for the broader area of psychometrics, while Casalicchio et al. focused on a boosting subject-specific Bradley-Terry-Luce models <cit.>. 

Napolitano et al. <cit.> developed a sampled boosting algorithm for the analysis of brain perfusion images: Gradient boosting is carried out multiple times on different training sets. Each base-learner refers to a voxel and after every sampling iteration a fixed fraction of selected voxels is randomly left out from the following boosting fit, in order to force the algorithm to select new voxels. The final model is then computed as the global sum of all solutions. Feilke et al. <cit.> proposed a voxelwise boosting approach for the analysis of contrast-enhanced magnetic resonance imaging data (DCE-MRI), which was additionally enhanced to account for the regional structure of the voxels via a spatial penalty. 

Pybus et al. <cit.> proposed a hierarchical boosting algorithm for classification in an approach to detect positive selection in genomic regions (cf., <cit.>). Truntzer et al. <cit.> compared the classification performance of gradient boosting with other methods combining clinical variables and high-dimensional mass spectrometry data and concluded that the variable selection properties of boosting led also to a very good performance regarding prediction accuracy. 

Regarding boosting location and scale models (modelling both expected value and variance in the spirit of GAMLSS <cit.>), Messner et al. <cit.> proposed a boosting algorithm for predictor selection in ensemble postprocessing to better calibrate ensemble weather forecasts. The idea of ensemble forecasting is to account for model errors and to quantify forecast uncertainty. Mayr et al. <cit.> used boosted location and scale models in combination with permutation tests to assess simultaneously systematic bias and random measurement errors of medical devices. The use of a permutation test tackles one of the remaining problems of statistical boosting approaches in practical biomedical research: The lack of standard errors for effect estimates makes it necessary to incorporate resampling procedures to construct confidence intervals or to assess significance of effects. 

The methodological development in <cit.>, analogously to many of the extensions presented in this article, was motivated by the applied analysis of biomedical data. Statistical boosting algorithms, however, have been applied over the last few years in various biomedical applications without the need for methodological extensions that could be described here. Most application focus on prediction modelling or variable selection. We want to briefly mention a selection of the most recent ones from the last two years: The different research questions comprise the development of birth weight prediction formulas <cit.> for particularly small babies, the prediction of smoking cessation and its relapse in HIV-infected patients <cit.>, Escherichia coli Fed-Batch Fermentation Modeling <cit.>, the prediction of cardiovascular death on older patients in the emergency department <cit.> and the identification of factors influencing therapeutic decisions regarding rheumatoid arthritis <cit.>.




§ DISCUSSION


After Friedman et al. <cit.> discovered the link between boosting and additive modelling in their seminal paper, most research on boosting methods has been focused on the development of methodology within the univariate GAM framework. This line of research included, among many other achievements, the estimation of smooth  predictor effects via spline base-learners <cit.> and the extension of boosting  to other GAM families than binary classification and Gaussian regression <cit.>. We have summarized these methods and described their relationships in an earlier review <cit.>.

In this article, we have highlighted several new research areas in the field of statistical boosting leaving the traditional GAM modeling approach. A particularly active research area during the last few years addresses the development of boosting algorithms for new model classes extending the GAM framework. These include, among others, the simultaneous modelling of location, scale and shape parameters within the GAMLSS framework <cit.>, the modelling of functional data <cit.>, and, recently, the class of joint models for longitudinal  and survival data <cit.>. It goes without saying that these developments will make boosting algorithms available for practical use in much more sophisticated clinical and epidemiological applications than before.

Another line of research, which we described in detail in Sections <ref> and <ref>, aims at exploring the connections between statistical boosting methods and machine learning techniques that were originally developed independently of boosting. An important example is stability selection, a generic methodology that, at the time of its development, mainly focussed on penalized regression models such as the lasso. Only in recent years, stability selection has been adapted to become a tool for variable selection within the boosting framework (e.g. <cit.>). Other work in this context is the analysis of the connections between boosting and penalized regression <cit.> and the work by Sariyar et al. <cit.> exploring a combination of boosting and random forest methods.

Finally, as already noted by Hothorn <cit.>, boosting may not solely be regarded as a framework for regularized model fitting but also as a generic optimization tool on its own right. In particular, boosting constitutes a robust algorithm for the optimization of objective functions that, due to their structure or complexity, may pose problems for Newson-Raphson-type and related methods. This was, for example, the motivation for the use of boosting in the articles by Hothorn et al. <cit.> and Weinhold et al. <cit.>.


Regarding future research, a huge challenge for the use of boosting algorithms in biomedical applications arises from the era of big data. Unlike other machine learning methods like random forests, the sequential nature of boosting methods hampers the use of parallelization techniques within the algorithm, which may result in issues with the fitting and tuning of complex models with multidimensional predictors and/or sophisticated base-learners like splines or higher-sized trees. To overcome these problems in classification and univariate regression, Chen and Guestrin <cit.> developed the extremely fast and sophisticated xgboost environment. However, for the more recent extensions discussed in this paper, big data solutions for statistical boosting still have to be developed.



 §.§ Conflict of interests


The authors declare that there is no conflict of interest regarding the publication of this
paper.



 §.§ Acknowledgements


The authors thank Corinna Buchstaller for her help with the literature search. The work on this article was supported by the Deutsche Forschungsgemeinschaft (DFG) (), grant no. SCHM 2966/1-2 (grant to MS and OG) and the Interdisciplinary Center for Clinical Research (IZKF) of the Friedrich-Alexander-University Erlangen-Nürnberg via the Projects J49 (grant to AM) and J61 (grant to EW).







§ DEVELOPMENTS REGARDING THE MBOOST PACKAGE
 

This appendix describes important changes during the last years that were implemented in the  package mboost after the tutorial paper <cit.> on its use was published. 

Starting from mboost 2.2, the default for the degrees of freedom was changed, they are now defined as 

    df(λ) = trace(2S - S^⊤S),

with smoother matrix S = X(X^⊤X + λ K)^-1 X. Analyses have shown, that  this leads to a reduced selection bias, see <cit.>. Earlier versions used the trace of the smoother matrix as degrees of freedom, i.e., 
df(λ) = trace(S). One can change to the old definition 
by setting . For parallel computations of cross-validated stopping values, mboost now uses the package parallel, which is included in the standard  installation. The behavior of  was changed when  is a factor: the intercept is simply dropped from the design matrix and the coding can be specified as usual for factors. Additionally, a new contrast was introduced:  (see the manual of  for details). Finally, the computation of B-spline basis at the boundaries was changed such that equidistant boundary knots are used per default.

With mboost 2.3, constrained effects  <cit.>
are fitted per default using quadratic programming methods (option ) 
improving the speed of computation drastically. Additional to monotonic, convex and concave
effects, new constraints were introduced to fit  or  effects or effects with boundary constraints (see  for details). Additionally, a new function to assign m_stop values to a model 
object was added () as well as  two new distribution families  <cit.> and  <cit.>. Finally, a new option was implemented to allow for stopping based on out-of-bag data during fitting (via 
).


With mboost 2.4, bootstrap confidence intervals were implemented in the 
novel  function <cit.>. The stability
selection procedure was moved to a dedicated package stabs <cit.>, 
while a specific function for gradient boosting was implemented in package 
mboost.

From mboost 2.5 onward, cross-validation does not stop on errors in single 
folds anymore and was sped up by setting  if parallel
computations via  are used. A documentation for the function
 was added, which allows to visualize model results. Values outside
the boundary knots are now forbidden during fitting, while linear extrapolation 
is used for prediction. 

With mboost 2.6 a lot of bug fixes and small improvements were provided. 
Most notably, the development of the package is now hosted entirely on github in 
the collaborative project https://github.com/boost-R/mboost/boost-R/mboost 
and the package maintainer changed.

The current CRAN version mboost 2.7 provides a new family 
 <cit.>, variable importance measures () 
and improved plotting facilities.

Changes in the current development version which will be deployed to CRAN with the
next release of mboost include major changes to distribution families, allowing to
specify link functions. The  family will additionally
provide an alternative implementation of Binomial regression models along the lines 
of the classic  implementation, which can be used via . 
This family also works with a two-column matrix containing the number of successes 
and number of failures. Furthermore, models with zero steps (i.e., models containing 
only the offset) will be supported as well as cross-validated models without 
base-learners.



