biased (), has multiple leaves () or is some other function . Thus we have derived the AdaBoost algorithm: At each iteration, choose the classifier , which minimizes the total weighted error , use this to calculate the error rate , use this to calculate the weight , and finally use this to improve the boosted classifier to . Statistical understanding of boosting Boosting is a form of linear regression in which the features of each sample are the outputs of some weak learner applied to . While regression tries to fit to as precisely as possible without loss of generalization, typically using least square error , whereas the AdaBoost error function takes into account the fact that only the sign of the final result is used, thus can be far larger than 1 without increasing error. However, the exponential increase in the error for sample as increases. resulting in excessive weights being assigned to outliers. One feature of the choice of exponential error function is that the error of the final additive model is the product of the error of each stage, that is, . Thus it can be seen that the weight update in the AdaBoost algorithm is equivalent to recalculating the error on after each stage. There is a lot of flexibility allowed in the choice of loss function. As long as the loss function is monotonic and continuously differentiable, the classifier is always driven toward purer solutions. Zhang (2004) provides a loss function based on least squares, a modified Huber loss function: This function is more well-behaved than LogitBoost for close to 1 or -1, does not penalise ‘overconfident’ predictions (), unlike unmodified least squares, and only penalises samples misclassified with confidence greater than 1 linearly, as opposed to quadratically or exponentially, and is thus less susceptible to the effects of outliers. Boosting as gradient descent Boosting can be seen as minimization of a convex loss function over a convex set of functions. Specifically, the loss being minimized by AdaBoost is the exponential loss , whereas LogitBoost performs logistic regression, minimizing . In the gradient descent analogy, the output of the classifier for each training point is considered a point in n-dimensional space, where each axis corresponds to a training sample, each weak learner corresponds to a vector of fixed orientation and length, and the goal is to reach the target point (or any region where