Title: LinXGBoost: Extension of XGBoost to Generalized Local Linear Models,
Abstract: XGBoost is often presented as the algorithm that wins every ML competition.
Surprisingly, this is true even though predictions are piecewise constant. This
might be justified in high dimensional input spaces, but when the number of
features is low, a piecewise linear model is likely to perform better. XGBoost
was extended into LinXGBoost that stores at each leaf a linear model. This
extension, equivalent to piecewise regularized least-squares, is particularly
attractive for regression of functions that exhibits jumps or discontinuities.
Those functions are notoriously hard to regress. Our extension is compared to
the vanilla XGBoost and Random Forest in experiments on both synthetic and
real-world data sets.