Title: Stacking and stability,
Abstract: Stacking is a general approach for combining multiple models toward greater
predictive accuracy. It has found various application across different domains,
ensuing from its meta-learning nature. Our understanding, nevertheless, on how
and why stacking works remains intuitive and lacking in theoretical insight. In
this paper, we use the stability of learning algorithms as an elemental
analysis framework suitable for addressing the issue. To this end, we analyze
the hypothesis stability of stacking, bag-stacking, and dag-stacking and
establish a connection between bag-stacking and weighted bagging. We show that
the hypothesis stability of stacking is a product of the hypothesis stability
of each of the base models and the combiner. Moreover, in bag-stacking and
dag-stacking, the hypothesis stability depends on the sampling strategy used to
generate the training set replicates. Our findings suggest that 1) subsampling
and bootstrap sampling improve the stability of stacking, and 2) stacking
improves the stability of both subbagging and bagging.