Title: $\left( β, \varpi \right)$-stability for cross-validation and the choice of the number of folds,
Abstract: In this paper, we introduce a new concept of stability for cross-validation,
called the $\left( \beta, \varpi \right)$-stability, and use it as a new
perspective to build the general theory for cross-validation. The $\left(
\beta, \varpi \right)$-stability mathematically connects the generalization
ability and the stability of the cross-validated model via the Rademacher
complexity. Our result reveals mathematically the effect of cross-validation
from two sides: on one hand, cross-validation picks the model with the best
empirical generalization ability by validating all the alternatives on test
sets; on the other hand, cross-validation may compromise the stability of the
model selection by causing subsampling error. Moreover, the difference between
training and test errors in q\textsuperscript{th} round, sometimes referred to
as the generalization error, might be autocorrelated on q. Guided by the ideas
above, the $\left( \beta, \varpi \right)$-stability help us derivd a new class
of Rademacher bounds, referred to as the one-round/convoluted Rademacher
bounds, for the stability of cross-validation in both the i.i.d.\ and
non-i.i.d.\ cases. For both light-tail and heavy-tail losses, the new bounds
quantify the stability of the one-round/average test error of the
cross-validated model in terms of its one-round/average training error, the
sample sizes $n$, number of folds $K$, the tail property of the loss (encoded
as Orlicz-$\Psi_\nu$ norms) and the Rademacher complexity of the model class
$\Lambda$. The new class of bounds not only quantitatively reveals the
stability of the generalization ability of the cross-validated model, it also
shows empirically the optimal choice for number of folds $K$, at which the
upper bound of the one-round/average test error is lowest, or, to put it in
another way, where the test error is most stable.