Choice of V for V-Fold Cross-Validation in Least-Squares Density Estimation

Abstract: This paper studies V-fold cross-validation for model selection in least-squares density estimation. The goal is to provide theoretical grounds for choosing V in order to minimize the least-squares loss of the selected estimator. We first prove a non-asymptotic oracle inequality for V-fold cross-validation and its bias-corrected version (V-fold penalization). In particular, this result implies that V-fold penalization is asymptotically optimal in the nonparametric case. Then, we compute the variance of V-fold cross-validation and related criteria, as well as the variance of key quantities for model selection performance. We show that these variances depend on V like 1+4/(V-1), at least in some particular cases, suggesting that the performance increases much from V=2 to V=5 or 10, and then is almost constant. Overall, this can explain the common advice to take V=5---at least in our setting and when the computational power is limited---, as supported by some simulation experiments. An oracle inequality and exact formulas for the variance are also proved for Monte-Carlo cross-validation, also known as repeated cross-validation, where the parameter V is replaced by the number B of random splits of the data.

\section{Introduction} \label{sec.intro}
Cross-validation methods are widely used in statistics,
for estimating the risk of a given statistical estimator \cite{Sto:1974,All:1974,Gei:1975} and for selecting among a family of estimators.
For instance, cross-validation can be used for model selection, where a collection of linear spaces is given (the models) and the problem is to choose the best least-squares estimator over one of these models.
We refer to \cite{Arl_Cel:2010:surveyCV} for more references about cross-validation for model selection.

\medskip

Then, a natural question arises: which cross-validation method should be used for minimizing the risk of the selected estimator?
For instance, a popular family of cross-validation methods is $V$-fold cross-validation \cite[often called $k$-fold cross-validation]{Gei:1975}, which depends on an integer parameter $V$, and enjoys a smaller computational cost than other classical cross-validation methods.
The question becomes (1) which $V$ is optimal, and (2) can we do almost as well as the optimal $V$ with a small computational cost, that is, a small $V\,$?
Answering the second question is particularly useful for practical applications where the computational power is limited.

Surprisingly, few theoretical results exist for answering these two questions, especially with a non asymptotic point of view \cite{Arl_Cel:2010:surveyCV}.
In short, it is proved in least-squares regression that at first order, $V$-fold cross-validation is suboptimal for model selection if $V$ stays bounded, because $V$-fold cross-validation is biased \cite{Arl:2008a}.
When correcting for the bias \cite{Bur:1989,Arl:2008a}, we recover asymptotic optimality whatever $V$, but without any theoretical result distinguishing among values of $V$ in second order terms in the risk bounds \cite{Arl:2008a}.

Intuitively, if there is no bias, increasing $V$ should reduce the variance of the $V$-fold cross-validation estimator of the risk, hence reduce the risk of the final estimator, as confirmed by some simulation experiments \cite[for instance]{Arl:2008a}.
But variance computations for unbiased $V$-fold methods have only been made in a very specific regression setting, and they are asymptotic \cite{Bur:1989}.

\medskip

This paper aims at providing theoretical grounds for the choice of $V$ by two means: a non-asymptotic oracle inequality valid for any $V$ (Section~\ref{sec.oracle}) and exact variance computations shedding light on the influence of $V$ on the variance (Section~\ref{sec.variance}).
In particular, we would like to understand why the common advice in the literature is to take $V=5$ or $10$, based on simulation experiments \cite[for instance]{Has_Tib_Fri:2001v2009}.

The results of the paper are proved in the least-squares density estimation framework, because, we can then benefit from explicit closed-form formulas and simplifications for the $V$-fold criteria.
In particular, we show $V$-fold cross-validation and all leave-$p$-out methods are particular cases of $V$-fold penalties in least-squares density estimation (Lemma~\ref{le.penVF-VFCV}).

The first main contribution of the paper (Theorem~\ref{thm.oracle-penVF.cas_reel}) is an oracle inequality with leading constant $1+\varepsilon_n$ with $\varepsilon_n \to 0$ as $n\to \infty$ (for unbiased $V$-fold methods) that holds for any value of $V$.
To the best of our knowledge, Theorem~\ref{thm.oracle-penVF.cas_reel} is the first non asymptotic oracle inequality for $V$-fold methods enjoying such properties: the leading constant $1+o(1)$ is new in density estimation, and the fact that it holds whatever the value of $V$ had never been obtained in any framework.
Theorem~\ref{thm.oracle-penVF.cas_reel} relies on a new concentration inequality for the $V$-fold penalty (Proposition~\ref{prop:concpenvf}).
The second main contribution of the paper (Theorem~\ref{theo.variance.penVF}) is the first non asymptotic variance computation for $V$-fold criteria that allows to understand precisely how the {\em model selection performance} of $V$-fold cross-validation or penalization depends on $V$.
Previous results only focused on the variance of the $V$-fold criterion \cite{Bur:1989,Ben_Gra:2005, Cel:2008:phd, Cel:2008,Cel_Rob:2006}, which is not sufficient for our purpose, as explained in Section~\ref{sec.oracle.key-quant}; see also the remarks after Theorem~\ref{theo.variance.penVF}.
In our setting, we can explain theoretically why taking, say, $V>10$ is not necessary for getting a performance close to the optimum,
as confirmed by experiments on synthetic data in Section~\ref{sec.simus}.

\paragraph{Notation}
For any integer $k\geq 1$, $\inter{k}$ denotes $\set{1,\ldots,k}$, and for any $B\subset \inter{n}$, $\xi_{B}$ denotes $\set{\xi_{i},i\in B}$, $|B|$ its cardinality and $B^{c}=\inter{n}\setminus B$.
For any real numbers $t,u$, we define $t \vee u \defegal \max\sset{t,u}$, $u_+ \egaldef u \vee 0$ and $u_-  \egaldef (-u) \vee 0$.
All asymptotic results and notation $o(\cdot)$ or $O(\cdot)$ are for the regime when the number $n$ of observations tends to infinity.
\\
All references to the supplementary material (e.g., to sections, equations or figures) are of the form S.$x$, and references to the appendix are of the form A.$x$.

\section{Least-squares density estimation and definition of $V$-fold procedures} \label{sec.cadre}
This section introduces the framework of the paper, the main procedures studied, and some useful notation.

\subsection{General statistical framework} \label{sec.cadre.general}
Let $\xi,\xi_1,...,\xi_n$ be independent random variables taking value in a Polish space $\mathcal{X}$, with common distribution $P$ and density $\bayes$ with respect to some known measure $\mu$. Suppose that $ \bayes \in L^{\infty}(\mu)$ so $\bayes\in L^2(\mu)$.
The goal is to estimate $\bayes$ from  $\xi_{\inter{n}}=(\xi_1,\ldots,\xi_n)$, that is, to build an estimator $\ERM = \ERM(\xi_{\inter{n}}) \in L^2(\mu)$ such that its loss $\perte{\ERM}$ is as small as possible, where for any $t \in L^2(\mu)$, $\norm{t}^2\egaldef \int_{\mathcal{X}} t^2 d\mu$.

Projection estimators are among the most classical estimators in this framework, see for example \cite{Dev_Lor:1993, Mas:2003:St-Flour}.
Given a separable linear subspace $S_m$ of $L^2(\mu)$ (called a model), the projection estimator of $\bayes$ onto $S_m$ is defined by
\begin{equation}
\label{def.ERM}
\ERM_m \egaldef \mathop{\argmintmp}_{t\in S_m} \set{ \norm{t}^2 - 2 P_n (t) } \enspace ,
\end{equation}
where $P_n$ is the empirical measure; for any function $t \in L^2(\mu)$, $P_n(t) = \int t dP_n = n^{-1} \sum_{i=1}^n t\paren{\xi_i} $.
The quantity minimized in the definition of $\ERM_m$ is often called the empirical risk, and can be denoted by
\[ P_n \gamma(t) =  \norm{t}^2 - 2 P_n (t) \qquad \mbox{where} \quad \forall x \in \mathcal{X} \, , \, \forall t \in L^2(\mu) \, , \quad \gamma(t;x)=\norm{t}^2 - 2 t(x) \enspace . \]
The function $\gamma$ is called the least-squares contrast.
Note that $S_m \subset L^1(P)$ since $\bayes \in L^2(\mu)$.

\subsection{Model selection} \label{sec.cadre.modsel}
When a finite collection of models $(S_m)_{m \in \M_n}$ is given, following \cite{Mas:2003:St-Flour}, we want to choose from data one among the corresponding projection estimators $(\ERM_m)_{m \in \M_n}$.
The goal is to design a model selection procedure $\widehat{m}: \mathcal{X}^n \mapsto \mathcal{M}_n$ so that the final estimator $\tilde{s}\egaldef\ERM_{\widehat{m}}$ has a quadratic loss as small as possible, that is, comparable to the oracle loss $\inf_{m \in \M_n} \perte{\ERM_m}$.
More precisely, we aim at proving an oracle inequality of the form
\[
\perte{\ERM_{\widehat{m}}} \leq C_n \inf_{m \in \M_n} \set{ \perte{\ERM_m} } + R_n
\]
with large probability.
The procedure $\widehat{m}$ is called asymptotically optimal when $R_n$ is negligible in front of the oracle loss and $C_n\to 1$.

\medskip

In this paper, we focus on model selection procedures of the form
\[
\widehat{m} \egaldef \mathop{\argmintmp}_{m \in \M_n}\set{\crit(m)} \enspace,
\]
where $\crit:\mathcal{M}_n \mapsto \R$ is some data-driven criterion.
Since our goal is to satisfy an oracle inequality, an ideal criterion is
\[
\critid(m) = \perte{\ERM_m}-\norm{\bayes}^2 = -2P(\ERM_m)+\norm{\ERM_m}^2 = P \gamma\paren{\ERM_m} \enspace .
\]

Penalization is a popular way of designing a model selection criterion \cite{Bar_Bir_Mas:1999,Mas:2003:St-Flour}:
\[ \crit(m) = P_n \gamma\paren{\ERM_m} + \pen(m)  \]
for some penalty function $\pen: \mathcal{M}_n \flens \R$, possibly data-driven.
From the ideal criterion $\critid$, we get the ideal penalty
\begin{align}
\label{def.penid}
\penid(m) &\egaldef \critid(m) - P_n\gamma\paren{\ERM_m}
= (P-P_n) \gamma\paren{\ERM_m}
= 2(P_n-P)(\ERM_m)
\\
\notag
&= 2(P_n-P)(\ERM_m-\bayes_m)+2(P_n-P)(\bayes_m)
= 2 \norm{ \ERM_m-\bayes_m }^2 +2(P_n-P)(\bayes_m)
 \enspace ,
\\
\mbox{where}
\quad
\bayes_m &\egaldef
\mathop{\argmintmp}_{t \in S_m} \set{ P\gamma(t)}
= \mathop{\argmintmp}_{t\in S_m} \set{\perte{t}}
\notag
\end{align}
is the orthogonal projection of $\bayes$ onto $S_m$ in $L^2(\mu)$.
Let us finally recall some useful and classical reformulations of the main term in the ideal penalty \eqref{def.penid}, that proves in particular the last equality in Eq.~\eqref{def.penid}:
If $\mathbb{B}_m = \set{t\in S_m \telque \norm{t}\leq 1}$ and $(\psil)_{\lamm}$ denotes an orthonormal basis of $S_m$ in $L^2(\mu)$, then,
\begin{equation}
\label{eq.p1.4formules}
\begin{split}
(P_n-P)(\ERM_m-\bayes_m)
&= \sum_{\lamm} \croch{(P_n-P)(\psil)}^2
\\
&= \norm{ \ERM_m-\bayes_m }^2
= \sup_{t\in \mathbb{B}_{m}}[(P_{n}-P)t]^{2}
\enspace ,
\end{split}
\end{equation}
where the last equality follows from Eq.~\eqref{eq.sup-CS}.

\subsection{$V$-fold cross validation} \label{sec.cadre.VFCV}
A standard approach for model selection is cross-validation.
We refer the reader to \cite{Arl_Cel:2010:surveyCV} for references and a complete survey on cross-validation for model selection.
This section only provides the minimal definitions and notation necessary for the remainder of the paper.

For any subset $A\subset \inter{n}$, let
\begin{gather*}
P_{n}^{(A)} \egaldef \frac{1}{|A|}\sum_{i\in A}\delta_{\xi_{i}}
\quad \mbox{and} \quad
\ERM_{m}^{(A)} \egaldef \mathop{\argmintmp}_{t\in S_m} \set{ \norm{t}^2 - 2 P^{(A)}_n (t) }
\enspace .
\end{gather*}
The main idea of cross-validation is data splitting: some $T\subset \inter{n} $ is chosen, one first trains $\ERM_m(\cdot)$ with $\xi_{ T}$, then test the trained estimator on the remaining data $\xi_{ T^c}$.
The hold-out criterion is the estimator of $\critid(m)$ obtained with this principle
\begin{equation}\label{def:crit.HO}
\crit_{\ensuremath{\mathrm{HO}}}(m, T) \egaldef P_{n}^{(T^{c})} \gamma\paren{\ERM_m^{(T)}} = -2P_{n}^{(T^{c})} \paren{ \ERM_{m}^{(T)}} +\norm{\ERM^{(T)}_{m}}^{2}\enspace ,
\end{equation}
and all cross-validation criteria are defined as averages of hold-out criteria with various subsets $T$.

Let $V \leq n$ be a positive integer and let $\mathcal{B}=\mathcal{B}_{\inter{V}}=(\mathcal{B}_1,\ldots,\mathcal{B}_V)$ be some partition of $\inter{n}$.
The $V$-fold cross validation criterion is defined by
\[ \critVF(m,\mathcal{B}) \egaldef \frac{1}{V} \sum_{K=1}^{V}\crit_{\ensuremath{\mathrm{HO}}}(m, \mathcal{B}_K^c) \enspace . \]
Compared to the hold-out, one expects cross-validation to be less variable thanks to the averaging over $V$ splits of the sample into $\xi_{ \mathcal{B}_K}$ and $\xi_{ \mathcal{B}_K^c}$.

Since $\critVF(m,\mathcal{B})$ is known to be a biased estimator of $\E\croch{\critid(m)}$, Burman \cite{Bur:1989} proposed the bias-corrected $V$-fold cross-validation criterion
\begin{align*}
\critcorrVF(m, \mathcal{B})
&\egaldef \critVF(m,\mathcal{B})+ P_n \gamma\paren{ \ERM_m } - \frac{1}{V} \sum_{K=1}^{V} { P_n\gamma\paren{ \ERM_{m}^{(\mathcal{B}_K^{c})} }  }
\enspace .
\end{align*}
This criterion is studied in \cite[Sec. 7.2.1, p.204--205]{Mas:2003:St-Flour} in the particular case where  $V=n$ under the name cross-validation estimator (see in Lemma~\ref{le.penVF-VFCV}).

\subsection{Resampling-based and $V$-fold penalties} \label{sec.cadre.penVF}
Another approach for building general data-driven model selection criteria is penalization with a resampling-based estimator of the expectation of the ideal penalty, as proposed by Efron \cite{Efr:1983} with the bootstrap and recently generalized to all resampling schemes \cite{Arl:2009:RP}.
Let $W \sim \mathcal{W}$ be some random vector of $\R^n$ independent from $\xi_{\inter{n}}$ with $n^{-1}\sum_{i=1}^nW_i=1$, and denote by $P_n^W =  n^{-1}\sum_{i=1}^n W_i\delta_{\xi_i}$ the weighted empirical distribution of the sample.
Then, the resampling-based penalty associated with $\mathcal{W}$ is defined as
\begin{equation}\label{def.pen.Res}
\pen_{\mathcal{W}}(m) \egaldef C_{\mathcal{W}} \E_W \croch{ (P_n - P_n^W) \gamma\paren{\ERM_m^W} }\enspace,
\end{equation}
where $\ERM_m^W \in \mathop{\argmintmp}_{t \in S_m} \set{ P_n^W \gamma\paren{t}}$, $\E_W\croch{\cdot}$ denotes the expectation with respect to $W$ only (that is, conditionally to the sample $\xi_{\inter{n}}$), and $C_{\mathcal{W}}$ is some positive constant.
Resampling-based penalties have been studied recently in the least-squares density estimation framework \cite{Le09}, assuming $W$ is exchangeable, i.e., its distribution is invariant by any permutation of its coordinates.

Since computing exactly $\pen_{\mathcal{W}}(m)$ has a large computational cost in general for exchangeable $W$, some non-exchangeable resampling schemes were introduced in \cite{Arl:2008a}, inspired by $V$-fold cross-validation:
given some partition $\mathcal{B}=\mathcal{B}_{\inter{V}}$ of $\inter{n}$, the weight vector $W$ is defined by $W_i=(1-\card(\mathcal{B}_J)/n)^{-1}\ensuremath{\mathbf{1}}_{i\notin \mathcal{B}_J}$ for some random variable $J$ with uniform distribution over $\inter{V}$.
Then, $P_n^W = P_n^{(\mathcal{B}_J^{c})}$ so that the associated resampling penalty, called {\em $V$-fold penalty}, is defined by
\begin{align} \label{eq.def.penVF}
\penVF(m , \mathcal{B} , x) &\egaldef \frac{x}{V} \sum_{K=1}^V  \croch{ \paren{P_n - P_n^{(\mathcal{B}_K^{c})} } \gamma\paren{ \ERM_m^{(\mathcal{B}_K^{c})}} }\nonumber\\
&= \frac{2 x}{V} \sum_{K=1}^V  { \paren{P_n^{(\mathcal{B}_K^{c})} - P_n } \paren{ \ERM_m^{(\mathcal{B}_K^{c})}} }
\end{align}
where $x>0$ is left free for flexibility, which is quite useful according to Lemma~\ref{le.penVF-VFCV}.

\subsection[Links between $V$-fold penalties and other procedures]{Links between $V$-fold penalties, resampling penalties and (corrected) $V$-fold cross-validation} \label{sec.cadre.links}
In this paper, we focus our study on $V$-fold penalties because formula \eqref{eq.def.penVF} covers all $V$-fold and resampling-based procedures mentioned in Sections \ref{sec.cadre.VFCV} and \ref{sec.cadre.penVF}.

First, when $V=n$, the only possible partition is $\mathcal{B}_{{\rm LOO}}=\set{\set{1}, \ldots, \set{n}}$, and the $V$-fold penalty is called the leave-one-out penalty $\penLOO(m,x)\egaldef \penVF(m,\mathcal{B}_{{\rm LOO}},x)$.
The associated weight vector $W$ is exchangeable, hence Eq.~\eqref{eq.def.penVF} leads to all exchangeable resampling penalties since they are all equal up to a deterministic multiplicative factor in the least-squares density estimation framework, as proved in \cite{Le09}.

For $V$-fold methods, let us assume $\mathcal{B}$ is a regular partition of $\inter{n}$, that is,
\begin{equation}
\label{hyp.part-reg.exact}
\tag{\bf Reg}
V = |\mathcal{B}| \mbox{ divides } n\quad \mbox{and}\quad \forall K \in \inter{V} \, , \; \absj{\mathcal{B}_K} = \frac{n}{V}
\enspace .
\end{equation}
Then, we get the following connection between $V$-fold penalization and cross-validation methods.
\begin{lemma} \label{le.penVF-VFCV}
In least-squares density estimation, under assumption \eqref{hyp.part-reg.exact},
\begin{align}
\label{eq.le.penVF-corrVFCV}
\critcorrVF(m,\mathcal{B}) &=  P_n\gamma\paren{\ERM_m} + \penVF\paren{ m,\mathcal{B},V-1 }
\\
\label{eq.le.penVF-VFCV}
\critVF(m,\mathcal{B}) &=  P_n\gamma\paren{\ERM_m} + \penVF\paren{ m,\mathcal{B},V-\frac{1}{2} }
\\
\label{eq.le.penLPO-LPO}
\critLPO(m,p) &=  P_n\gamma\paren{\ERM_m} +  \penLPO\paren{ m,p,\frac{n}{p}-\frac{1}{2} }
\\
\label{eq.le.penLOO-LPO}
&= P_n\gamma\paren{\ERM_m} +    \penLOO\paren{ m,(n-1)\frac{n/p - 1/2}{ n/p - 1} }\\
\notag
&= P_n\gamma\paren{\ERM_m} +    \penVF\paren{ m,\mathcal{B}_{{\rm LOO}},(n-1)\frac{n/p - 1/2}{ n/p - 1} }
\end{align}
where for any $p \in \inter{n-1}$, the leave-$p$-out cross-validation criterion is defined by
\begin{multline*}
\critLPO(m,p) \egaldef \frac{1}{|\mathcal{E}_p|} \sum_{A \in \mathcal{E}_p} P_n^{(A)} \gamma\paren{\ERM_m^{(A^{c})}}\mbox{with} \quad \mathcal{E}_p \egaldef \set{A \subset \inter{n} \telque |A|=p}
\end{multline*}
and the leave-$p$-out penalty is defined by
\[ \forall x >0 \, , \quad \penLPO(m,p,x) \egaldef \frac{x}{|\mathcal{E}_p|} \sum_{A \in \mathcal{E}_p} (P_n - P_n^{(A^{c})}) \gamma\paren{\ERM_m^{(A^{c})}} \enspace . \]
\end{lemma}
Lemma~\ref{le.penVF-VFCV} is proved in Section~\ref{sec.app.proof.lemmeLPO}.
\begin{remark} \label{rk.le.penVF-VFCV.Alain}
Eq.~\eqref{eq.le.penVF-corrVFCV} was first proved in \cite{Arl:2008a} in a general framework that includes least-squares density estimation, assuming only \eqref{hyp.part-reg.exact}.
Eq.~\eqref{eq.le.penLOO-LPO} follows from Lemma~6.11 in \cite{Le09} since $\penLPO$ belongs to the family of exchangeable resampling penalties, with weights $W_i\egaldef(1-p/n)^{-1}\ensuremath{\mathbf{1}}_{i\notin A}$ and $A$ is randomly chosen uniformly over $\mathcal{E}_p$.
It can also be deduced from \cite[Proposition~3.1]{Cel:2008}, see Section~\ref{sec.app.proof.lemmeLPO}.
\end{remark}
\begin{remark} \label{rk.le.penVF-VFCV.Pascal}
It is worth mentioning the cross-validation estimators given in \cite[Chapter~7]{Mas:2003:St-Flour}.
First, the unbiased cross-validation criterion defined in \cite[Section 7.2.1]{Mas:2003:St-Flour} is exactly $\critcorrVF(m,\mathcal{B}_{{\rm LOO}})$.
Then, the penalized estimator of \cite[Theorem 7.6]{Mas:2003:St-Flour} is the estimator selected by the penalty
 \[\penLOO \paren{ m, \frac{(1+\epsilon)^6(n-1)^2}{2 [ n-(1+\epsilon)^6]} } \]
for some $\epsilon>0$ such that $(1+\epsilon)^6<n$  (see Section~\ref{sec.app.proof.lemmeLPO} for details).
\end{remark}
As a conclusion of this section, in the least-squares density estimation framework and assuming only \eqref{hyp.part-reg.exact}, Lemma~\ref{le.penVF-VFCV} shows it is sufficient to study $V$-fold penalization with a free multiplicative factor $x$ in front of the penalty for studying also $V$-fold cross-validation ($x=V-1/2$), corrected $V$-fold cross-validation ($x=V-1$), the leave-$p$-out ($V=n$ and $x=(n-1)(n/p-1/2)/(n/p-1)$) and all exchangeable resampling penalties.
For any $C>0$ and $\mathcal{B}$ some partition of $\inter{n}$, taking $x=C(V-1)$, the $V$-fold penalization criterion with is denoted by
\begin{equation}
\label{eq.crit-penVF-gal}
\CV_{(C,\mathcal{B})}(m) \egaldef P_n\gamma\paren{\ERM_m}+\penVF(m,\mathcal{B},C(V-1))
\enspace .
\end{equation}
A key quantity in our results is the bias $\E\scroch{\CV_{(C,B)}(m)-\critid(m)}$.
From \lemref{exact.formula.penvf} in Section~\ref{sect:proofthmconcpenvf}, we have
\begin{align}\label{eq:ExpPenid}
 \E\croch{\penVF(m,\mathcal{B},V-1)}=\E\croch{\penid(m)}=2\E\croch{\norm{\ERM_m-\bayes_m}^2}\enspace,
\end{align}
so that for any $C>0$,
\begin{equation}
\label{eq.biais.critpenVF}
\E\croch{ \CV_{(C,\mathcal{B})}(m) - \critid(m) }
= 2 (C-1) \E\croch{\norm{\ERM_m-\bayes_m}^2}
\enspace .
\end{equation}
In Sections~\ref{sec.oracle}--\ref{sec.algo}, we focus our study on $V$-fold methods, that is, we study the performance of the $V$-fold penalized estimators $\ERM_{\widehat{m}}$, defined by
\begin{equation}
\label{eq.mh-crit-penVF-gal}
\widehat{m} = \widehat{m}(\CV_{(C,\mathcal{B})}) =\mathop{\argmintmp}_{m \in \M_n}\set{ \CV_{(C,\mathcal{B})}(m) }
\enspace,
\end{equation}
for all values of $V$ and $C>1/2$.
Additional results on hold-out (penalization) are given in Section~\ref{sec.discussion.hold-out} to complete the picture.

\section{Oracle inequalities} \label{sec.oracle}
In this section, we state our first main result, that is, a non-asymptotic oracle inequality satisfied by $V$-fold procedures.
This result holds for any $V \in \inter{n}$, any constant $x=C (V-1)$ in front of the penalty with $C>1/2$, and provides an asymptotically optimal oracle inequality for the selected estimator when $C\to1$.
In addition, as proved by Section~\ref{sec.cadre.links}, it implies oracle inequalities satisfied by leave-$p$-out procedures for all $p$.

\subsection{Concentration of $V$-fold penalties}
Concentration is the key property to establish oracle inequalities. Let us start with some concentration results for $V$-fold penalties.
\begin{proposition}\label{prop:concpenvf}
Let $\xi_{\inter{n}}$ be i.i.d. real-valued random variables with density $\bayes\in L^{\infty}(\mu)$, $\mathcal{B}$ some partition of $\inter{n}$ into $V$ pieces satisfying \eqref{hyp.part-reg.exact}, $S_m$ a separable linear space of measurable functions and $(\psil)_{\lambda\in\Lambda_m}$ an  orthonormal basis of $S_m$.
Define $\mathbb{B}_m=\set{t\in S_m \telque \norm{t}\leq 1}$, $\Psi_m=\sum_{\lambda\in\Lambda_m}\psil^{2}=\sup_{t\in\mathbb{B}_m}t^2$,
$b_{m}\egaldef\norm{\sqrt{\Psi_m}}_\infty$,
\[
\mathcal{D}_{m}\egaldef P\Psi_m-\norm{\bayes_{m}}^{2}
=n\Ex{\norm{\bayes_m-\ERM_m}^2}
\]
where $\ERM_m$ is defined by Eq.~\eqref{def.ERM},
and for any $x,\epsilon>0$, let
\begin{equation*}
\rho_1\paren{m,\epsilon,\bayes,x,n}
\egaldef \frac{\norm{\bayes}_{\infty} x}{ \epsilon n}+\frac{(b_m^2+\norm{\bayes}^2)x^2}{ \epsilon^3 n^2}
\enspace.
\end{equation*}
Then, an absolute constant $\kappa$ exists such that for any $x \geq 0$,
with probability at least $1-8e^{-x}$,
for any $\epsilon \in (0,1]$,
the following two inequalities hold true:
\begin{align}
\label{eq:conc.penvf.exp}
\absj{ \penVF(m,\mathcal{B},V-1) - \frac{2 \mathcal{D}_{m}}n} &\leq \epsilon\frac{\mathcal{D}_{m}}{n} + \kappa \rho_1\paren{m,\epsilon,\bayes,x,n}
\\
\label{eq:concpenvf.penid}
\absj{ \penVF(m,\mathcal{B},V-1) - 2 \norm{\bayes_m-\ERM_m}^2 } &\leq \epsilon\frac{\mathcal{D}_{m}}{n} + \kappa \rho_1\paren{m,\epsilon,\bayes,x,n}
\enspace.
\end{align}
\end{proposition}
\noindent
Proposition~\ref{prop:concpenvf} is proved in Section~\ref{sect:proofthmconcpenvf}.
Eq.~\eqref{eq:conc.penvf.exp} gives the concentration of the $V$-fold penalty around its expectation $2 \mathcal{D}_m / n = \E\scroch{\penid(m)}$, see Eq.~\eqref{eq:ExpPenid}.
Eq.~\eqref{eq:concpenvf.penid} gives the concentration of the $V$-fold penalty around the ideal penalty, see Eq.~\eqref{def.penid}.
Optimizing over $\epsilon$, the first order of the deviations of $\penVF(m,\mathcal{B},V-1)$ around $\penid(m)$ is driven by $\sqrt{\mathcal{D}_m} / n$.
The deviation term in Proposition~\ref{prop:concpenvf} does not depend on $V$ and cannot therefore help to discriminate between different values of this parameter.

\subsection{Example: histogram models} \label{sec.ex.histos}
Histograms on $\R$ are a classical example of collections of models.
Let $\mathcal{X}$ be a measurable subset of $\R$, $\mu$ denote the Lebesgue measure on $\mathcal{X}$ and $m$ be some countable partition of $\mathcal{X}$ such that $\mu(\lambda)>0$ for any $\lambda\in m$.
The histogram space $S_{m}$ based on $m$ is the linear span of the functions $(\ensuremath{\mathbf{1}}_{\lambda})_{\lambda\in m}$.
More precisely, we illustrate our results with the following examples.
\begin{example}\label{ex:regular}[Regular histograms on $\mathcal{X}=\R$]
\[\mathcal{M}_{n}=\set{m_{h},h\in\inter{n}}\;\mbox{where}\;\forall h\in\inter{n},\;m_{h}=\set{\left[\frac{\lambda}{h},\frac{\lambda+1}{h} \right),\lambda\in\Z}\enspace.\]
\end{example}
In \exref{regular}, defining $d_{m_{h}} = h$ for every $h \in \inter{n}$, for every $m \in \M_n$,  $\mathcal{D}_m=d_m-\norm{\bayes_{m}}^{2}$ since $\Psi_m$ is constant and equal to $d_m$.
Therefore Proposition~\ref{prop:concpenvf} shows $\penVF(m,\mathcal{B},V-1)$ is asymptotically equivalent to $ \pendim(m) \egaldef 2 d_m/n$ when $d_m\to \infty$.
Penalties of the form of $\pendim$ are classical and have been studied for instance by \cite{Bar_Bir_Mas:1999}.
\begin{example}\label{ex:krupt}[$k$-rupture points on $\mathcal{X}=[0,1]$]
\[\mathcal{M}_{n}=\set{m_{h_{\inter{k+1}},x_{\inter{k}}},\;x_{1}<\cdots<x_{k} \in\inter{n-1}, \;h_{i}\in[x_{i}-x_{i-1}]}\enspace,\]
where, with the conventions $x_{0}=0$, $x_{k+1}=n$, for any $x_{1},\ldots,x_{k}\in \inter{n-1}, \;x_{1}<\cdots<x_{k}, \;h_{\inter{k+1}}\in\N^{k+1}$, $m_{h_{\inter{k+1}},x_{\inter{k}}}$ is defined as the union
\[\bigcup_{i\in\inter{k}}\set{\left[\frac{x_{i-1}}n+\frac{(x_{i}-x_{i-1})(\lambda-1)}{n h_{i}},\frac{x_{i-1}}n+\frac{(x_{i}-x_{i-1})\lambda}{n h_{i}}\right),\lambda\in\inter{h_{i}}}\enspace.\]
\end{example}
In \exref{krupt}, the function $\Psi_m$ is constant on each interval $[x_{i-1},x_{i})$, equal to $h_{i}$, therefore,
\[\mathcal{D}_m=\sum_{i=1}^{k+1} h_i\Pr{\xi\in[x_{i-1}, x_i)}-\norm{\bayes_{m}}^{2}\enspace.\]

\subsection{Oracle inequality for $V$-fold procedures} \label{sec.discussion.oracle}
In order to state the main results, we introduce the following hypotheses:
\begin{itemize}
 \item {\it for all $m \in \M_n$, }
\begin{equation}
\label{hyp.NormSupNorm2}\tag{\ensuremath{\mathbf{H1}}}
 b_m^2\egaldef\sup_{t\in \mathbb{B}_m}\norm{t}_{\infty}^2\leq n
 \quad \mbox{where} \quad \mathbb{B}_m\egaldef\set{t\in S_m,\norm{t}\leq1}
 \enspace  ,
\end{equation}
\item {\it the family of the projections of $\bayes$ is uniformly bounded}: for some $a>0$,
\begin{equation}
\label{hyp.UBbayes}\tag{\ensuremath{\mathbf{H2}}}
 \forall m \in \M_n \,, \qquad \norm{\bayes_m}_{\infty}\leq a
 \enspace,
\end{equation}
\item {\it the collection of models is nested}
\begin{equation}
\label{hyp.Nested}\tag{\ensuremath{\mathbf{H2'}}}
 \forall (m, m^{\prime}) \in \mathcal{M}_{n}^2 \, , \quad S_{m}\cup S_{m^{\prime}}\in \left\{S_{m}, S_{m^{\prime}}\right\}\enspace .
\end{equation}
\end{itemize}
Hereafter, we define $A \egaldef a\vee \norm{\bayes}_{\infty}$ when \eqref{hyp.UBbayes} holds and $A \egaldef \norm{\bayes}_\infty$ when \eqref{hyp.Nested} holds.
On histogram spaces, \eqref{hyp.NormSupNorm2}  holds if and only if $\inf_{m\in\mathcal{M}_n}\inf_{\lambda\in m}\mu(\lambda)\geq n^{-1}$,
and \eqref{hyp.UBbayes} holds with $a=\norm{\bayes}_\infty$.

\begin{theorem}
\label{thm.oracle-penVF.cas_reel
Let $\xi_{\inter{n}}$ be i.i.d real-valued random variables with common density $\bayes \in L^{\infty}(\mu)$,
$\mathcal{B}$ some partition of $\inter{n}$ into $V$ pieces satisfying \eqref{hyp.part-reg.exact}
and $(S_m)_{m\in\mathcal{M}_n}$ be a collection of separable linear spaces satisfying \eqref{hyp.NormSupNorm2}.
Assume that either \eqref{hyp.UBbayes} or \eqref{hyp.Nested} holds true.
Let $C \in (1/2,2]$,
\(
\delta\egaldef 2(C-1)\) and, for any $x,\epsilon>0$,
\begin{equation*}
\rho_2\paren{\epsilon,\bayes,x,n}
\egaldef
\frac{(\norm{\bayes}_{\infty} + A) x}{\epsilon n} + \paren{ 1 + \frac{ \norm{\bayes}^2}{n} } \frac{x^2}{\epsilon^3 n}
\enspace.
\end{equation*}
For every $m \in \M_n$, let $\ERM_m$ be the estimator defined by Eq.~\eqref{def.ERM},
and $\tilde{s} = \ERM_{\widehat{m}}$ where $\widehat{m} = \widehat{m}(\CV_{(C,\mathcal{B})})$ is defined by Eq.~\eqref{eq.mh-crit-penVF-gal}.
Then, an absolute constant $\kappa$ exists such that, for any $x>0$, with probability at least $1-e^{-x}$, for any $\epsilon \in (0,1]$,
\begin{equation}
\label{eq.thm.oracle-penVF.cas_reel}
\frac{1-\delta_{-}-\epsilon}{1+\delta_{+}+\epsilon} \perte{\tilde{s}}
\leq
\inf_{m\in\mathcal{M}_{n}}\perte{\ERM_{m}}+\kappa\rho_2(\epsilon,\bayes,x_n,n)
\enspace,
\end{equation}
where $x_{n}=x+\log|\mathcal{M}_{n}|$.
\end{theorem}
\noindent
Theorem~\ref{thm.oracle-penVF.cas_reel} is proved in Section~\ref{sec.app.proof.thm-oracle.histo}.

Taking $\epsilon>0$ small enough in Eq.~\eqref{eq.thm.oracle-penVF.cas_reel}, Theorem~\ref{thm.oracle-penVF.cas_reel} proves $V$-fold model selection procedures satisfy an oracle inequality with large probability.
The remainder term can be bounded under the following classical assumption:
\begin{equation}
\label{hyp.PolColMod}\tag{\ensuremath{\mathbf{A3}}}
\exists a'>0 \, , \, \forall n\in \N^{\star}, \;|\mathcal{M}_n|\leq n^{a'}\enspace .
\end{equation}
For instance, \eqref{hyp.PolColMod} holds in \exref{regular} with $a'=1$ and in \exref{krupt} with $a'=k$.
Under \eqref{hyp.PolColMod}, the remainder term in Eq.~\eqref{eq.thm.oracle-penVF.cas_reel} is bounded by $L (\log n)^2 / (\epsilon^3 n)	$ for some $L>0$.

\noindent
The leading constant in the oracle inequality~\eqref{eq.thm.oracle-penVF.cas_reel} is $(1+\delta_+)/(1-\delta_-) + o(1)$ by choosing $\epsilon = o(1)$, so the first-order behaviour of the upper bound on the loss is driven by $\delta$.
An asymptotic optimality result can be derived from Eq.~\eqref{eq.thm.oracle-penVF.cas_reel} only if $\delta=o(1)$.
The meaning of $\delta=2(C-1)$ is the amount of bias of the $V$-fold penalization criterion, as shown by Eq.~\eqref{eq.biais.critpenVF}.
Given this interpretation of $\delta$, the model selection literature suggests no asymptotic optimality result can be obtained in general when $\delta \neq o(1)$, see for instance \cite{Sha:1997}.
Therefore, even if the leading constant $(1+\delta_+)/(1-\delta_-)$ is only an upper bound, we conjecture it cannot be taken as small as $1+o(1)$ unless $\delta=o(1)$;
such a result can be proved in our setting using similar arguments and assumptions as in \citep{Arl:2008a} for instance.

\noindent
For bias-corrected $V$-fold cross-validation, that is, $C=1$ hence $\delta=0$, Theorem~\ref{thm.oracle-penVF.cas_reel} shows a first-order optimal non-asymptotic oracle inequality, since the leading constant $(1+\epsilon)/(1-\epsilon)$ can be taken equal to $1+o(1)$, and the remainder term is small enough under assumption \eqref{hyp.PolColMod}, for instance.
Such a result valid with no upper bound on $V$ had never been obtained before in any setting.

\noindent
Regular $V$-fold cross-validation is also analyzed by Theorem~\ref{thm.oracle-penVF.cas_reel}, since by Lemma~\ref{le.penVF-VFCV} it corresponds to $C=1+1/(2(V-1))$, hence $\delta=1/(V-1)$.
When $V$ is fixed, the oracle inequality is asymptotically sub-optimal, which is consistent with the result proved in regression by \cite{Arl:2008a}.
On the contrary, if $\mathcal{B}=\mathcal{B}_n$ has $V_n$ blocs, with $V_n\to\infty$, Theorem~\ref{thm.oracle-penVF.cas_reel} implies under assumption \eqref{hyp.PolColMod} the asymptotic optimality of $V_n$-fold cross-validation as soon as the oracle loss is  much larger than $(\log n)^2 / n$.

\medskip

\noindent
The bound obtained in Theorem~\ref{thm.oracle-penVF.cas_reel} can be integrated and we get
\begin{equation*}
\frac{1-\delta_{-}-\epsilon}{1+\delta_{+}+\epsilon}\E\croch{\perte{\tilde{s}}}\leq \E\croch{\inf_{m\in\mathcal{M}_{n}}\perte{\ERM_{m}}}+\kappa^{\prime} \rho_2(\epsilon,\bayes,\log( |\mathcal{M}_n|))
\end{equation*}
for some absolute constant $\kappa^{\prime}>0$.

\noindent
Assuming $C>1/2$ is necessary according to minimal penalty results proved in \cite{Le09}.
Assuming $C \leq 2$ only simplifies the presentation; if $C>2$, the same proof shows Theorem~\ref{thm.oracle-penVF.cas_reel} holds with $\kappa$ replaced by $C \kappa$.

\noindent
An oracle inequality similar to Theorem~\ref{thm.oracle-penVF.cas_reel} holds in a more general setting, as proved in a previous version of this paper \cite{Arl_Ler:2012:penVF.v1}; we state a less general result here for simplifying the exposition, since it does not change the message of the paper.
First, assumption \eqref{hyp.part-reg.exact} can be relaxed into assuming the partition $\mathcal{B}$ is close to regular, that is,
\begin{equation}
\label{hyp.part-reg} \tag{\bf Reg'}
\mathcal{B} \mbox{ is a partition of } \inter{n} \mbox{ of size $V$ and }
\sup_{k\in\inter{V}} \absj{ \card(\mathcal{B}_k) - \frac{n}{V} } \leq 1
\enspace ,
\end{equation}
which can hold for any $V \in \inter{n}$.
Second, data $\xi_1, \ldots, \xi_n$ can belong to a general Polish space $\mathcal{X}$, at the price of some additional technical assumption, see \cite{Arl_Ler:2012:penVF.v1}.

\subsection{Comparison with previous works on $V$-fold procedures} \label{sec.oracle.previous-works}
Few non-asymptotic oracle inequalities have been proved for $V$-fold penalization or cross-validation procedures.
Concerning cross-validation, previous oracle inequalities are listed in the survey \cite{Arl_Cel:2010:surveyCV}.
In the least-squares density estimation framework, oracle inequalities were proved by \cite{vdL_Dud_Kel:2004} in the $V$-fold case, but compared the risk of the selected estimator with the risk of an oracle trained with $n(V-1)/V$ data.
In comparison, Theorem~\ref{thm.oracle-penVF.cas_reel} considers the strongest possible oracle, that is, trained with $n$ data.
Optimal oracle inequalities were proved by \cite{Cel:2008} for leave-$p$-out estimators with $p \ll n$, a case also treated in Theorem~\ref{thm.oracle-penVF.cas_reel} by taking $V=n$ and $C=(n/p-1/2)/(n/p-1)$ as shown by Lemma~\ref{le.penVF-VFCV}.
If $p \ll n$, $C \sim 1$, hence $\delta=o(1)$ and we recover the result of \cite{Cel:2008}.
Concerning $V$-fold penalization, previous results were either valid for $V=n$ only (\cite[Theorem 7.6]{Mas:2003:St-Flour} and \cite{Le09} for least-squares density estimation, \cite{Arl:2009:RP} for regressogram estimators), or for $V$ bounded when $n$ tends to infinity (for regressogram estimators \cite{Arl:2008a}).
In comparison, Theorem~\ref{thm.oracle-penVF.cas_reel} provides a result valid for all $V$, except for the assumption that $V$ divides $n$, which can be removed, see \cite{Arl_Ler:2012:penVF.v1}.
In particular, the loss bound in \cite{Arl:2008a} deteriorates when $V$ grows, while it remains stable in our result.
The latter corresponds to the typical behavior of the loss ratio $ \snorm{ \bayes-\tilde{s} }^2 / \inf_{m \in \M_n} \snorm{ \bayes-\ERM_m }^2 $ of $V$-fold penalization as a function of $V$ in simulation experiments, see Section~\ref{sec.simus} and \cite{Arl:2008a}, for instance.

\section{How to compare theoretically the performances of model selection procedures for estimation?}
\label{sec.oracle.key-quant}

The main goal of the paper is to compare the model selection performances of several ($V$-fold) cross-validation methods, when the goal is estimation, that is minimizing the loss $\perte{\ERM_{\widehat{m}}}$ of the selected estimator.
In this section, we discuss how such a comparison can be made on theoretical grounds, in a general setting.

\medskip

\noindent
For some data-driven function $\CV: \mathcal{M} \to \R$, the goal is to understand how $\sperte{\ERM_{\widehat{m}(\CV)}}$ depends on $\CV$ when the selected model is
\begin{equation}
\label{eq.def.mhCVgal}
\widehat{m}(\CV) \in \mathop{\argmintmp}_{m \in \M_n} \set{\CV(m)}
\enspace .
\end{equation}
From now on, in this section, $\CV$ is assumed to be a cross-validation estimator of the risk, but the heuristic developed here applies to the general case.

\paragraph{Ideal comparison.}
Ideally, for proving $\CV_1$ is a better method than $\CV_2$ in some setting, we would like to prove that
\begin{equation}
\label{eq.ideal-comparison}
\perte{\ERM_{\widehat{m}(\CV_1)}} < (1-\varepsilon_n) \perte{\ERM_{\widehat{m}(\CV_2)}}
\end{equation}
with a large probability, for some $\varepsilon_n \geq 0$.

\paragraph{Previous works and their limits.}
The classical way to analyze the performance of a model selection procedure is to prove an oracle inequality, that is, to {\em upper bound} (with a large probability or in expectation)
\[
\perte{\ERM_{\widehat{m}(\CV)}} - \inf_{m \in \M_n} \set{\perte{\ERM_m}}
\quad \mbox{or} \quad
\mathfrak{R}_n(\CV) \egaldef \frac{\perte{\ERM_{\widehat{m}(\CV)}}} { \inf_{m \in \M_n} \set{\perte{\ERM_m}} }
\enspace .
\]
Alternatively, asymptotic results show that when $n$ tends to infinity, $\mathfrak{R}_n(\CV) \to 1$ (asymptotic optimality of $\CV$) or that $\mathfrak{R}_n(\CV_1) \sim \mathfrak{R}_n(\CV_2)$ (asymptotic equivalence of $\CV_1$ and $\CV_2$).
A review of such results can be found in \cite[Section~6]{Arl_Cel:2010:surveyCV}.
Nevertheless, proving Eq.~\eqref{eq.ideal-comparison} requires a lower bound on $\mathfrak{R}_n(\CV)$ (asymptotic or not), which has been done only once for some cross-validation method, to the best of our knowledge.
In some least-squares regression setting, $V$-fold cross-validation ($\ensuremath{\CV^\mathrm{VF}}$) performs (asymptotically) worse than all asymptotically optimal model selection procedures since $\mathfrak{R}_n(\ensuremath{\CV^\mathrm{VF}}) \geq \kappa(V) > 1$ with a large probability \cite{Arl:2008a}.

\noindent
The major limitation of all these previous results is they can only compare $\CV_1$ to $\CV_2$ at first order, that is, according to $\lim_{n \to \infty} \mathfrak{R}_n(\CV_1)/\mathfrak{R}_n(\CV_2)$, which only depends on the bias of $\CV_i(m)$ ($i=1,2$) as an estimator of $\E\scroch{\sperte{\ERM_m}}$, hence, on the asymptotic ratio between the training set size and the sample size \cite[Section~6]{Arl_Cel:2010:surveyCV}.
For instance, the leave-$p$-out and the hold-out with a training set of size $(n-p)$ cannot be distinguished at first order, while the leave-$p$-out performs much better in practice, certainly because  its ``variance'' is much smaller.

\paragraph{Beyond first-order.}
So, we must go beyond the first-order of $\mathfrak{R}_n(\CV)$ and take into account the variance of $\CV(m)$.
Nevertheless, proving a lower bound on $\mathfrak{R}_n(\CV)$ is already challenging at first order---probably the reason why only one has been proved up to now, in a specific setting only---so the challenge of computing a precise lower bound on the second order term of $\mathfrak{R}_n(\CV)$ seems too high for the present paper.
We propose instead a heuristic showing the variances of some quantities---depending on $(\CV_i)_{i=1,2}$ and on $\mathcal{M}_n$---can be used as a proxy to a proper comparison of the second-order terms of $\mathfrak{R}_n(\CV_1)$ and $\mathfrak{R}_n(\CV_2)$.
Since we focus on second-order terms, from now on, we assume $\CV_1$ and $\CV_2$ have the same bias, that is,
\begin{equation}
\label{hyp.heur-var.same-bias}
\tag{\ensuremath{\mathbf{SameBias}}}
\forall m \in \M_n \, ,  \quad \E\scroch{\CV_1(m)} = \E\scroch{\CV_2(m)} \enspace .
\end{equation}
In least-squares density estimation, given Lemma~\ref{le.penVF-VFCV}, this means for $i \in \sset{1,2}$, $\CV_i = \CV_{(C,\mathcal{B}_i)}$ as defined by Eq.~\eqref{eq.crit-penVF-gal}, with different partitions $\mathcal{B}_i$ satisfying \eqref{hyp.part-reg.exact} with different $V=V_i$, but the same constant $C>0$; $C=1$ corresponds to the unbiased case.

\paragraph{The variance of the cross-validation criteria is not the correct quantity to look at.}
If we were only comparing cross-validation methods $\CV_1, \CV_2$ as estimators of $\E\scroch{\perte{\ERM_m}}$ for every single $m \in \M_n$, we could naturally compare them through their mean squared errors.
Under assumption \eqref{hyp.heur-var.same-bias}, this would mean to compare their variances.
This can be done from Eq.~\eqref{eq.pro.variance.critpenVF} below, but it is not sufficient to solve our problem, since it is known the best cross-validation estimator of the risk does not necessarily yield the best model selection procedure \cite{Bre_Spe:1992}.
More precisely,
the selected model $\widehat{m}(\CV)$ defined by Eq.~\eqref{eq.def.mhCVgal} is unchanged when $\CV(m)$ is translated by any random quantity, but such a translation does change $\var\sparen{\CV(m)}$ and can make it as large as desired.
For model selection, what really matters is that
\begin{equation}
\label{eq.modsel-ranking}
\sign \paren{ \CV(m_1) - \CV(m_2) } = \sign \paren{ \perte{\ERM_{m_1}} - \perte{\ERM_{m_2}} }
\end{equation}
as often as possible for every $(m_1,m_2) \in \mathcal{M}_n^2$,
and that most mistakes in the ranking of models occur when $\sperte{\ERM_{m_1}} - \sperte{\ERM_{m_2}}$ is small, so that
$\sperte{\ERM_{\widehat{m}(\CV)}}$ cannot be much larger than $\inf_{m \in \M_n} \sset{\sperte{\ERM_m}}$.

\paragraph{Heuristic.}
The heuristic we propose goes as follows.
Assume for simplicity $\mo = \mathop{\argmintmp}_{m \in \M_n} \E\scroch{\sperte{\ERM_m}}$ is uniquely defined.
For any $\CV$, the smallest is $\Prob(m=\widehat{m}(\CV))$ for all $m \neq \mo$, the better should be the performance of $\ERM_{\widehat{m}(\CV)}$.
Our idea is to find a proxy for $\Prob(m=\widehat{m}(\CV))$, that is, a quantity that should behave similarly as a function of $\CV$ and its ``variance'' properties.
For all $m,m^{\prime} \in \mathcal{M}_n$, let $\Delta_{\CV}(m,m^{\prime}) \egaldef \CV(m) - \CV(m^{\prime})$, $\xi$ some standard Gaussian random variable and $\overline{\Phi}(t) = \Prob\paren{\xi > t}$ for all $t \in \R$.
Then, for every $m \in \M_n$,
\begin{align}
\notag
& \quad \, \, \Prob\paren{\widehat{m}(\CV) = m}
= \Prob\paren{\forall m^{\prime} \neq m \, , \, \Delta_{\CV}(m,m^{\prime}) < 0 }
\\
\label{eq.moresimus.heur.approx1}
&\asymp
\min_{m^{\prime} \neq m} \Prob\paren{\Delta_{\CV}(m,m^{\prime}) < 0 }
\\
\label{eq.moresimus.heur.approx2}
&\approx \min_{m^{\prime} \neq m} \Prob\paren{\E\croch{ \Delta_{\CV}(m,m^{\prime}) } + \xi \sqrt{\var\paren{\Delta_{\CV}(m,m^{\prime})}} < 0 }
\\
\notag
&=   \overline{\Phi} \paren{ \SR_{\CV}(m) }
\quad \mbox{where} \quad
\SR_{\CV}(m) \egaldef \max_{m^{\prime} \neq m} \frac{\E\croch{ \Delta_{\CV}(m,m^{\prime}) }}{\sqrt{\var\paren{\Delta_{\CV}(m,m^{\prime})}}}
\enspace .
\end{align}
So, if $\SR_{\CV_1}(m) > \SR_{\CV_2}(m)$  for all $m \neq \mo$, $\CV_1$ should be better than $\CV_2$.
Under assumption \eqref{hyp.heur-var.same-bias}, this leads to the following heuristic:
\begin{equation}
\label{eq.heuristic.variance-comp}
\forall m \neq m^{\prime}  \, , \,
\var\paren{\ensuremath{\Delta}_{\CV_1}(m,m^{\prime})} < \var\paren{\ensuremath{\Delta}_{\CV_2}(m,m^{\prime})}
\Rightarrow
\CV_1 \mbox{ better than } \CV_2
\enspace .
\end{equation}
Let us make some remarks.
\begin{itemize}
\item The quantity $\ensuremath{\Delta}_{\CV}(m,m^{\prime})$ appears in relative bounds \cite[Section~1.4]{Cat:2007} which can be used as a tool for model selection \cite{Aud:2004:tech}

\item Approximation \eqref{eq.moresimus.heur.approx1} is the strongest one.
Clearly, inequality $\leq$ holds true. The equality case is for a very particular dependence setting, when the events $\paren{\set{\Delta_{\CV}(m,m^{\prime}) < 0}}_{m^{\prime} \in \mathcal{M}}$ are nested.
In general, the left-hand side is significantly smaller than the right-hand side; we claim they vary similarly as a function of $\CV$.

\item The gaussian approximation \eqref{eq.moresimus.heur.approx2} for $\Delta_{\CV}(m,m^{\prime})$ does not hold exactly, but it seems reasonable to make it, at first order at least.

\end{itemize}
\noindent
In the heuristic \eqref{eq.heuristic.variance-comp}, all $(m,m^{\prime})$ do not matter equally for explaining a quantitative difference in the performances of $\CV$.
First, we can fix $m^{\prime} = \mo$, since intuitively, the strongest candidate against any $m \neq \mo$ is $\mo$, which clearly holds in all our experiments, see Figures~\ref{fig.variance.SR-vs-Rmo.SiG5-Li01Regu.n100} and~\ref{fig.variance.SR-vs-Rmo.SiG5-Li01Regu.n500}.
Second, if $m$ and $\mo$ are very close, that is, $\sperte{\ERM_m}/\sperte{\ERM_{\mo}}$ is smaller than the minimal order of magnitude we can expect for $\mathfrak{R}_n(\CV)$ with a data-driven $\CV$, taking $m$ instead of $\mo$ does not decrease the performance significantly.
Third, if $ \overline{\Phi} \paren{ \SR_{\CV}(m) }$ is much too small, changing it even by an order of magnitude will not affect the performance of $\widehat{m}(\CV)$ significantly; hence, all $m$ such that, say,  $\SR_{\CV}(m) \gg (\log(n))^{\alpha}$ for all $\alpha>0$, can also be discarded.
Overall, pairs $(m,m^{\prime})$ that  really matter in \eqref{eq.heuristic.variance-comp} are pairs $(m,\mo)$ that are at a ``moderate distance'', in terms of $\E\scroch{ \sperte{\ERM_m} - \sperte{\ERM_{\mo}} }$.

\section{Dependence on $V$ of $V$-fold penalization and cross-validation} \label{sec.variance}
Let us now come back to the least-squares density estimation setting.
Our goal is to compare the performance of classical cross-validation methods having the same bias, that is, according to Section~\ref{sec.cadre.links}, $\widehat{m}(\CV_{(C,\mathcal{B})})$ with the same constant $C$ but different partitions $\mathcal{B}$, where $\widehat{m}(\CV_{(C,\mathcal{B})})$ is defined by Eq.~\eqref{eq.mh-crit-penVF-gal}.
\begin{theorem} \label{theo.variance.penVF}
Let $\xi_{\inter{n}}$ be i.i.d. random variables with common density $\bayes\in L^{\infty}(\mu)$, $\mathcal{B}$ some partition of $\inter{n}$ into $V$ pieces satisfying \eqref{hyp.part-reg.exact}, and $(\psil)_{\lambda\in\Lambda_{m_1}}$, $(\psil)_{\lambda\in\Lambda_{m_2}}$ two orthonormal families in $L^2(\mu)$.
For $m \in \set{m_1,m_2}$, $S_{m}$ denotes the linear span of $\paren{\psil}_{\lambda\in\Lambda_{m}}$, $\mathbb{B}_{m}=\{t \in S_{m} \, / \, \norm{t} \leq 1\}$ and $\Psi_{m} \egaldef \sup_{t\in\mathbb{B}_{m}}t^2$. For any $\Lambda,\Lambda^{\prime} \in \set{\Lambda_{m_1},\Lambda_{m_2}}$,
\begin{align*}
 \termeBvar{\Lambda,\Lambda^{\prime}}
 &\egaldef \sum_{\lambda\in\Lambda,\lambda^{\prime}\in\Lambda^{\prime}} \paren{\Ex{\paren{\psil(\xi_1)-P\psil}\paren{\psi_{\lambda^{\prime}}(\xi_1)-P\psi_{\lambda^{\prime}}}}}^2
 \\
\mbox{and} \quad
\termeBvaracr{m_1,m_2}
&\egaldef\termeBvar{\Lambda_{m_1} , \Lambda_{m_1}} + \termeBvar{\Lambda_{m_2} , \Lambda_{m_2} } - 2\termeBvar{\Lambda_{m_1} , \Lambda_{m_2} }
\enspace .
\end{align*}
Then, for every $C>0$,
\begin{align}
\label{eq.pro.variance.critpenVF}
 &\Var{\CV_{(C,\mathcal{B})}(m_1)}=\frac{2}{n^2}\paren{1+\frac{4C^2}{V-1}-\frac{\paren{2C-1}^2}n}\termeBvar{\Lambda_{m_1},\Lambda_{m_1}}\\
\notag
&\qquad +\frac{4}{n}\Var{\paren{1+\frac{2C-1}{n}}\bayes_{m_1}(\xi_1)-\frac{2C-1}{2n}\Psi_{m_1}(\xi_1)}
\\
\mbox{and} \quad
\label{eq.pro.variance.penVF-penid-incrReg}
&\Var{ \CV_{(C,\mathcal{B})}(m_1) - \CV_{(C,\mathcal{B})}(m_2)}=\frac{2}{n^2}\paren{1+\frac{4C^2}{V-1}-\frac{(2C-1)^2}{n}} \termeBvaracr{m_1 , {m_2} }
\\
\notag
&\qquad +\frac{4}{n} \Var{\paren{1+\frac{2C-1}{n}} (\bayes_{m_1}-\bayes_{m_2}) (\xi_1)-\frac{2C-1}{2n}(\Psi_{m_1}-\Psi_{m_2})(\xi_1)}
\enspace .
\end{align}
\end{theorem}
Theorem~\ref{theo.variance.penVF} is proved in Section~\ref{sec.proof.variance.main}.

\paragraph{In the unbiased case,} that is, $C=1$,
\[\Var{\CV_{(1,\mathcal{B})}(m_1)-\CV_{(1,\mathcal{B})}(m_2) } = \paren{1+\frac{4}{V-1}-\frac{1}{n}} a + b\]
for some $a,b \geq 0$ depending on $n,m_1,m_2$ but not on $V$.
Given the heuristic arguments of Section~\ref{sec.oracle.key-quant}, this shows the model selection performance of bias-corrected $V$-fold cross-validation improves when $V$ increases, but the improvement is at most in a second order term as soon as $V$ is large.
In particular, even if $b \ll a$, the improvement from $V=2$ to $5$ or $10$ is much larger than from $V=10$ to $V=n$, which justifies the commonly used principle that taking $V=5$ or $V=10$ is large enough.
Assuming in addition $S_{m_1}$ and $S_{m_2}$ are regular histogram models (Example~\ref{ex:regular} in Section~\ref{sec.ex.histos}) and $d_{m_1}$ divides $d_{m_2}$, then, by Lemma~\ref{lem:var2} in Section~\ref{sec.app.variance.eval-terms-histos},
\begin{align*}
a &= \frac{2}{n^2} \termeBvaracr{m_1,m_2}
\asymp \norm{\bayes_{m_2}}^2 \frac{d_{m_2}}{n^{2}}
\\
\mbox{and} \quad
b&=\frac{4}{n} \paren{1+\frac{1}{n}}^2 \Var{\bayes_{m_1}(\xi) - \bayes_{m_2}(\xi)}
\approx \mathcal{O} \paren{ \frac{1}{n} \norm{\bayes_{m_1}-\bayes_{m_2}}^2 }
\enspace .
\end{align*}
When $d_{m_2} / n$ is at least as large as $\snorm{\bayes_{m_1}-\bayes_{m_2}}^2$, we obtain that $V$ drives the constant of the first order term in the variance through the multiplicative factor $1+4/(V-1)$ in front of $a$.
Let $\CV_{\rm{id}}(m) \egaldef P_n\gamma(\ERM_m)+\E\croch{\penid(m)}$ be the criterion we could use if we knew the expectation of the ideal penalty.
From Proposition~\ref{pro.variance} in Section~\ref{sec.app.proof.thm-variance},
\begin{multline*}
\Var{\CV_{\rm{id}}(m_1)-\CV_{\rm{id}}(m_2)}=\frac{2}{n^2}\paren{1-\frac{1}{n}} \termeBvaracr{m_1 , m_2 }\\
+\frac{4}{n} \Var{\paren{1-\frac{1}{n}}(\bayes_{m_1}-\bayes_{m_2})(\xi_1) + \frac{1}{2n}(\Psi_{m_1}-\Psi_{m_2}) (\xi_1)}
\end{multline*}
which easily compares to formula~\eqref{eq.pro.variance.penVF-penid-incrReg} obtained for the $V$-fold criterion when $C=1$.
Up to smaller order terms, the difference lies in the first term, where $(1+4/(V-1)-1/n)$ is replaced by $(1-1/n)$ when using the expectation of the ideal penalty instead of a $V$-fold penalty.
In other words, the leave-one-out penalty---that is, taking $V=n$---behaves like the expectation of the ideal penalty.

\paragraph{Regular $V$-fold cross-validation} and the leave-$p$-out are also covered by Theorem~\ref{theo.variance.penVF}, according to Lemma~\ref{le.penVF-VFCV}, respectively with $C=1+1/(2(V-1))$ and with $V=n$ and $C=1 + 1/(2(n/p-1))$.
The conclusion is similar: increasing $V$ decreases the variance, and $V$-fold cross-validation performs almost as well as the leave-$(n/V)$-out as soon as $V$ is larger than $5$ or $10$.

Similarly, the variances of the $V$-fold cross-validation and leave-$p$-out criteria---for instance---can be derived from Eq.~\eqref{eq.pro.variance.critpenVF}.
In the leave-$p$-out case, we recover formulas obtained in \cite{Cel:2008,Cel_Rob:2006}, with a different grouping of the variance components;
Eq.~\eqref{eq.pro.variance.critpenVF} clearly emphasizes the influence of the bias---through $(C-1)$---on the variance.
For $V$-fold cross-validation, we believe Eq.~\eqref{eq.pro.variance.critpenVF} shows in a simpler way how the variance depends on $V$, compared to the result of \cite{Cel_Rob:2006} which was focusing on the difference between $V$-fold cross-validation and the leave-$(n/V)$-out; here the difference can be written
\[
8 \paren{\frac{1}{V-1}-\frac{1}{n-1}} \paren{1+\frac{1}{2(V-1)}}^2 n^{-2}
\termeBvar{\Lambda_{m_1},\Lambda_{m_1}}
\enspace.\]
A major novelty in Eq.~\eqref{eq.pro.variance.critpenVF} is also to cover a larger set of criteria, such as corrected-$V$-fold cross-validation.
Note that $\var(\CV_{(C,\mathcal{B})}(m_1))$ is generally much larger than $\var(\CV_{(C,\mathcal{B})}(m_1)-\CV_{(C,\mathcal{B})}(m_2))$, which illustrates again why computing the former quantity might not help for understanding the model selection properties of $\CV_{(C,\mathcal{B})}$, as explained in Section~\ref{sec.oracle.key-quant}.
For instance, comparing Eq.~\eqref{eq.pro.variance.critpenVF} and~\eqref{eq.pro.variance.penVF-penid-incrReg}, changing $\bayes_{m_1}$ into $\bayes_{m_1} - \bayes_{m_2}$ in the second term can reduce dramatically the variance when $\bayes_{m_1}$ and $\bayes_{m_2}$ are close, which happens for the pairs $(m_1,m_2)$ that matter for model selection according to Section~\ref{sec.oracle.key-quant}.

The variance of hold-out criteria and their increments $\Delta_{\CVho}(m_1,m_2)$ are also computed in Proposition~\ref{pro.variance.penHO} in Section~\ref{sec.supmat.penHO.variance}.

\begin{remark}
The term $\termeBvaracr{m_1,m_2}$ does not depend on the choice of particular bases of $S_{m_1}$ and $S_{m_2}$: as proved in Proposition~\ref{pro.calcul.termes.variance} in Section~\ref{sec.app.proof.thm-variance},
\begin{equation}
\label{eq.calcvar.B.simplifie}
\termeBvaracr{m_1,m_2}=\Var{\paren{\ERM_{m_1} - \ERM_{m_2}} (\xi)}-\frac{n+1}{n} \Var{\paren{\bayes_{m_1} - \bayes_{m_2}} (\xi)}
\enspace,
\end{equation}
where $\xi$ denotes a copy of $\xi_1$, independent of $\xi_{\inter{n}}$.
\end{remark}

\section{Simulation study} \label{sec.simus}
This section illustrates the main theoretical results of the paper with some experiments on synthetic data.

\subsection{Setting} \label{sec.simus.setting}
In this section, we consider $\mathcal{X}=[0,1]$ and $\mu$ is the Lebesgue measure on $\mathcal{X}$.
Two examples are considered for the target density $\bayes$ and for the collection of models $(S_m)_{m \in \M_n}$.

\noindent {\bf Two density functions} $\bayes$ are considered, see Figure~\ref{fig.cadres}:
\begin{itemize}
\item Setting L:
$\bayes(x) = \frac{10 x}{3} \ensuremath{\mathbf{1}}_{0 \leq x < 1/3} + \sparen{ 1 + \frac{x}{3} } \ensuremath{\mathbf{1}}_{1 \geq x \geq 1/3}\,$.
\item Setting S:
$\bayes$ is the mixture of the piecewise linear density $x \mapsto (8 x - 4 ) \ensuremath{\mathbf{1}}_{1 \geq x \geq 1/2}  $ (with weight 0.8) and four truncated gaussians with means $(k/10)_{k=1 , \ldots , 4}$ and standard deviation $1/60$ (each with weight 0.05).
\end{itemize}
\begin{figure}
\begin{minipage}[b]{.48\linewidth}
\includegraphics[width=\textwidth]{figure_density_C111007_Li01_n500_seed2.pdf}
\end{minipage}
\hspace{.025\linewidth}
\begin{minipage}[b]{.48\linewidth}
\includegraphics[width=\textwidth]{figure_density_C111007_SiG5_n500_seed2.pdf}
\end{minipage}
\caption{The two densities considered. Left: setting L. Right: setting S.
\label{fig.cadres}
}
\end{figure}

\noindent {\bf Two collections of models} are considered, both leading to histogram estimators: for every $m \in \M_n$, $S_m$ is the set of piecewise constant functions on some partition $\Lambda_m$ of  $\mathcal{X}\,$.
\begin{itemize}
\item ``Regu'' for regular histograms: $\mathcal{M}_n = \set{1, \ldots, n}$ where for every $m \in \M_n\,$, $ \Lambda_m $ is the regular partition of $[0,1]$ into $m$ bins.
\item ``Dya2'' for dyadic regular histograms with two bin sizes and a variable change-point: $\mathcal{M}_n = \bigcup_{ k \in \set{1, \ldots, \widetilde{n} } } \set{ k } \times \set{ 0 , \ldots, \floor{\log_2(k)} }  \times \set{ 0 , \ldots, \floor{\log_2( \widetilde{n} - k)} } $ where $\widetilde{n}=\floor{n/\log(n)}$ and for every $(k,i,j) \in \mathcal{M}_n$, $\Lambda_{(k,i,j)}$ is the union of the regular partition of $[0,k/\widetilde{n})$ into $2^i$ pieces and the regular partition of $[k/\widetilde{n},1]$ into $2^j$ pieces.
\end{itemize}

The difference between ``Regu'' and ``Dya2'' can be visualized on Figure~\ref{fig.oracles.Regu-Dya2.SiG5}, where the corresponding oracle models have been plotted in setting S. While ``Regu'' is one of the simplest and most classical collections for density estimation, the flexibility of ``Dya2'' allows to adapt to the variability of the smoothness of $\bayes$. Intuitively, in settings L and S, the optimal bin size is smaller on $[0,1/2]$ (where $\bayes$ is varying fastly) than on $[1/2,1]$ (where $\absj{\bayes^{\prime}}$ is much smaller).
\begin{figure}
\begin{minipage}[b]{.48\linewidth}
\includegraphics[width=\textwidth]{figure_estim_oracle_C111007_SiG5Regu_n500_seed3}
\end{minipage}
\hspace{.025\linewidth}
\begin{minipage}[b]{.48\linewidth}
\includegraphics[width=\textwidth]{figure_estim_oracle_C111007_SiG5Dya2_n500_seed3}
\end{minipage}
\caption{Oracle model for one sample of size $n=500$, in setting S. Left: Regu. Right: Dya2.
\label{fig.oracles.Regu-Dya2.SiG5}
}
\end{figure}

Another point of comparison of Regu and Dya2 is given by Table~\ref{tab.comp.Regu-Dya2}, that reports values of the quadratic risks obtained depending on the collection of models considered.
Table~\ref{tab.comp.Regu-Dya2} shows that in settings L and S, the collection Dya2 helps reducing the quadratic risk by approximately 20\% (when comparing the best data-driven procedures of our experiment), and even more when comparing oracle estimators (30\% in setting S, 59\% in setting L).
Therefore, in settings L and S, it is worth considering more complex collections of models (such as Dya2) than regular histograms.
\begin{table}
\caption{Comparison of Regu and Dya2: quadratic risks $\E\scroch{ \snorm{ \bayes - \ERM_{\widehat{m}}}^2 }$ of ``Oracle'' and ``Best'' estimators \textup{(}multiplied by $10^{3}$\textup{)} with the two collections of models. ``Best'' means that $\widehat{m}$ is the data-driven procedure minimizing $\E\scroch{\snorm{ \bayes - \ERM_{\widehat{m}}}^{2}}$ among all the data-driven procedures we considered in our experiments \textup{(}see Section~\ref{sec.simus.proc}\textup{)}. ``Oracle'' means that $\widehat{m} \in \mathop{\argmintmp}_{m \in \M_n} \snorm{ \bayes - \ERM_m}^2$ is the oracle model for each sample. \label{tab.comp.Regu-Dya2}}
\begin{center}
\begin{tabular}
{p{0.16\textwidth}@{\hspace{0.025\textwidth}}p{0.18\textwidth}@{\hspace{0.025\textwidth}}p{0.16\textwidth}@{\hspace{0.025\textwidth}}p{0.17\textwidth}@{\hspace{0.025\textwidth}}p{0.18\textwidth}}
\hline
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
Setting      & Oracle(Regu)   & Oracle(Dya2)    & Best(Regu)      & Best(Dya2)    \\
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
\hline
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
L            & $13.4 \pm 0.1$ & $5.46 \pm 0.02$ & $25.8 \pm 0.1$  & $19.4 \pm 0.1$ \\
S            & $62.4 \pm 0.1$ & $43.9 \pm 0.1$  & $100.9 \pm 0.2$ & $83.4 \pm 0.2$ \\
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
\hline
\end{tabular}
\end{center}
\end{table}

Let us finally remark that Dya2 does not reduce the quadratic risk in all settings as significantly as in settings L and S.
We performed similar experiments with a few other density functions, sometimes leading to less important differences between Regu and Dya2 in terms of risk (results not shown).
The oracle model was always better with Dya2, but in two cases, the risk of the best data-driven procedure with Dya2 was larger than with Regu by 6 to 8\%.

\subsection{Procedures compared} \label{sec.simus.proc}
In each setting, we consider the following model selection procedures:
\begin{itemize}
\item $\pendim$ \cite{Bar_Bir_Mas:1999}: penalization with $\pen(m) = 2 \card(\Lambda_m) / n\,$.
\item $V$-fold cross-validation with $V \in \sset{2,5,10,n}$, see Section~\ref{sec.cadre.VFCV}.
\item $V$-fold penalties (with leading constant $x=V-1$, that is, bias-corrected $V$-fold cross-validation), for $V \in \set{2,5,10,n}$, see Section~\ref{sec.cadre.penVF}.
\item for comparison, penalization with $\E\scroch{\penid(m)}$, that is, $\widehat{m}(\CV_{\rm{id}})$.
\end{itemize}

Since it is often suggested to multiply the usual penalties by some factor larger than one \cite{Arl:2008a}, we consider all penalties above multiplied by a factor chosen among $\set{1, 1.25, 1.5, 2}$.
Complete results can be found in Table~\ref{tab.complet.Li01.SiG5} in Section~\ref{sec.supmat.simus}.

\subsection{Model selection performances} \label{sec.simus.modsel}
In each setting, all procedures are compared on $N=10\,000$ independent synthetic data sets of size $n=500$.
For measuring their respective model selection performances, for each procedure $\widehat{m}(\CV)$ we estimate
\[ \ensuremath{C_{\mathrm{or}}}(\CV) \egaldef \E\croch{ \mathfrak{R}_n(\CV) } = \E\croch{ \frac{\perte{\ERM_{\widehat{m}(\CV)}}}{ \inf_{m \in \M_n} \perte{ \ERM_m } } } \]
which represents the constant that would appear in front of an oracle inequality.
The uncertainty of estimation of $\ensuremath{C_{\mathrm{or}}}(\CV)$ is measured by the empirical standard deviation of $\mathfrak{R}_n(\CV)$ divided by $\sqrt{N}$.
The results are reported in Table~\ref{tab.simus-results} for settings L and S, with the collection Dya2.
\begin{table}
\caption{Estimated model selection performances, see text. `LOO' is a shortcut for `leave-one-out', that is, $V$-fold with $V=n=500$.
\label{tab.simus-results}}
\begin{center}
\begin{tabular}
{p{0.15\textwidth}@{\hspace{0.05\textwidth}}p{0.12\textwidth}@{\hspace{0.05\textwidth}}p{0.12\textwidth}}
\hline
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
Procedure          & L--Dya2              & S--Dya2              \\
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
\hline
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
$\pendim$              & $8.27 \pm 0.07$      & $3.21 \pm 0.01$       \\
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
\hline
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
pen2F         & \hspace{-0.15cm}$10.21 \pm 0.08$ & $2.39 \pm 0.01$ \\
pen5F         & $7.47 \pm 0.06$                  & $2.16 \pm 0.01$ \\
pen10F        & $6.89 \pm 0.06$                  & $2.11 \pm 0.01$ \\
penLOO        & $6.35 \pm 0.05$                  & $2.06 \pm 0.01$ \\
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
\hline
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
2FCV               & $6.41 \pm 0.05$       & $2.05 \pm 0.01$       \\
5FCV               & $6.27 \pm 0.05$       & $2.05 \pm 0.01$       \\
10FCV              & $6.24 \pm 0.05$       & $2.05 \pm 0.01$       \\
LOO                & $6.34 \pm 0.05$       & $2.06 \pm 0.01$       \\
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
\hline\hline
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
$\E\scroch{\penid}$ & $6.52 \pm 0.05$       & $2.07 \pm 0.01$       \\
\noalign{\vspace*{0.075cm}}}% autre option: \noalign{\smallskip
\hline
\end{tabular}
\end{center}
\end{table}

Results for Regu are not reported here since dimensionality-based penalties are already known to work well with Regu \cite{Le09}, so $V$-fold methods cannot improve significantly its performance, with a larger computational cost. Complete results (including Regu, with $n=100$ and $n=500$) are given in Tables~\ref{tab.complet.Li01.SiG5} and~\ref{tab.complet.Li01.SiG5.n100} in Section~\ref{sec.supmat.simus}, showing the performances of $\pendim$ and $V$-fold methods indeed are very close.

\paragraph{Performance as a function of $V$}
Let us first consider $V$-fold penalization.
In both settings L and S, as suggested by our theoretical results, $\ensuremath{C_{\mathrm{or}}}$ decreases when $V$ increases.
The improvement is large when $V$ goes from 2 to 5  ($27$\% for L, $10$\% for S) and small when $V$ goes from 5 to 10 and when $V$ goes from 10 to $n=500$ (each time, $8$\% for L, $2$\% for S).
Since the main influence of $V$ is on the variance of the $V$-fold penalty, these experiments confirm our interpretation of Theorem~\ref{theo.variance.penVF} in Section~\ref{sec.variance}: increasing $V$ helps much more from 2 to 5 or 10 than from 10 to $n$.

The picture is less clear for $V$-fold cross-validation, for which almost no difference is observed among $V \in \set{2,5,10,n}$---less than $2\%$---, and \ensuremath{C_{\mathrm{or}}}\ is minimized for $V\in \set{5,10}$.
Indeed, increasing $V$ simultaneously decreases the bias and the variance of the $V$-fold cross-validation criterion, leading to various possible behaviours of \ensuremath{C_{\mathrm{or}}}\ as a function of $V$, depending on the setting.
The same phenomenon has been observed in regression \cite{Arl:2008a}.

\paragraph{Other comments}
Table~\ref{tab.simus-results} confirms in the least-squares density estimation framework several facts previouly observed in least-squares regression \cite{Arl:2008a}:
\begin{itemize}
\item $\pendim$ performs much worse than $V$-fold penalization (except $V=2$ in setting L) with the collection Dya2. On the contrary, $\pendim$ does well with Regu (see Table~\ref{tab.complet.Li01.SiG5}), but $V$-fold penalization then performs as well.
\item $V$-fold penalization and $\E\scroch{\penid}$ perform much better when multiplying the penalty by some $C>1$.
The best overpenalization factor is $C=2$ for L-Dya2 and $C=1.5$ for S-Dya2, see Table~\ref{tab.complet.Li01.SiG5}.
Such a phenomenon can also be observed in regression \cite{Arl:2008a} and can explain in part the behaviour of $V$-fold cross-validation.
\item Once $V$-fold penalties are multiplied by a well-chosen $C$, they perform significantly better than $V$-fold cross-validation, except for the fact that $2$-fold cross-validation coincides with $2$-fold penalization multiplied by $1.5$ as shown by Lemma~\ref{le.penVF-VFCV}.
Nevertheless, making a bad choice for $C$ (which depends on the setting) can lead to worse performance with $V$-fold penalization, especially when $V = 2$, see Table~\ref{tab.complet.Li01.SiG5}.
\end{itemize}

In other settings considered in a preliminary phase of our experiments, differences between $V=2$ and $V=5$ were sometimes smaller or not significant, but always with the same ordering (that is, the worse performance for $V=2$ when $C$ is fixed).
In a few settings, for which the ``change-point'' in the smoothness of $\bayes$ was close to the median of $\bayes d\mu$, we found $\pendim$ among the best procedures with collection Dya2; then, $V$-fold penalization and cross-validation always had a performance very close to $\pendim$.
Both phenomena lead us to discard all settings for which there were no significant difference to comment.

\subsection{Variance as a function of $V$} \label{sec.simus.variance}
We now illustrate the results of Section~\ref{sec.variance} about the variance of $V$-fold penalization and the heuristic of Section~\ref{sec.oracle.key-quant} about its influence on model selection.
We focus on the unbiased case, that is, criteria $\CV_{(1,\mathcal{B})}$ with partitions $\mathcal{B}$ satisfying \eqref{hyp.part-reg.exact}.
Since the distribution of $(\CV_{(1,\mathcal{B})}(m))_{m \in \M_n}$ then only depends on $V = |\mathcal{B}|$, we write $\CV_V$ instead of $\CV_{(1,\mathcal{B})}$ by abuse of notation.
All results presented in this subsection have been obtained from $N=10\,000$ independent samples in setting S with a sample size $n=100$ and the collection Regu---for which models are naturally indexed by their dimension.

First, Figure~\ref{fig.variance.std.SiG5Regu.n100} shows the variance of $\Delta_{\CV_V}(m,\mo)=\CV_V(m)-\CV_V(\mo)$ as a function of the dimension $m$ of $S_m$, illustrating the conclusions of Theorem~\ref{theo.variance.penVF}: the variance decreases when $V$ increases.
More precisely, the variance decrease is significant between $V=2$ and $V=5$, an order of magnitude smaller between $V=5$ and $V=10$ and between $V=10$ and $V=n$, while the leave-one-out $\CV_n$ is hard to distinguish from the ideal penalized criterion $\CV_{\rm{id}}$.
On Figure~\ref{fig.variance.std.SiG5Regu.n100}, we can remark that for $m > \mo$,
\begin{equation*}
\var\paren{ \Delta_{\CV_V}(m,m^{\star}) } \approx \frac{1}{n^2} \croch{ K_1 \paren{ 1 + \frac{K_2}{V-1}}  + K_3 \paren{ 1 + \frac{K_4}{V-1}} (m - \mo) }
\end{equation*}
with $K_1 \approx 29$, $K_2 \approx 0.81$, $K_3 \approx 3.7$ and $K_4 \approx 3.8$.
The shape of the dependence on $V$ already appears in Theorem~\ref{theo.variance.penVF}, the above formula clarifies the relative importance of the terms called $a$ and $b$ in Section~\ref{sec.variance}, and their dependence on the dimension $m$ of $S_m$.
Remark the same behaviour holds when $n=500$ with very close values for $K_3$ and $K_4$ (see Figure~\ref{fig.variance.var.SiG5Regu.n500}),
as well as in setting L with $n=100$ or $n=500$ with $K_3 \approx 2.1$ and $K_4 \approx 4.2$ (see Figures~\ref{fig.variance.var.Li01Regu.n100} and~\ref{fig.variance.var.Li01Regu.n500}).
The fact that $K_4$ is close to $4$ in both settings confirms the term $1+4/(V-1)$ appearing  Theorem~\ref{theo.variance.penVF} indeed drives how $\var\sparen{ \Delta_{\CV_V}(m,m^{\star}) }$ depends on $V$.
\begin{figure}
\begin{minipage}[b]{.80\linewidth}
\includegraphics[width=\textwidth]{figure_var_Delta_oracle_regglobale_C140121_SiG5Regu_n100_seed2.pdf}
\end{minipage}
\caption{%
Illustration of the variance heuristic:
$\var(\Delta_{\CV}(m,\mo))$ as a function of $m$ for five different $\CV$.
Setting S-Regu, $n=100$.
The black diamond shows $\mo=7$.
The black lines show the linear approximation $n^{-2} [ 29 (1 + \frac{0.81}{V-1} ) + 3.7 ( 1 + \frac{3.8}{V-1} ) (m-\mo)]$ for $m > \mo$.
\label{fig.variance.std.SiG5Regu.n100}
}
\end{figure}

Figures~\ref{fig.variance.Pmh.SiG5Regu.n100} and~\ref{fig.variance.PhiSR.SiG5Regu.n100} respectively show $\Prob(\widehat{m}(\CV)=m)$ and its proxy $\overline{\Phi}(\SR_{\CV}(m))$ as a function of $m$ for $\CV=\CV_V$ with $V \in \set{2, 5, 10, n}$ and for $\CV=\CV_{\rm{id}}$.
First, remark both quantities behave similarly as a function of $m$ and $\CV$---see also Figure~\ref{fig.variance.SR-vs-Pmh.SiG5Regu.n100}---confirming empirically the heuristic of Section~\ref{sec.oracle.key-quant}.
The decrease of the variance observed on Figure~\ref{fig.variance.std.SiG5Regu.n100} when $V$ increases here translates into a better concentration of the distribution of $\widehat{m}(\CV_V)$ around $\mo$, which explains the performance improvement observed in Section~\ref{sec.simus.modsel}.
Figures~\ref{fig.variance.Pmh.SiG5Regu.n100}--\ref{fig.variance.PhiSR.SiG5Regu.n100} actually show how the decrease of the variance quantitatively influences the distribution of $\widehat{m}(\CV_V)$:
$\widehat{m}(\CV_5)$ is significantly more concentrated than $\widehat{m}(\CV_2)$, while the difference between $V=10$ and $V=5$ is much smaller and comparable to the difference between $V=n$ and $V=10$;
$\CV_n$ is hard to distinguish from $\CV_{\rm{id}}$.
Similar experiments with $n=500$ and in setting L are reported in Section~\ref{sec.supmat.simus}, leading to similar conclusions.
\begin{figure}
\begin{minipage}[b]{.80\linewidth}
\includegraphics[width=\textwidth]{figure_Proba_mh_zoom_C140121_SiG5Regu_n100_seed2.pdf}
\end{minipage}
\caption{%
$\Prob(\widehat{m}(\CV)=m)$ as a function of $m$ for five different $\CV$.
Setting S-Regu, $n=100$.
The black diamond shows $\mo=7$.
\label{fig.variance.Pmh.SiG5Regu.n100}
}
\end{figure}
\begin{figure}
\begin{minipage}[b]{.80\linewidth}
\includegraphics[width=\textwidth]{figure_Phibar_supratio_C140121_SiG5Regu_n100_seed2.pdf}
\end{minipage}
\caption{%
Illustration of the variance heuristic: $\overline{\Phi}(\SR_{\CV}(m))$ as a function of $m$ for five different $\CV$.
Setting S-Regu, $n=100$.
The black diamond shows $\mo=7$.
\label{fig.variance.PhiSR.SiG5Regu.n100}
}
\end{figure}

\section[Fast algorithm for computing V-fold penalties]{Fast algorithm for computing V-fold penalties for least-squares density estimation} \label{sec.algo}
Since the use of $V$-fold algorithms is motivated by computational reasons, it is important to discuss the actual computational cost of $V$-fold penalization and cross-validation as a function of $V$.
In the least-squares density estimation framework, two approaches are possible: a naive one---valid for all frameworks---and a faster one---specific to least-squares density estimation.
For clarifying the exposition, we assume in this section \eqref{hyp.part-reg.exact} holds true---so, $V$ divides $n$.
The general algorithm for computing the $V$-fold penalized criterion and/or the $V$-fold cross-validation criterion consists in training the estimator with data sets $(\xi_i)_{i \notin \mathcal{B}_j}$ for $j=1, \ldots, V$ and then testing each trained estimator on the data sets $(\xi_i)_{i \in \mathcal{B}_j}$ and/or $(\xi_i)_{i \notin \mathcal{B}_j}$.
In the least-squares density estimation framework, for any model $S_m$ given through an orthonormal family $\paren{\psil}_{\lamm}$ of elements of $L^2(\mu)$, we get the ``naive'' algorithm described and analysed more precisely in Section \ref{sec.algo.naive}, whose complexity is of order $n V \card(\Lambda_m)$.

Several simplifications occur in the least-squares density estimation framework, that allow to avoid a significant part of the computations made in  the naive algorithm.
\begin{proc} \label{proc.penVF.fast}\hfill \\
\vspace{-0.5cm}
\begin{enumerate}
\item[] {\bf Input:} $\mathcal{B}$ some partition of $\set{1,...,n}$ satisfying \eqref{hyp.part-reg.exact},
 $\xi_1, \ldots, \xi_n \in \mathcal{X}$
 and $\paren{\psil}_{\lamm}$ a finite orthonormal family of $L^2(\mu)$.
\item For $i \in \set{1, \ldots, V}$ and $\lamm$, compute $A_{i,\lambda} \egaldef \frac{V}{n} \sum_{j \in B_i} \psil(\xi_j) $
\item For $i,j \in \set{1, \ldots, V}$, compute $ C_{i,j} \egaldef \sum_{\lamm} A_{i,\lambda} A_{j,\lambda} $
\item Compute  $\mathcal{S} \egaldef \sum_{1 \leq i,j \leq V} C_{i,j}$ and $\mathcal{T} \egaldef \tr(C)\,$.
\item[] {\bf Output:} \\
Empirical risk: $P_n \gamma\paren{\ERM_m} = - \mathcal{S} / V^2$ \\
$V$-fold cross-validation criterion: $\critVF(m) = \frac{\mathcal{T}}{V(V-1)} - \frac{\mathcal{S} - \mathcal{T}}{(V-1)^2}$ \\
$V$-fold penalty:  $\penVF(m) = (\critVF(m) - P_n \gamma\paren{\ERM_m}) \frac{V-1/2}{V-1}$.
\end{enumerate}
\end{proc}
To the best of our knowledge, Algorithm~\ref{proc.penVF.fast} is new, even for computing the $V$-fold cross-validation criterion.
Its correctness and complexity are analyzed with the following proposition.
\begin{proposition} \label{pro.proc.penVF-fast.general-density}
Algorithm~\ref{proc.penVF.fast} is correct and has a computational complexity of order $(n + V^2) \card(\Lambda_m) \,$.

In the histogram case, that is, when $\Lambda_m$ is a partition of $\mathcal{X}$ and $\forall \lamm$, $\psil = \absj{\lambda}^{-1/2} \ensuremath{\mathbf{1}}_{\lambda}\,$, the computational complexity of Algorithm~\ref{proc.penVF.fast} can be reduced to the order of $n + V^2 \card(\Lambda_m)$.
\end{proposition}
Proposition~\ref{pro.proc.penVF-fast.general-density} is proved in Section~\ref{sec.app.proof.pro.algo}.
Note that closed-form formulas are available for the leave-$p$-out criterion in least-squares density estimation \cite{Cel:2008}, allowing to compute it with a complexity of order $n \card(\Lambda_m)$ in general, and smaller in some particular cases---for instance, $n$ for histograms.

\section{Discussion} \label{sec.discussion}
Before discussing how to choose $V$ when using $V$-fold methods for model selection, we state an additional result and we discuss the model selection literature in least-squares density estimation.

\subsection{Hold-out penalization} \label{sec.discussion.hold-out}
Our analysis of $V$-fold procedures for model selection can be extended to hold-out methods, that is, when data are split only once.
Similarly to the definition of the hold-out criterion in Eq.~\eqref{def:crit.HO}, the hold-out penalty is defined as
\begin{equation} \label{def.penHO}
\forall x \geq 0 \, , \quad
\penHO(m,\ItHO,x) \egaldef 2 x \paren{ P^{(\ItHO)}_n-P_{n}} \paren{ \ERM^{(\ItHO)}_m-\ERM_{m}}
\enspace ,
\end{equation}
that is, the hold-out estimator of the expectation of the ideal penalty, written as $\E\scroch{2 (P_n - P) (\ERM_m - \bayes_m)}$, see Eq.~\eqref{def.penid}.
We do not define $\penHO$ by Eq.~\eqref{eq.def.penVF}  with $V=1$ and $\ItHO=\mathcal{B}_1^c$---that is, the hold-out estimator of $\E\scroch{(P-P_n)\gamma(\ERM_m)}$, which amounts to removing the centering term $-\ERM_{m}$ in Eq.~\eqref{def.penHO}--- because this would dramatically increase its variability.
Note that adding such a term  $-\ERM_{m}$ in Eq.~\eqref{eq.def.penVF} does not change the value of the $V$-fold penalty under \eqref{hyp.part-reg.exact} since $\sum_{K=1}^V (P_n^{(\mathcal{B}_K^c)} - P_n) = 0$.

Denoting by $\tau=\absj{T}/n$, it comes from \lemref{DecPenHO} that
\[
\Ex{\penHO(m,\ItHO,x)}
= x \frac{1-\tau}{\tau} \Ex{\penid(m)}
\enspace.\]
In the following, we choose $x=C\tau/(1-\tau)$ so that $C=1$ corresponds to the unbiased case, as in the previous sections for the $V$-fold penalty.

\begin{remark}
Since $P_n=\tau P_n^{(\ItHO)}+(1-\tau)P_n^{(\ItHO^c)}$, by linearity of the estimator $\ERM_m$,
\begin{equation} \notag
\penHO(m,\ItHO,x) \egaldef 2 x (1-\tau)^2 \paren{ P^{(\ItHO)}_n-P^{(\ItHO^c)}_{n}} \paren{ \ERM^{(\ItHO)}_m-\ERM^{(\ItHO^c)}_{m}}
\end{equation}
which is symmetric in $\ItHO$ and $\ItHO^c$, hence
$\penHO(m,\ItHO^c,x)=\penHO(m,\ItHO,x)$.
In particular, if $|T|=n/2$, the $2$-fold penalty computed on the partition $\mathcal{B}=\sset{\ItHO,\ItHO^c}$ and the hold-out penalty coincide:
\begin{equation}
\notag
\forall x >0, \quad
\penVF(m,\sset{\ItHO,\ItHO^c},x) = \penHO(m,\ItHO,x)
\enspace .
\end{equation}
\end{remark}

\begin{theorem}\label{thm:PenHo}
Let $\xi_{\inter{n}}$ be i.i.d real-valued random variables with common density $\bayes \in L^{\infty}(\mu)$,
$T \subset \inter{n}$ with $\tau = |T|/n \in (0,1)$
and $(S_m)_{m\in\mathcal{M}_n}$ be a collection of separable linear spaces satisfying \eqref{hyp.NormSupNorm2}.
Assume that either \eqref{hyp.UBbayes} or \eqref{hyp.Nested} holds true.
Let $C \in (1/2,2]$ and \( \delta\egaldef 2(C-1)\).
For every $m \in \M_n$, let $\ERM_m$ be the projection estimator onto $S_m$ defined by Eq.~\eqref{def.ERM}, and
$\tilde{s}_{HO}= \ERM_{\widehat{m}_{\ensuremath{\mathrm{HO}}}}$ where
\begin{equation*
\widehat{m}_{\ensuremath{\mathrm{HO}}}
= \mathop{\argmintmp}_{m\in\mathcal{M}_{n}} \set{ P_n\gamma\paren{\ERM_{m}} +  \penHO\paren{m,\ItHO,\frac{C\tau}{1-\tau}}}
\enspace .
\end{equation*}
Then, an absolute constant $\kappa$ exists such that, for any $x>0$, defining $x_n = x + \log |\mathcal{M}_n|$, with probability at least $1-e^{-x}$, for any $\epsilon \in (0,1]$,
\begin{multline}
\label{eq.ineq-oracle.penHO}
\frac{1-\delta_{-}-\epsilon}{1+\delta_{+}+\epsilon}
\perte{ \tilde{s}_{HO} }\\
\leq \inf_{m \in \M_n} \perte{\ERM_m}+\kappa\paren{\frac{Ax_n}{ \epsilon n}+\frac{\tau^2+(1-\tau)^2}{\tau(1-\tau)}\frac{x_n^2}{ \epsilon^3 n}}\enspace .
\end{multline}
\end{theorem}
\thmref{PenHo} is proved in Section~\ref{sec.supmat.penHO.oracle}.

Theorem~\ref{thm:PenHo} extends Theorem~\ref{thm.oracle-penVF.cas_reel} to hold-out penalties, under similar assumptions.
As in Theorem~\ref{thm.oracle-penVF.cas_reel}, $\delta$ quantifies the bias of the hold-out penalized criterion, and plays the same role in the leading constant of the oracle inequality \eqref{eq.ineq-oracle.penHO}.

The main difference between Theorems~\ref{thm.oracle-penVF.cas_reel} and~\ref{thm:PenHo} lies in the remainder term.
For making proper comparisons, let $V$ be some divisor of $n$ and $T \subset \inter{n}$ such that $|T|=n-n/V$.
Then, the remainder term in Eq.~\eqref{eq.ineq-oracle.penHO} is larger than the one of Eq.~\eqref{eq.thm.oracle-penVF.cas_reel} in Theorem~\ref{thm.oracle-penVF.cas_reel} by a factor of order $V$
when $V$ is large.
These only are upper bounds, but at least they are consistent with the common intuition about the stabilizing effect of averaging over $V$ folds.

Similarly to Theorem~\ref{theo.variance.penVF}, the variance terms can be computed for the hold-out penalty in order to understand separately the roles of the training sample size and of averaging over the $V$ splits, in the $V$-fold criteria.
See Proposition~\ref{pro.variance.penHO} in Section~\ref{sec.supmat.penHO.variance} for details.

\subsection{Other model selection procedures for density estimation} \label{sec.discussion.other-proc}
Although the primary topic of the paper is the study of $V$-fold procedures, let us compare briefly our results to other oracle inequalities that have been proved in the least-squares density estimation setting.
For projection estimators, \cite[Section~7.2]{Mas:2003:St-Flour} prove an oracle inequality for some penalization procedures, but they are suboptimal since the leading constant $C_n$ does not tend to~1 as $n$ goes to $+\infty$.
Oracle inequalities have also been proved for other estimators: blockwise Stein estimators in \cite{Ri06} and some $T$-estimators in \cite{Bi08}.
Note the models considered in \cite{Bi08} are more general than ours, but the corresponding estimators are not computable in practice, and the oracle inequality in \cite{Bi08} also has a suboptimal constant $C_n$.
Some aggregation procedures also satisfy oracle inequalities, as proved for instance in \cite{RT07,Bun_Tsy_Weg_Bar:2010}.
Overall, under our assumptions, none of these results imply strictly better bounds than ours.

Let us finally mention \cite{BR03} proposed a precise evaluation of the penalty term in the case of regular histogram models and the log-likelihood contrast.
Their final penalty is a function of the dimension, only slightly modified compared to $\pendim$, performing very well on regular histograms.
These performances are likely to become much worse on the collection Dya2 presented in Section~\ref{sec.simus}.
This can be seen, for example, in Table~\ref{tab.complet.Li01.SiG5} where we presented the performances of $\pendim$ with different over-penalizing constants.

\subsection{Conclusion on the choice of $V$} \label{sec.discussion.choice-V}
Overall, choosing $V$ requires a trade-off between:
\begin{itemize}
\item Computational complexity, usually proportional to $V$, slightly different in the least-squares density estimation setting since it can be reduced to $(n+V^2) \card(\Lambda_m)$ or even to $n + V^2 \card(\Lambda_m)$, see Section~\ref{sec.algo}.
\item Statistical performance in terms of risk, which is better when the bias and the variance are small.
The bias decreases as $V$ increases for $V$-fold cross-validation, but it can be removed completely or fixed to any desired value by using $V$-fold penalization instead, see Lemma~\ref{le.penVF-VFCV}.
The variance decreases as $V$ increases, but it almost reaches its minimal value by taking, say, $V=5$ or $V=10$, as shown by theoretical and empirical arguments in Sections~\ref{sec.variance} and~\ref{sec.simus}.
\end{itemize}

The most common advices for choosing $V$ in the literature \cite[for instance][Section~7.10.1]{Has_Tib_Fri:2001v2009} are between $V=5$ and $V=10$.
This article provides clear evidence why taking $V$ larger does not reduce the variance significantly.
Concerning the bias, Lemma~\ref{le.penVF-VFCV} shows $5$-fold (resp. $10$-fold) cross-validation corresponds to overpenalization by a factor $1+1/8$ (resp. $1+1/18$), which is likely to be a good amount in many cases; in our simulation experiments, the best overpenalization factor is even larger, see also \cite{Arl:2008a}.

Note however our results are only valid for some least-squares algorithms, and it is reported in the literature \cite{Arl_Cel:2010:surveyCV} that $V$-fold cross-validation behaves differently as a function of $V$ in other settings.

\medskip

Finally, we would like to address the question of choosing between $V$-fold cross-validation and penalization.
The answer is rather simple---at least in least-squares density estimation---since Lemma~\ref{le.penVF-VFCV} shows $V$-fold cross-validation is a particular instance of $V$-fold penalization, with $C=1+1/(2(V-1))\,$.
So, if one wants to overpenalize by a factor $1+1/(2(V-1))$, $V$-fold cross-validation is definitely the good choice.
Otherwise, the best choice would be $V$-fold penalization with another value for $C$, depending on how much one wants to overpenalize.

\paragraph{Acknowledgments}
The authors thank gratefully Yannick Baraud and Guillaume Obozinski for precious comments on an earlier version of the paper, and Nelo Magalh\~{a}es for his careful reading and helpful remarks on this earlier version.
The authors also acknowledge the support of the French Agence Nationale de la Recherche (ANR) under reference ANR-09-JCJC-0027-01 ({\sc Detect} project)  and ANR 2011 BS01 010 01 (projet Calibration), and the first author acknowledges the support of the GARGANTUA project funded by the Mastodons program of CNRS.

\medskip

\bibliographystyle{alpha}

Statistics Theory (math.ST); Machine Learning (cs.LG)