Surprises in High-Dimensional Ridgeless Least Squares Interpolation

Abstract: Interpolators -- estimators that achieve zero training error -- have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In this paper, we study minimum $\ell_2$ norm ("ridgeless") interpolation in high-dimensional least squares regression. We consider two different models for the feature distribution: a linear model, where the feature vectors $x_i \in {\mathbb R}^p$ are obtained by applying a linear transform to a vector of i.i.d. entries, $x_i = \Sigma^{1/2} z_i$ (with $z_i \in {\mathbb R}^p$); and a nonlinear model, where the feature vectors are obtained by passing the input through a random one-layer neural network, $x_i = \varphi(W z_i)$ (with $z_i \in {\mathbb R}^d$, $W \in {\mathbb R}^{p \times d}$ a matrix of i.i.d. entries, and $\varphi$ an activation function acting componentwise on $W z_i$). We recover -- in a precise quantitative way -- several phenomena that have been observed in large-scale neural networks and kernel machines, including the "double descent" behavior of the prediction risk, and the potential benefits of overparametrization.

\section{Introduction}

Modern deep learning models involve a huge number of parameters.  In many
applications, current practice suggests that we should design
the network to be sufficiently complex so that the model (as trained, typically,
by gradient descent) interpolates the data, i.e., achieves zero training
error. Indeed, in a thought-provoking experiment, \citet{zhang2016understanding}
showed that state-of-the-art deep neural network architectures are complex
enough that they can be trained to interpolate the data even when the actual
labels are replaced by entirely random ones.

Despite their enormous complexity, deep neural networks are frequently observed
to generalize well in practice. At first sight, this seems to defy conventional
statistical wisdom: interpolation (vanishing training error) is commonly taken to
be a proxy for overfitting, poor generalization (large gap between training
and test error), and hence large test error.  In an insightful series of papers,
\citet{belkin2018understand,belkin2018does,belkin2018reconciling} pointed out
that these concepts are in general distinct, and interpolation does not
contradict generalization. For example, recent work has investigated interpolation ---via
kernel ridge regression--- in reproducing kernel Hilbert spaces \citep{liang2018just,ghorbani2019linearized}.
While in low dimension a positive regularization is needed to achieve good interpolation, in certain high
dimensional settings interpolation can be nearly optimal.

In this paper, we investigate these phenomena in the context of simple linear models.
We assume to be given i.i.d. data $(y_i,x_i)$, $i\le n$, with $x_i\in{\mathbb R}^p$ a feature vector and
$y_i\in {\mathbb R}$ a response variable. These are distributed according to the model (see Section \ref{sec:prelim}
for further definitions)
\begin{gather}
\label{eq:data_x}
(x_i, \epsilon_i) \sim P_x \times P_\epsilon, \quad i=1,\ldots,n, \\
\label{eq:data_y}
y_i = x_i^T \beta + \epsilon_i, \quad i=1,\ldots,n,
\end{gather}
where, $P_x$ is a
distribution on $\mathbb{R}^p$ such that $\mathbb{E}(x_i)=0$, $\mathrm{Cov}(x_i)=\Sigma$, and
$P_\epsilon$ is a distribution on $\mathbb{R}$ such that $\mathbb{E}(\epsilon_i)=0$,
$\mathrm{Var}(\epsilon_i)=\sigma^2$.

We estimate $\beta$ by linear regression.
Since our focus is on the overparametrized regime $p>n$,
the usual least square objective does not have a unique minimizer, and needs to be regularized.
We consider two approaches:  min-norm regression, which estimates $\beta$ by the
least squares solution with minimum $\ell_2$ norm; and ridge regression, which penalizes a coefficients
vector $\beta$ by its $\ell_2$ norm square $\|\beta\|_2^2$. We denote these estimates by $\hat{\beta}$ and $\hat{\beta}_{\lambda}$
($\lambda$ being the regularization parameter), and note that $\lim_{\lambda\to 0}\hat{\beta}_{\lambda}=\hat{\beta}$.
If the design matrix has full row rank, which is generically the case for $p>n$, the min-norm estimator is an interpolator,
namely $x_i^T\hat{\beta}=y_i$ for all $i\le n$.
In order to evaluate these methods, we will study the prediction risk at a new (unseen) test point $(y_0,x_0)$.

We study  the model \eqref{eq:data_y}  in the proportional
regime  $p\asymp n$,  with a special focus on the overparametrized
case $p>n$.
Our main contribution is to show that, by considering different choices of the features distribution $P_x$, we
 can reproduce a  number of statistically interesting phenomena that have emerged in the context of deep learning.

From a technical perspective, our main results are: Theorems \ref{thm:risk_gen} and \ref{thm:risk_ridge},
which assume the linear model
$x_i=\Sigma^{1/2} z_i$ with $z_i$ a vector with independent  coordinates;
and Theorem \ref{cor:purely_nonlinear}, which assumes a  nonlinear model
$x_i = \varphi(Wz_i)$ with $z_i\sim N(0,I_d)$. While the linear model
has already a attracted significant
amount of work (see Section \ref{sec:Related} for an overview), Theorems \ref{thm:risk_gen} and  \ref{thm:risk_ridge}
provide a more accurate non-asymptotic approximation of the prediction risk, as
compared to available results in the literature.

The prediction risk depends on the  geometry of the pair $(\Sigma,\beta)$. We consider
a few different choices for this geometry, which are broadly motivated by our objective to understand
overparametrized models, and specialize our formulas to these special cases:
\begin{enumerate}
\item \emph{Isotropic features.} This is the simplest case, in which  $\Sigma=I_p$ and therefore
  ---as we will see--- the asymptotic risk depends on $\beta$ only through its norm $\|\beta\|_2$. This simple model
  captures some interesting features of overparametrization, but misses others.

  We first consider a well specified case in which $x_i \in{\mathbb R}^p$ and we regress against $x_i$. We then pass to a  misspecified case,
 in which  the model \eqref{eq:data_y} holds for covariates  $x_i\in{\mathbb R}^{p+q}$, but we regress only against the first $p$ covariates.
\item \emph{Latent space features.} In the overparametrized regime, it is natural to
  assume that both the covariates $x_i$, and the coefficients vector $\beta$ lie close to a low-dimensional
  subspace.   In order to model this property, we assume $\Sigma = WW^T+I$, with $W\in{\mathbb R}^{p\times d}$, $d\ll p$,
  and $\beta$ lies in the span of the columns of $W$. Interestingly, this
  model reproduces many phenomena observed in more complex nonlinear models, and has a more direct
  connection to neural networks.
\item \emph{Nonlinear model.} In all of the previous cases, the distribution of $x_i$  is of the form $x_i=\Sigma^{1/2}z_i$
  where $z_i$ is a vector with independent coordinates. In order to test the generality of our results, we consider a
  model in which $x_i$ is obtained by passing  $z_i\sim{ N}(0,I_d)$ through a one-layer neural net with random first layer weights.
\end{enumerate}
We will summarize our results for these four examples in the next subsection.

A skeptical reader might ask what linear models have to do with neural networks.
We emphasize that linear models provide more than a simple analogy, and a recent line of work outlines
a concrete connection between the two settings
\citep{jacot2018neural,du2018gradient,du2018gradient2,allen2018convergence,chizat2018note}.
We will discuss this connection in Section \ref{sec:Connection}.

\subsection{Summary of results}

As mentioned above, we analyze the out-of-sample prediction risk of the minimum
$\ell_2$ norm (or min-norm, for short) least squares estimator, and of ridge-regularized
least squares.

We denote by $\gamma:=p/n \in (0,\infty)$ the overparametrization ratio.
When $\gamma < 1$, we call the problem \emph{underparametrized}, and
when $\gamma > 1$, we call it \emph{overparametrized}.
Our most general results for the linear model (Theorem \ref{thm:risk_gen} and \ref{thm:risk_ridge})
apply to a non-asymptotic setting in which $n,p$ are finite, and provide a deterministic approximation of
the risk with error bounds that are uniform in the distribution of the data.
We will occasionally consider a simpler asymptotic scenario in which both $p$ and $n$ diverge
with $p/n\to\gamma$.

We assume the model \eqref{eq:data_y} and denote by $\mathrm{SNR}= \|\beta\|_2^2/\sigma^2$ the
signal-to-noise ratio. We refer to Figure \ref{fig:summary} for supporting plots of the
asymptotic risk curves for different cases of interest.

\begin{figure}[htbp]
\centering
\includegraphics[width=0.725\textwidth]{fig/risk_sum_sr_cb.pdf}
\caption{\footnotesize Asymptotic risk curves for the linear feature model, as a
  function of the limiting aspect ratio $\gamma$.  Black and yellow: risks for min-norm least
  squares in the isotropic well specified model, for $\mathrm{SNR}=1$ and $\mathrm{SNR}=5$,
  respectively.  These two match for $\gamma<1$ but differ for $\gamma>1$.
  The null risks for $\mathrm{SNR}=1$ and $\mathrm{SNR}=5$ are marked by the dotted black and
  yellow lines, respectively. Light blue: risk for  a misspecified model with
  significant approximation bias ($a=1.5$ in \eqref{eq:poly_decay}), when
  $\mathrm{SNR}=5$. Green: optimally-tuned (equivalently, CV-tuned) ridge
  regression, in the same misspecified setup as for the light blue. Red: latent space model
  of Section \ref{sec:LatentSpace}, with $r=7$, $\sigma=0$. The
  points denote finite-sample risks, with $n=200$, $p=[\gamma n]$,
  across various values of $\gamma$.  Meanwhile, the  `$\times$' points mark finite-sample risks for a
  nonlinear feature model, with $n=200$, $p=[\gamma n]$, $d=100$, and
  $X=\varphi(ZW^T)$, where $Z$ has i.i.d.\ $N(0,1)$ entries, $W$ has i.i.d.\
  $N(0,1/d)$ entries, and $\varphi(t) = a(|t|-b)$ is a ``purely nonlinear''
  activation function, for constants $a,b$. Theorem \ref{cor:purely_nonlinear} predicts that this
  nonlinear risk should converge to the linear risk with $p$ features
  (regardless of $d$).}
\label{fig:summary}
\end{figure}

Our main results are twofold: $(i)$~We show that by taking $\Sigma\neq I_p$, we can easily construct scenarios
in which the minimum of the risk is achieved in the overparamertized regime $p>n$; $(ii)$~We show that these findings
are robust to the details of the distribution of $(y_i,x_i)$.

As a preliminary remark, note that in the underparametrized regime  ($\gamma<1$), the min-norm estimator
coincides with the standard least squares estimator. Its  risk is purely
  variance (there is no bias), and does not depend on $\beta,\Sigma$
  (see Proposition \ref{thm:risk_lo}).  Interestingly,  the asymptotic risk diverges as we approach
  the interpolation boundary (as $\gamma \to 1$).

   In contrast, in the overparametrized regime ($\gamma>1$), the risk is composed of both
  bias and variance\footnote{Note that in the overparametrized regime the bias is non-vanishing even in the interpolation
    limit $\lambda\to 0$. The reason is that the set of interpolators is an affine space of dimension $p-n$, and  the min-norm criterion selects one specific  interpolator which has norm smaller than $\beta$.},
  and generally depends on $\beta,\Sigma$ (see Theorem
  \ref{thm:risk_gen}).

We next highlight some concrete results for the four models discussed in the previous section
(unless explicitly said, we refer to the min-norm estimator).
\begin{description}
\item[Isotropic features.] The asymptotic risk depends on the coefficients vector only through its norm
  $\|\beta\|_2^2$ or, up to a scaling, on $\mathrm{SNR}= \|\beta\|_2^2/\sigma^2$.
  \begin{enumerate}
  \item If the model is \emph{well-specified}, we observe two different behaviors.
    For $\mathrm{SNR} \leq 1$, the risk is decreasing for $\gamma \in (1,\infty)$.
   For $\mathrm{SNR} > 1$, the risk has a {\it local} minimum on $\gamma \in
 (1,\infty)$.

 In either case, the risk approaches the null risk  as $\gamma \to \infty$, and achieves its global minimum
 in the underparametrized regime (see Section  \ref{sec:risk_iso}).
 \item If the model is \emph{misspecified}, when $\mathrm{SNR} > 1$, the risk can attain its {\it
    global} minimum in the overparametrized regime $\gamma \in (1,\infty)$ (when there is strong enough
  approximation bias, see Section \ref{sec:poly_bias}). However, the risk is again increasing for $\gamma$
  large enough.
  %
\item Optimally-tuned ridge regression uses a non-vanishing regularization $\lambda>0$,
  and dominates the min-norm least squares
  estimator in risk, across all values of $\gamma$ and $\mathrm{SNR}$,  both the
  well-specified and misspecified settings.  For a misspecified model,
  optimally-tuned ridge regression attains its global minimum around $\gamma=1$
  (see Section \ref{sec:ridge}).
  \item Optimal tuning of the ridge penalty can be achieved by leave-one-out cross-validation
  (see Theorem \ref{thm:risk_cv}).
\end{enumerate}
\item[Anisotropic features.] In this case $\Sigma\neq {{I}}$ and the risk depends on the geometry of $(\Sigma,\beta)$,
  and in particular on how $\beta$ aligns with the eigenvectors of $\Sigma$.
  %
  \begin{enumerate}
  \item If the coefficients vector is equidistributed along the eigenvectors of $\Sigma$, the behavior is
    qualitatively similar to the isotropic case. This situation arises, for instance, if $\beta$ is itself random with a spherical prior.
    %
  \item If $\beta$ is aligned with the top eigenvectors of $\Sigma$, the situation is qualitatively different.
    As an example we obtain an explicit formula for the asymptotic risk in the latent space model discussed above, see Figure
    \ref{fig:risk_latent_space_n} for an illustration.
    We find that, for natural choices of the model parameters, the risk is monotone decreasing in
  the overparametrized regime, and reaches its global minimum as $\gamma\to \infty$. This
  qualitative behavior matches the one observed for neural networks (see Section \ref{sec:LatentSpace}).
  %
\item For the latent space model, we observe that, at large overparametrization, the minimum error is
  achieved as $\lambda\to 0$, i.e. by min-norm interpolators (see Section \ref{sec:LatentRidge},
  and Section \ref{sec:Related} for related work).
\end{enumerate}
\item[Nonlinear model.] Finally, we consider a nonlinear model in which $x_i = \varphi(Wz_i)$
  where $\varphi$ is a non-linear activation function applied componentwise, $W\in{\mathbb R}^{p\times d}$,
  and $z_i\sim{ N}(0,I_d)$. We compute the limiting risk of min-norm
  regression for a purely non-linear activation, and  show that this  matches the one for Gaussian $x_i$.
  This confirms that the results found for the case $x_i=\Sigma^{1/2}z_i$ with $z_i$ an i.i.d. vector
  are likely to hold in substantially greater generality
  (see Theorem \ref{cor:purely_nonlinear}).
\end{description}

From a technical viewpoint, analysis of the isotropic covariates model is straightforward and relies on
standard random matrix theory results. However, we believe it provides useful insights.

In contrast, the results for general covariance and coefficients structure $(\Sigma,\beta)$ is
technically novel. We discuss related work in Section \ref{sec:Related}.
Our results for the  nonlinear
model are also novel. In this setting, we derive a new asymptotic result on resolvents of
certain block matrices, which may be of independent interest (see Lemma
\ref{thm:ResolventKernel}).

We next discuss some intuition behind and implications of our results.

\paragraph{Bias and variance}  The shape of the asymptotic risk curve for
min-norm least squares is, of course, controlled by its components: bias and
variance.  For fully specified models,  the bias increases with $\gamma$ in  the overparametrized regime,
which is intuitive.  When $p>n$, the min-norm least squares estimate of
$\beta$ is constrained to lie the row space of $X$, the training feature
matrix.  This is a subspace of dimension $n$ lying in a feature space of
dimension $p$.  Thus as $p$ increases, so does the bias, since this row space
accounts for less and less of the ambient $p$-dimensional feature space.

Meanwhile, we find that, in the overparametrized regime, the variance {\it decreases} with
$\gamma$.  This may seem counterintuitive at first, because it says, in a
sense, that the min-norm least squares estimator becomes {\it more}
regularized as $p$ grows.  However, this too can be explained intuitively, as
follows.  As $p$ grows, the minimum $\ell_2$ norm least squares solution---i.e.,
the minimum $\ell_2$ norm solution to the linear system $Xb=y$, for a
training feature matrix $X$ and response vector $y$---will generally have
decreasing $\ell_2$ norm.  Why?  Compare two such linear systems: in each, we
are asking for the min-norm solution to a linear system with the same $y$, but
in one instance we are given more columns in $X$, so we can generally
decrease the components of $b$ (by distributing them over more columns),
and achieve a smaller $\ell_2$ norm.  This can in fact be formalized
asymptotically, see Corollaries \ref{cor:norm_iso} and \ref{cor:norm_gen}.

\paragraph{Double descent} Recently, \citet{belkin2018reconciling} pointed out
a fascinating empirical trend where, for popular methods like neural networks
and random forests, we can see a {\it second} bias-variance tradeoff in the
out-of-sample prediction risk beyond the interpolation limit.  The risk curve
here resembles a traditional U-shape curve before the interpolation limit,
and then descends again beyond the interpolation limit, which these authors call
``double descent''.  A closely related phenomenon was found earlier by
\citet{spigler2018jamming}, who studied the ``jamming transition'' from
underparametrized to overparametrized neural networks. Our results formally
verify that this double descent phenomenon occurs even in the simple and
fundamental case of least squares regression.  The appearance of the second
descent in the risk, past the interpolation boundary ($\gamma=1$), is explained
by the fact that the variance decreases as $\gamma$ grows, as discussed above.

In the misspecified case, the variance still decreases with $\gamma$ (for the
same reasons), but interestingly, the bias can now also decrease with $\gamma$,
provided $\gamma$ is not too large (not too far past the interpolation
boundary). The intuition here is that in a misspecified model, some part of the
true regression function is always unaccounted for, and adding features
generally improves our approximation capacity. As a consequence, the double
descent phenomenon can be even more pronounced in the misspecified case
(depending on the strength of the approximation bias), and that the risk can
attain its global minimum past the interpolation limit.

Finally in the latent space model, we observe that the overall risk can be monotone decreasing
in the over parametrized regime, and attain its global minimum for large
overparametrization $\gamma\to \infty$ (after $p,n\to\infty$). In this case
we can write the design matrix as $X=ZW^T+U$, where $U$ is noise, and $Z$ is
the $n\times d$ matrix of latent covariates. Equivalently, the $i$-th column of $X$ (the $i$-th feature) takes the form $\tilde{x}_i = Zw_i+\tilde{u}_i$,
where $w_i$ is the $i$-th column of $W^T$ and $\tilde{u}_i$ is the $i$-th column of $U$.
Therefore, each  new feature provides new information about the underlying low-dimensional latent variables $Z$.
As $p$ gets large, ridge regression with respect to the feature matrix $X$ approximates increasingly well a
ridge regression with respect to the latent variables $Z$.

\paragraph{Interpolation versus regularization} The min-norm least squares
estimator can be seen as the limit of ridge regression as the tuning parameter
tends to zero. It is also the convergence point of gradient descent run on the
least squares loss.  We would not in general expect the best-predicting ridge
solution to be at the end of its regularization path. Our results,
comparing min-norm least squares to optimally-tuned ridge regression,
show that (asymptotically) this is never the case, when $\beta$ is
incoherent with respect to the eigenvectors of $\Sigma$. This is for instance the case
when $\Sigma=I_p$, or $\beta$ is distributed according to a spherically symmetric prior.
In contrast, \cite{kobak2020optimal} recently pointed out that ---when $\beta$ is aligned with the leading eigenvectors of
$\Sigma$--- min-norm regression can have optimal  risk (i.e. the optimal regularization vanishes).
We show that this is indeed the case in the latent space model mentioned above:
this provides indeed an extremely simple example of a phenomenon that has been observed in the past
for kernel methods \cite{liang2018just}.

As mentioned above, early-stopped gradient descent is known to
be closely connected to ridge regularization, see, e.g., \citet{ali2019continuous} which proves
a tight coupling between the two (see Section \ref{sec:Related} for further related work).

In practice, of course, we would not have access to the optimal tuning
parameter for ridge (optimal stopping for gradient descent), and we would rely
on, e.g., cross-validation (CV).  Our theory shows that for ridge regression, CV
tuning is asymptotically equivalent to optimal tuning (and we would expect
the same results to carry over to gradient descent, but have not pursued this
formally).

Historically, the debate between interpolation and regularization has been alive
for the last 30 or so years. Support vector machines find maximum-margin
decision boundaries, which often perform very well for problems where the Bayes
error is close to zero. But for less-separated classification tasks, one needs
to tune the cost parameter \citep{hastie2004entire}. Relatedly, in
classification, it is common to run boosting until the training error is
zero, and similar to the connection between gradient descent and $\ell_2$
regularization, the boosting path is tied to $\ell_1$ regularization
\citep{rosset2004boosting,tibshirani2015general}. Again, we now know that
boosting can overfit, and the number of boosting iterations should be treated
as a tuning parameter.

\subsection{Connection to neural networks}
\label{sec:Connection}

As mentioned above, recent literature has pointed out a direct connection between
linear models and more complex models such as neural networks
\citep{jacot2018neural,du2018gradient,du2018gradient2,allen2018convergence,chizat2018note}.
Our work contributes to this line of work and in particular points at the important role played by \emph{universality},
a core concept in random matrix theory \cite{tao2012topics}.

Assume to be given data i.i.d. $(y_i,z_i)$, $i\le n$, $y_i\in{\mathbb R}$, $z_i\in {\mathbb R}^d$.
Consider learning a neural network with parameters (weights) ${\boldsymbol{\theta}}\in{\mathbb R}^p$,
$f(\,\cdot\,;{\boldsymbol{\theta}}):{\mathbb R}^d\to{\mathbb R}$, $z\mapsto f(z;{\boldsymbol{\theta}})$. The specific form or
architecture of the network is not important for our discussion. However it is important to
emphasize that the form of $f$ need not be related to the actual distribution of $(y_i,{{z}}_i)$.

In some settings, the number of parameters $p$ is so
large that training effectively changes $\theta$ only by a small amount
with respect to a random initialization $\theta_0 \in \mathbb{R}^p$. It thus
makes sense to linearize the model around $\theta_0$. Further, supposing that
the initialization is such that $f(z;\theta_0) \approx 0$, and letting $\theta =
\theta_0 + \beta$, we can approximate the statistical model $z\mapsto f(z;{\boldsymbol{\theta}})$ by
\begin{equation}
\label{eq:linearized}
z\mapsto \nabla_\theta f(z;\theta_0)^T\beta\, .
\end{equation}
This model is still nonlinear in the input $z$, but is linear in the parameters $\beta$.
We are therefore led to consider a linear regression problem, with random
features $x_i=\nabla_\theta f(z_i;\theta_0)$, $i=1,\ldots,n$, of
high-dimensionality ($p$ much greater than $n$).  Notice that the features are
random because of the initialization $\theta_0$.  Further, since $p>n$, many vectors $\beta$
give rise to a model that interpolates the data.

The above scenario was made rigorous in a number of papers
\cite{jacot2018neural,du2018gradient,du2018gradient2,allen2018convergence,chizat2018note}. In
particular, \cite{chizat2018note} shows ---under some technical conditions--- that the linearization \eqref{eq:linearized}
can be accurate if the model is overparametrized
($p>n$), and closed under scalings (if $f(\,\cdot\, )$ is encoded by a neural network,
then $sf(\,\cdot\,)$ is also  a neural network for any $s\in{\mathbb R}$). Under these conditions, there exists a
scaling of the network's parameters such that gradient-based training converges to a model that can be
approximated arbitrarily well by \eqref{eq:linearized}. Further,
under the linearization \eqref{eq:linearized},
gradient descent converges to the interpolator that minimizes \footnote{Understanding the bias induced by
  gradient-based algorithms on fully nonlinear models is a broadly open problem, which has attracted considerable attention recently, see e.g.  \cite{gunasekar2018characterizing,gunasekar2018implicit}.} the $\ell_2$ norm $\|\beta\|_2$
(see Proposition \ref{prop:grad_ls} below).

What are the \emph{statistical consequences} of these linearization results? In principle, one could proceed
as follows: assume $\{(y_i,z_i)\}_{i\le n}$ to be i.i.d. samples with a certain population distribution $P_{y,z}$,
and then study the behavior of minimum $\ell_2$ norm interpolator of the form \eqref{eq:linearized}
under this data model. Of course, carrying out such a study in any generality is an outstanding mathematical challenge:
we refer to Section \ref{sec:Related} for some pointers to recent progress in this direction
after a first appearance of this manuscript.

A different approach relies on a  \emph{universality hypothesis}.
Recall that $\{(y_i,z_i)\}_{i\le n}$ have joint distribution $P_{y,z}$, and $x_i= \nabla_\theta f(z_i;\theta_0)$ and assume these
  quantities to have zero mean\footnote{We further note that the assumption of zero mean is mainly for convenience, as a
non-zero mean of the covariates can be effectively compensated by an intercept.}. Universality posits that  the asymptotic  risk of ridge regression (or min-norm interpolation) does not change   if we replace the joint distribution $(x_i,z_i)$ by a Gaussian with the same covariance.

While  universality is not expected to hold for any distribution of the data $(y_i,z_i)$,
and for any function $f$,  it has been confirmed in a number of examples of interest (see Section \ref{sec:Related}).
In fact, our analysis of the nonlinear model in Section \ref{sec:nonlinear} provides a concrete example in which universality
can be confirmed rigorously.

Under the universality hypothesis, we can assume $(z_i,x_i)$ to be jointly Gaussian.
If the true joint distribution of the data $(y_i,z_i)$ is also Gaussian, it is sufficient to study the data distribution
\begin{align*}
  &y_i = \beta^Tx_i+\epsilon_i,\;\;\;\;\;\; x_i\sim N(0,\Sigma)\, ,  \epsilon_i \sim N(0,\sigma^2) \,,
\end{align*}
where $\Sigma, \sigma$ are chosen to match the second order statistics of the original model.
This is a special case of our `linear model' \eqref{eq:data_y}.
Summarizing, replacing the original nonlinear model $y_i = f(z_i;\theta)$, with the
linear model  rests on two key steps. The first step is the linearization in
Eq.~\eqref{eq:linearized}: this has been proved to be a good approximation under
certain `lazy training' schemes \cite{chizat2018note}. The second step is the universality hypothesis,
which is quite common (albeit for simpler models) in random matrix theory \cite{tao2012topics}.

Finally, although we believe our our results are relevant to
understanding overparametrized neural networks, the concrete correspondence outlined
above only holds in a certain `lazy training' regime, in which network weights do not change much during training,
More generally, in a neural network, the feature representation and
the regression function or classifier are learned {\it simultaneously}.  In both
our linear and nonlinear model settings, the features $X$ are not learned, but
observed. Learning $X$ could significantly change some aspects of the behavior
of an interpolator. (See for instance Chapter 9 of \citet{goodfellow2016deep},
and also \citet{chizat2018note,zhang2019layers}, which emphasize the importance
of learning the representation.)

\subsection{Related work}
\label{sec:Related}

The present work connects to and is motivated by the recent interest in
interpolators in machine learning
\citep{belkin2018understand,belkin2018reconciling,liang2018just,
belkin2018does,geiger2019scaling}.
Several authors have argued that minimum $\ell_2$ norm least squares regression
captures the basic behavior of deep neural networks, at least in early (lazy)
training
\citep{jacot2018neural,du2018gradient,du2018gradient2,allen2018convergence,
zou2018stochastic,chizat2018note,lee2019wide}.
The connection between neural networks and kernel ridge regression arises when
the number of hidden units diverges. The same limit was also studied (beyond the
linearized regime) in
\cite{mei2018mean,rotskoff2018neural,sirignano2018mean,chizat2018global}.

Ridge regression with random features has been studied in the past.
\citet{dicker2016ridge}  considers a model in which the covariates
are isotropic Gaussian $x_i\sim N(0,I_p)$ and  computes the asymptotic risk of ridge
regression in the proportional asymptotics $p,n\to\infty$, with $p/n\to\gamma\in (0,\infty)$.
\citet{dobriban2018high} generalize these results to $x_i = \Sigma^{1/2}z_i$,
where $z_i$ has independent entries with bounded $12$-th moment.

Recently, \citet{advani2017high} study the effect of early stopping and ridge regularization
in a model with isotropic Gaussian covariates $x_i\sim N(0,I_p)$, again focusing on the
proportional asymptotics $p,n\to\infty$, with $p/n\to\gamma\in (0,\infty)$. They
show that this simple model reproduces several phenomena observed in neural networks training.
The same model is reconsidered in concurrent work by  \citet{belkin2019two}, who obtain
exact results for the expected risk of min-norm regression, relying on the
jointly Gaussian distribution of $(y_i,x_i)$.
We contribute to this line of work by extending the analysis to general covariance structures
and to misspecified models. As we will see, these generalizations allow to produce examples
for which the global minimum of the risk is achieved in the overparametrized regime $\gamma>1$.

The importance of the relation between the coefficient vector $\beta$ and the eigenvectors
of $\Sigma$ was emphasized by \citet{kobak2020optimal} and \citet{bartlett2020benign}.
These papers point out ---under different asymptotic settings--- that $\lambda= 0+$ (i.e., min-norm
regression) can be optimal or nearly optimal. After a preprint of this paper
appeared, \citet{wu2020optimal} and \citet{richards2020asymptotics} generalized our earlier
results to cover the case in which $\beta$ is potentially aligned with $\Sigma$.
We review in further detail these important generalizations in Section \ref{sec:correlated}.
We contribute to this line of work by obtaining non-asymptotic approximations
for the risk, with explicit and nearly optimal error bounds. These hold under weaker assumptions
on the geometry of $(\Sigma,\beta)$ than the results of \cite{wu2020optimal,richards2020asymptotics}.

For the nonlinear model, the random matrix theory literature is much sparser,
and focuses on the related model of kernel random matrices, namely, symmetric
matrices of the form $K_{ij}=\varphi(z_i^Tz_j)$.  \citet{el2010spectrum}
studied the spectrum of such matrices in a regime in which $\varphi$ can be
approximated by a linear function (for $i\neq j$) and hence the spectrum
converges to a rescaled Marchenko-Pastur law. This approximation does not hold
for the regime of interest here, which was studied instead by
\citet{cheng2013spectrum} (who determined the limiting spectral distribution)
and \citet{fan2015spectral} (who characterized the extreme eigenvalues). The
resulting eigenvalue distribution is the free convolution of a semicircle law
and a Marchenko-Pastur law. In the current paper, we must consider asymmetric
(rectangular) matrices $x_{ij} =\varphi(w_j^Tz_i)$, whose singular value
distribution was recently computed by \citet{pennington2017nonlinear}, using the
moment method. Unfortunately, the prediction variance depends on both the
singular values and vectors of this matrix. In order to address this issue, we
apply the leave-one out method of \citet{cheng2013spectrum} to compute the
asymptotics of the resolvent of a suitably extended matrix. We then extract the
information of interest from this matrix.
After appearance of a preprint of this paper, \citet{mei2019generalization} extended the
results presented here, to obtain a complete characterization of the risk
for the non-linear random features model.

Let us finally mention that the universality (or `invariance') phenomenon is quite common in random matrix
theory \cite{tao2012topics}. In the context of kernel inner product random matrices, it appears (somewhat implicitly)
in \cite{cheng2013spectrum} and (more explicitly) in \cite{fan2015spectral}.
After a first appearance of this manuscript, universality has been investigated in the context of neural networks
in several papers \cite{mei2019generalization,montanari2019generalization,gerace2020generalisation,goldt2020gaussian,hu2020universality,adlam2020neural}.

\subsection{Outline}

Section \ref{sec:prelim} provides important background. Sections
\ref{sec:isotropic}--\ref{sec:cv} consider the linear model case, focusing
on isotropic features, correlated features, misspecified models, ridge
regularization, and cross-validation, respectively.  Section \ref{sec:nonlinear}
covers the nonlinear model case.  Nearly all proofs are deferred until the
appendix.

\section{Preliminaries}
\label{sec:prelim}

We describe our setup and gather a number of important preliminary results.

\subsection{Data model and risk}

Assume we observe training data $(x_i,y_i) \in \mathbb{R}^p \times \mathbb{R}$, $i=1,\ldots,n$
from the model of Eqs.~\eqref{eq:data_x}, \eqref{eq:data_y}. We collect the responses in a vector $y \in
\mathbb{R}^n$, and the features in a matrix $X \in \mathbb{R}^{n\times p}$ (with rows $x_i \in
\mathbb{R}^p$, $i=1,\ldots,n$).

Consider a test point $x_0 \sim P_x$, independent of the training data. For an
estimator \smash{$\hat{\beta}$} (a function of the training data $X,y$), we define
its out-of-sample prediction risk (or simply, risk) as
$$
R_X(\hat{\beta};\beta) = \mathbb{E} \big[ ( x_0^T \hat{\beta} - x_0^T \beta )^2 \,|\, X \big]
= \mathbb{E} \big[ \| \hat{\beta} - \beta \|_\Sigma^2 \,| \, X \big],
$$
where $\|x\|_\Sigma^2 = x^T \Sigma x$.  Note that our definition of risk is
conditional on $X$ (as emphasized by our notation $R_X$).  Note also that we
have the bias-variance decomposition
$$
R_X(\hat{\beta};\beta) =
\underbrace{\|\mathbb{E}(\hat{\beta}|X) - \beta\|_\Sigma^2}_{B_X(\hat{\beta}; \beta)} {}+{}
\underbrace{{\sf Tr}[\mathrm{Cov}(\hat{\beta}|X) \Sigma]}_{V_X(\hat{\beta}; \beta)}.
$$

\subsection{Ridgeless least squares}

We consider the minimum $\ell_2$ norm (min-norm) least squares regression
estimator, of $y$ on $X$, defined by
\begin{equation}
\label{eq:min_ls}
\hat{\beta} = \arg\min \Big\{ \|b\|_2 : b \; \text{minimizes} \;
\|y-Xb\|_2^2 \Big\}\, .
\end{equation}
This can be equivalently written as $\hat{\beta} = (X^T X)^+ X^T y$, where   $(X^T X)^+$ is the pseudoinverse of $X^T X$.
An alternative name for \eqref{eq:min_ls} is the
``ridgeless'' least squares estimator, motivated by the fact that
\smash{$\hat{\beta} = \lim_{\lambda \to 0^+} \hat{\beta}_\lambda$}, where
\smash{$\hat{\beta}_\lambda$} denotes the ridge regression estimator:
\begin{equation}
\label{eq:ridge}
\hat{\beta}_\lambda = \arg\min_{b \in \mathbb{R}^p} \bigg\{ \frac{1}{n}\|y - Xb\|_2^2 +
\lambda \|b\|_2^2 \bigg\}  \, ,
\end{equation}
or, equivalently,
$\hat{\beta}_\lambda = (X^T X + n\lambda I)^{-1} X^T y$.

When $X$ has full column rank the
min-norm least squares estimator reduces to \smash{$\hat{\beta} = (X^T
  X)^{-1} X^T y$}, the usual least squares estimator.  When $X$ has rank $n$,
importantly, this estimator interpolates the training data: \smash{$y_i = x_i^T
  \hat{\beta}$}, for $i=1,\ldots,n$.

Lastly, the following is a well-known fact that connects the min-norm least
squares solution to gradient descent (as referenced in the introduction).
\begin{proposition}
\label{prop:grad_ls}
Initialize $\beta^{(0)} = 0$, and consider running gradient descent on the least
squares loss, yielding iterates
$$
\beta^{(k)} = \beta^{(k-1)} + t X^T (y-X\beta^{(k-1)}),
\quad k=1,2,3,\ldots,
$$
where we take $0 < t \leq 1/\lambda_{\max}(X^T X)$ (and $\lambda_{\max}(X^T
X)$ is the largest eigenvalue of $X^T X$).  Then \smash{$\lim_{k \to \infty}
  \beta^{(k)} = \hat{\beta}$}, the min-norm least squares solution in
\eqref{eq:min_ls}.
\end{proposition}

\begin{proof}
The choice of step size guarantees that \smash{$\beta^{(k)}$} converges to a
least squares solution as $k \to \infty$, call it \smash{$\tilde\beta$}.  Note
that \smash{$\beta^{(k)}$}, $k=1,2,3,\ldots$ all lie in the row space of $X$;
therefore \smash{$\tilde\beta$} must also lie in the row space of $X$; and the
min-norm least squares solution \smash{$\hat{\beta}$} is the unique least squares
solution with this property.
\end{proof}

\subsection{Bias and variance}

We recall expressions for the bias and variance of the min-norm least squares
estimator, which are standard.

\begin{lemma}
\label{lem:bias_var}
Under the model \eqref{eq:data_x}, \eqref{eq:data_y}, the min-norm least squares
estimator \eqref{eq:min_ls} has bias and variance
$$
B_X(\hat{\beta}; \beta) = \beta^T \Pi \Sigma \Pi \beta \quad \text{and} \quad
V_X(\hat{\beta}; \beta) = \frac{\sigma^2}{n} {\sf Tr} (\hat\Sigma^+ \Sigma),
$$
where \smash{$\hat\Sigma=X^T X/n$} is the (uncentered) sample covariance of $X$,
and \smash{$\Pi = I-\hat\Sigma^+\hat\Sigma$} is the projection onto the null space of
$X$.
\end{lemma}

\begin{proof}
As \smash{$\mathbb{E}(\hat{\beta}|X) = (X^T X)^+ X^T X \beta = \hat\Sigma^+ \hat\Sigma
  \beta$} and \smash{$\mathrm{Cov}(\hat{\beta}|X) = \sigma^2 (X^T X)^+ X^T X (X^T X)^+ =
  \sigma^2 \hat\Sigma^+/n$}, the bias and variance expressions follow from plugging
these into their respective definitions.
\end{proof}

\subsection{Underparametrized asymptotics}

We consider an asymptotic setup where $n,p \to \infty$, in such a way that
$p/n \to \gamma \in (0,\infty)$.  Recall that when $\gamma < 1$, we call the
problem underparametrized; when $\gamma > 1$, we call it overparametrized.
Here, we recall the risk of the min-norm least squares estimator in the
underparametrized case.  The rest of this paper focuses on the overparametrized
case.

The following is a known result in random matrix theory, and can be found in
Chapter 6 of \citet{serdobolskii2007multiparametric}.
It can also be found in the wireless communications literature, see Chapter 4 of
\citet{tulino2004random}.

\begin{proposition}
\label{thm:risk_lo}
Assume the model \eqref{eq:data_x}, \eqref{eq:data_y}, and assume $x \sim
P_x$ is of the form $x=\Sigma^{1/2} z$, where $z$ is a random vector with
i.i.d.\ entries that have zero mean, unit variance, and a finite 4th moment, and
$\Sigma$ is a (sequence of) deterministic positive definite matrix, such that
$\lambda_{\min}(\Sigma) \geq c > 0$, for all $n,p$ and a constant $c$
(here $\lambda_{\min}(\Sigma)$ is the smallest eigenvalue of $\Sigma$).
Then as $n,p \to \infty$, such that $p/n \to \gamma < 1$, the
risk of the least squares estimator \eqref{eq:min_ls} satisfies, almost surely,
$$
\lim_{n\to\infty}R_X(\hat{\beta}; \beta) = \sigma^2 \frac{\gamma}{1-\gamma}.
$$
\end{proposition}

As it can be seen from the last proposition, in the underparametrized case the risk is just variance.
In contrast, in the overparametrized case,
the bias \smash{$B_X(\hat{\beta}; \beta) = \beta^T \Pi
  \Sigma \Pi \beta$} is nonzero, because $\Pi$ is. This will be the focus of the next sections.

\section{Isotropic features}
\label{sec:isotropic}

We begin by considering the simpler case in which $\Sigma = I$. In this case
the limiting  bias is relatively straightforward to compute and depends on $\beta$ only through
$\|\beta\|_2^2$.  In Section \ref{sec:correlated}, we generalize our analysis and study the dependence
of the bias on the geometry of $\Sigma$ and $\beta$.

\subsection{Limiting bias}

As mentioned above, in the isotropic case the risk depends $\beta$ only on through
$r^2=\|\beta\|_2^2$.  To give some
intuition as to why this is true, consider the special case where $X$ has
i.i.d.\ entries from $N(0,1)$.  By rotational invariance, for any orthogonal $U
\in \mathbb{R}^{p \times p}$, the distribution of $X$ and $XU$ is the same.
Thus
\begin{align*}
B_X(\hat{\beta}; \beta) &= \beta^T \big(I - (X^T X)^+ X^T X \big) \beta \\
&\overset{d}{=} \beta^T \big(I - U^T (X^T X)^+ U U^T X^T X U\big) \beta \\
&= r^2 - (U \beta)^T (X^T X)^+ X^T X (U \beta).
\end{align*}
Choosing $U$ so that $U\beta=r e_i$, the $i$th standard basis vector, then
averaging over $i=1,\ldots,p$, yields
$$
\mathbb{E} B_X(\hat{\beta}; \beta)  = r^2 \mathbb{E}\big[ 1- {\sf Tr}\big((X^T X)^+X^T X\big)/p
\big] = r^2(1-n/p).
$$
It is possible to show that,  $B_X(\hat{\beta}; \beta)$ concentrates around its expectation ant therefore
$B_X(\hat{\beta}; \beta) \to r^2(1-1/\gamma)$, almost surely. This is stated formally in the next section.

\subsection{Limiting risk}
\label{sec:risk_iso}

As the next result shows, the independence of the risk on $\beta$ is still
true outside of the Gaussian case, provided the features are isotropic.
The next result can be proved as a corollary of the more general Theorem \ref{thm:risk_gen_asymp}
below. We give a simpler self-contained proof using a theorem of
\citet{rubio2011spectral} in Appendix \ref{sec:ProofRiskIso}.
\begin{theorem}
\label{thm:risk_iso}
Assume the model \eqref{eq:data_x}, \eqref{eq:data_y}, where $x \sim P_x$ has
i.i.d.\ entries with zero mean, unit variance, and a finite moment of order
$4+\eta$, for some $\eta>0$. Also assume that $\|\beta\|_2^2=r^2$ for all
$n,p$. Then for the min-norm least squares estimator \smash{$\hat{\beta}$} in
\eqref{eq:min_ls}, as $n,p \to \infty$, such that $p/n \to \gamma \in (0,\infty)$, it holds
almost surely that
\begin{align}
  B_X(\hat{\beta};\beta)&\to r^2\big(1-\frac{1}{\gamma}\big) \, ,\\
  V_X(\hat{\beta};\beta)&\to \sigma^2 \frac{1}{\gamma-1}\, .
\end{align}
Hence,  summarizing with Proposition \ref{thm:risk_lo}, we have
\begin{equation}
\label{eq:risk_iso}
R_X(\hat{\beta}; \beta) \to \begin{cases}
\sigma^2 \frac{\gamma}{1-\gamma} & \text{for $\gamma < 1$}, \\
r^2\big(1-\frac{1}{\gamma}\big) + \sigma^2 \frac{1}{\gamma-1} &
\text{for $\gamma > 1$}.
\end{cases}
\end{equation}
\end{theorem}

On $(0,1)$, there is no bias, and the variance increases with $\gamma$; on
$(1,\infty)$, the bias increases with $\gamma$, and the variance decreases with
$\gamma$. Let
$\mathrm{SNR}=r^2/\sigma^2$.  Observe that the risk of the null estimator
\smash{$\tilde\beta = 0$} is $r^2$, which we hence call the null risk.  The
following facts are immediate from the form of the risk curve in
\eqref{eq:risk_iso}.  See Figure \ref{fig:risk_iso} for an accompanying
plot when $\mathrm{SNR}$ varies from 1 to 5.

\begin{figure}[ht]
\centering
\includegraphics[width=0.725\textwidth]{fig/risk_id.pdf}
\caption{\footnotesize Asymptotic risk curves in \eqref{eq:risk_iso} for the min-norm
  least squares estimator, when $r^2$ varies from 1 to 5, and $\sigma^2=1$. For
  each value of $r^2$, the null risk is marked as a dotted line, and the points
  denote finite-sample risks, with $n=200$, $p=[\gamma n]$, across various
  values of $\gamma$, computed from features $X$ having i.i.d.\ $N(0,1)$
  entries.}
\label{fig:risk_iso}
\end{figure}

\begin{enumerate}
\item On $(0,1)$, the least squares risk $R(\gamma)$ is better than the null
  risk if and only if \smash{$\gamma < \frac{\mathrm{SNR}}{\mathrm{SNR}+1}$}.

\item On $(1,\infty)$, when $\mathrm{SNR} \leq 1$, the min-norm least squares
  risk $R(\gamma)$ is always worse than the null risk.  Moreover, it is
  monotonically decreasing, and approaches the null risk (from above) as $\gamma
  \to \infty$.

\item On $(1,\infty)$, when $\mathrm{SNR} > 1$, the min-norm least squares risk
  $R(\gamma)$ beats the null risk if and only if \smash{$\gamma >
    \frac{\mathrm{SNR}}{\mathrm{SNR}-1}$}.  Further, it has a local minimum at \smash{$\gamma=
    \frac{\sqrt{\mathrm{SNR}}}{\sqrt{\mathrm{SNR}}-1}$}, and approaches the null risk (from
  below) as $\gamma \to \infty$.
\end{enumerate}

\subsection{Limiting $\ell_2$ norm}

Calculation of the limiting $\ell_2$ norm of the min-norm least squares
estimator is quite similar to the study of the limiting risk in Theorem
\ref{thm:risk_iso}, and therefore we state the next result without proof.

\begin{corollary}
\label{cor:norm_iso}
Assume the conditions of Theorem \ref{thm:risk_iso}.  Then as $n,p \to \infty$,
such that $p/n \to \gamma$, the squared $\ell_2$ norm of the min-norm least
squares estimator \eqref{eq:min_ls} satisfies, almost surely,
$$
\mathbb{E}[\|\hat{\beta}\|_2^2 \,|\, X] \to
\begin{cases}
r^2 + \sigma^2 \frac{\gamma}{1-\gamma} & \text{for $\gamma < 1$}, \\
r^2 \frac{1}{\gamma} + \sigma^2 \frac{1}{\gamma-1} & \text{for $\gamma >
  1$}.
\end{cases}
$$
\end{corollary}

We can see that the limiting norm, as a function of $\gamma$, has a somewhat
similar profile to the limiting risk in \eqref{eq:risk_iso}: it is monotonically
increasing on $(0,1)$, diverges at the interpolation boundary, and is
monotonically decreasing on $(1,\infty)$.

\section{Correlated features}
\label{sec:correlated}

We broaden the scope of our analysis from the last section, where we
examined isotropic features.  In this section, we take $x \sim P_x$ to be of
the form  $x = \Sigma^{1/2} z$, where $z$ is a random vector with independent
entries that have zero mean and unit variance, and $\Sigma$ is arbitrary (but
still deterministic and positive definite).

The risk of min-norm regression depends on the geometry of $\Sigma$ and $\beta$.
Denote by $\Sigma = \sum_{i=1}^ps_i v_iv_i^T$ the eigenvalue
decomposition of $\Sigma$ with $s_1\ge s_2\ge \cdots\ge s_p\ge 0$.
The geometry of the problem is captured by the sequence of eigenvalues $(s_1,\dots,s_p)$,
and  by the coefficients of $\beta$ in the basis of eigenvectors $((v_1^T\beta),\dots, (v_p^T\beta))$. We encode
these via two probability distributions on ${\mathbb R}_{\ge0}$:
\begin{align}
  \widehat{H}_n(s) := \frac{1}{p}\sum_{i=1}^p 1_{\{s\ge s_i\}}\, ,\;\;\;\; \widehat{G}_n(s)  =\frac{1}{\|\beta\|_2^2}\sum_{i=1}^p \<\beta,v_i\>^2 1_{\{s\ge s_i\}}\, .
  \label{eq:HGdef}
\end{align}

We next state our assumptions about the data distribution: our results will be uniform with respect to the (large) constants $M$,
$\{C_k\}_{k\ge 2}$
appearing in this assumption.
\begin{assumption}\label{ass:CovariatesAssumption}
  The covariates vector $x\sim P_x$ is of the form $x = \Sigma^{1/2} z$, where, defining $\widehat{H}_n$ as per Eq.~\eqref{eq:HGdef}, we have
  %
  \begin{enumerate}
  \item[$(a)$] The vector  $z=(z_1,\dots, z_p)$ has independent (not necessarily identically distributed) entries with $\mathbb{E}\{z_i\}=0$,
    $\mathbb{E}\{z_i^2\}=1$, and $\mathbb{E}\{|z_i|^k\} \le C_k<\infty$ for all $k\ge 2$.
    %
  \item[$(b)$] $s_1= \|\Sigma\|_{op}\le M$, $\int s^{-1} d \widehat{H}_n(s)<M$.
    %
    \item[$(c)$] $|1-(p/n)|\ge 1/M$, $1/M\le p/n\le M$.
    %
      %
      \end{enumerate}
\end{assumption}
Condition $(a)$ bounds the tail probabilities on the covariates. Requiring finite moment of all order is
useful to get strong bounds on the deviations of the risk from its predicted value. As discussed below,
bounds on the first few moments are sufficient if we are satisfied in weaker probability bounds.

Conditions $(b)$ requires the eigenvalues of $\Sigma$ to be bounded, and not to
accumulate\footnote{The latter assumption could have been further relaxed, by requiring only $p_{+}$ of the
  $p$ eigenvalues to be non-vanishing and to satisfy the other conditions.  This would require to redefine $\gamma$
as $p_{+}/n$.} near $0$.
For the analysis of min-norm interpolation, we will add the additional assumption that the
minimum eigenvalue of $\Sigma$ is bounded away from zero. However
condition $(b)$ is sufficient for the analysis of ridge regression  in Section
\ref{sec:ridge}.

Finally, as our statements are non-asymptotic, we do not assume $p/n$  to converge to to a value. However
condition $(c)$ requires $p/n$ to be bounded and bounded away from the interpolation threshold $p/n=1$.

\subsection{Prediction risk}

\begin{definition}[Predicted bias and variance: min-norm regression]\label{def:LimitBiasVar}
    Let $\widehat{H}_n$ be the empirical distribution of eigenvalues of $\Sigma$,
    and $\widehat{G}_n$ the reweighted distribution as per Eq.~\eqref{eq:HGdef}.
    For $\gamma\in{\mathbb R}_{>0}$,
  define $c_0 = c_0(\gamma,\widehat{H}_n)\in{\mathbb R}_{> 0}$ to be the
  unique non-negative solution of
  %
  \begin{align}
    1-\frac{1}{\gamma} = \int \frac{1}{1+c_0\gamma s}\, d\widehat{H}_n(s)\, , \label{eq:c0def}
  \end{align}
  %
  We then define the predicted bias and variance by
  %
  \begin{align}
    \mathscrsfs{B}(\widehat{H}_n,\widehat{G}_n,\gamma) &:=\|\beta\|_2^2\left\{1+\gamma  c_0\frac{\int \frac{s^2}{(1+c_0\gamma s)^2}\, d\widehat{H}_n(s)}
                               {\int \frac{s}{(1+c_0\gamma s)^2}\, d\widehat{H}_n(s)}\right\}\cdot  \int \frac{s}{(1+c_0\gamma s)^2}\, d\widehat{G}_n(s)\, ,
    \label{eq:BiasPredMinNorm}\\
    \mathscrsfs{V}(\widehat{H}_n,\gamma) &:= \sigma^2\gamma  \frac{\int \frac{s^2}{(1+c_0\gamma s)^2}\, d\widehat{H}_n(s)}
{\int \frac{s}{(1+c_0\gamma s)^2}\, d\widehat{H}_n(s)}\, .
  \end{align}
\end{definition}

Note that evaluating $\mathscrsfs{B}(H,G,\gamma)$,  $\mathscrsfs{V}(H,\gamma)$ numerically is relatively straightforward,
with the most complex part being the solution of Eq.~\eqref{eq:c0def}.
The next theorem establishes that ---under suitable technical conditions---
the functions $\mathscrsfs{B}$, $\mathscrsfs{V}$ characterize the test error. Similar theorems were proved in
\cite{wu2020optimal,richards2020asymptotics}, which generalized an earlier version of this manuscript
to account for the geometry of $(\Sigma,\beta)$.
\begin{theorem}
\label{thm:risk_gen}
Assume the  data model \eqref{eq:data_x}, \eqref{eq:data_y} and that the covariates distribution satisfies
Assumption \ref{ass:CovariatesAssumption}. Further assume $s_p = \lambda_{\min}(\Sigma)>1/M$.
Define $\gamma=p/n$  and let
$\hat{\beta}$ be the min-norm least squares
estimator in Eq.~\eqref{eq:min_ls}.

Then for any constants $D>0$ (arbitrarily large)
  there exist $C=C(M,D)$  such that, with probability at least $1-Cn^{-D}$ the following hold
\begin{align}
  &R_X(\hat{\beta};\beta)  =B_X(\hat{\beta};\beta)  +V_X(\hat{\beta};\beta) \, ,\\
  &\big|B_X(\hat{\beta};\beta)-\mathscrsfs{B}(\widehat{H}_n,\widehat{G}_n,\gamma) \big|\le \frac{C\|\beta\|_2^2}{n^{1/7}}\, ,\label{eq:MinNormBias}\\
  &\big|V_X(\hat{\beta};\beta)-\mathscrsfs{V}(\widehat{H}_n,\gamma) \big|\le \frac{C}{n^{1/7}}\, ,\label{eq:MinNormVariance}
\end{align}
where $\mathscrsfs{B}$ and $\mathscrsfs{V}$ are given in Definition \ref{def:LimitBiasVar-lambda}.
\end{theorem}
 The proof of this theorem is deferred to Section \ref{sec:ProofGeneralFormulaMinNorm}.

The order of the error bound in Eqs.~\eqref{eq:MinNormBias}, \eqref{eq:MinNormVariance} is not optimal:
a central limit theorem heuristics suggests the deterministic approximation to be accurate up to  an
error of order $n^{-1/2}$. Indeed, we are able to establish the correct order in the case of ridge regression, see
Theorem \ref{thm:risk_ridge}.

 \begin{remark}\label{rmk:Comparison}
Note that Theorem \ref{thm:risk_gen} establishes deterministic approximations for the bias and variance,
that are valid at finite $n,p$. The overparametrization ratio $\gamma=p/n$ is a non-asymptotic
quantity, and the error bounds are uniform, i.e. depend uniquely on the constant $M$.
This is to be contrasted with the asymptotic setting of \cite{richards2020asymptotics,wu2020optimal}.
Both of these papers assume a sequence of regression problems with $n,p\to\infty$, $p/n\to \gamma$,
and obtain an asymptotically exact expression for the risk.

In order for the asymptotics to make sense, additional assumptions are required by \cite{richards2020asymptotics,wu2020optimal}.
In \cite{richards2020asymptotics}, this is achieved by assuming $\beta$ to be random
with $\mathbb{E}[\beta\beta^T]=r^2\Phi(\Sigma)/d$ for a certain (deterministic) function $\Phi:{\mathbb R}\to{\mathbb R}$
(promoted to a function on matrices in the usual way). In addition,
the empirical spectral distribution of $\Sigma$ is assumed to converge.
To state the assumptions in \cite{wu2020optimal}, recall that $(s_i)_{i\le p}$ are the eigenvalues
of $\Sigma$, and denote by $b_i = p\cdot (v_i^T\beta)^2$ the projection of $\beta$ onto the eigenvectors of $\Sigma$. Then \cite{wu2020optimal} assumes that the joint empirical distribution
$p^{-1}\sum_{i=1}^{p}\delta_{s_i,b_i}$ converges weakly as $n,p\to\infty$.

Technically, \cite{richards2020asymptotics,wu2020optimal} apply asymptotic random matrix theory results, such as
 \cite{ledoit2011eigenvectors}, while we have to take a longer detour to exploit
non-asymptotic results established in\cite{knowles2017anisotropic}. We believe that the non-asymptotic approach
provides more concrete and accurate statements.
\end{remark}

 Theorem \ref{thm:risk_gen} also implies asymptotic predictions under minimal assumptions.
  In particular,  if the two  probability measures $\widehat{H}_n$, $\widehat{G}_n$ converge weakly to probability measures
  $H$, $G$ on $[0,\infty)$, then we obtain $B_X(\hat{\beta};\beta)/\|\beta\|^2\to \mathscrsfs{B}(H,G,\gamma)$,
  $V_X(\hat{\beta};\beta)\to \mathscrsfs{V}(H,\gamma)$ almost surely.
\begin{theorem}\label{thm:risk_gen_asymp}
  Consider the setting of Theorem \ref{thm:risk_gen} but, instead of Assumption \ref{ass:CovariatesAssumption} $(a)$,
  assume that $(z_i)_{i\le p}$ are identically distributed and satisfy the conditions
  $\mathbb{E} z_i= 0$, $\mathbb{E}(z_i^2) = 1$, $\mathbb{E}(|z_i|^{4+\delta})\le C<\infty$.
  Further assume $p/n\to \gamma\in (0,\infty)$, $\widehat{H}_n\Rightarrow H$, $\widehat{G}_n\Rightarrow G$. Then,
  almost surely $B_X(\hat{\beta};\beta)/\|\beta\|^2_2\to \mathscrsfs{B}(H,G,\gamma)$,
  $V_X(\hat{\beta};\beta)\to \mathscrsfs{V}(H,\gamma)$.
 \end{theorem}
 %
As pointed out above the condition $\widehat{H}_n\Rightarrow H$, $\widehat{G}_n\Rightarrow G$ (here $\Rightarrow$ denotes weak convergence)
is strictly weaker than the condition assumed in \cite{wu2020optimal} to establish asymptotic results.
Further, we require weaker moment conditions.

  In the next sections we illustrate the role of the geometry of $\beta,\Sigma$
  by considering two models for which $\widehat{H}_n\Rightarrow H$, $\widehat{G}_n\Rightarrow G$ as $n,p\to\infty$. First we consider the case $G=H$,
  which we refer to as `equidistributed:' the components of $\beta$ are roughly equally distributed along the
  eigenvectors of $\Sigma$. In this case there is no special relation between $\beta$ and $\Sigma$.

  As a further application, we consider a latent space model in which $\beta$ is aligned with the top eigenvectors of $\Sigma$. This can be regarded as a misspecified model, and is therefore presented in
  Section \ref{sec:LatentSpace} below.

  \subsection{Equidistributed coefficients}

  In this section we assume $G=H$, $\|\beta\|_2\to r$,
  and $p/n\to \gamma$.
  One way to generate
  $\beta$ satisfying this condition is to draw it uniformly at random on the $p$-dimensional sphere
  of radius $\|\beta\|_2 = r$. In this case the conditions of Theorem \ref{thm:risk_gen} (or Theorem \ref{thm:risk_gen_asymp}), hold
  with $\widehat{G}_n\to G=H$.
    %
  \begin{corollary}\label{coro:equidistributed}
    Under the assumptions of Theorem \ref{thm:risk_gen_asymp},
    further assume $G=H$,  $\|\beta\|^2\to r^2$. Then
    for $n,p \to \infty$, with $p/n \to \gamma > 1$, almost surely
    %
    \begin{align}
      %
      B_X(\hat{\beta};\beta) & \to \mathscrsfs{B}_{\mbox{\tiny \rm equi}}(H,\gamma) := \frac{r^2}{c_0(H,\gamma)\gamma^2}\, ,\label{eq:Bequi}\\
       V_X(\hat{\beta};\beta) & \to \mathscrsfs{V}_{\mbox{\tiny \rm equi}}(H,\gamma) := \mathscrsfs{V}(H,\gamma) \, .
    \end{align}
    \end{corollary}
    %
    As a special case, we can revisit the isotropic case $\Sigma=I$, which results in $dH = \delta_1$. In this case
    $c_0(H,\gamma) = \gamma(\gamma-1)$ yielding immediately $\mathscrsfs{B}_{\mbox{\tiny \rm equi}}(H,\gamma) =1-\gamma^{-1}$ and
    $\mathscrsfs{V}_{\mbox{\tiny \rm equi}}(H,\gamma) =1/(\gamma-1)$.

\subsection{Limiting $\ell_2$ norm}

Again, as in the isotropic case, analysis of the limiting $\ell_2$ norm is
similar to analysis of the risk in Theorem \ref{thm:risk_gen}. We give the next result without proof, as it is an immediate generalization of previous results.
\begin{corollary}
  \label{cor:norm_gen}
    Under the assumptions of Theorem \ref{thm:risk_gen_asymp},
    further assume  $\|\beta\|^2\to r^2$, and let $c_0=c_0(H,\gamma)$ be defined as there.
Then as $n,p \to \infty$,
such that $p/n \to \gamma$,  the min-norm least
squares estimator \eqref{eq:min_ls} satisfies, almost surely,
\begin{align}
\|\hat{\beta}\|_2^2 \to
\begin{dcases}
r^2 + \sigma^2 \frac{\gamma}{1-\gamma} \int \frac{1}{s} \, dH(s)& \text{for $\gamma < 1$}, \\
\int\frac{c_0\gamma s}{1+c_0\gamma s}\,  dG(s) + c_0\gamma\sigma^2  & \text{for $\gamma >
  1$},
\end{dcases}
 \end{align}
\end{corollary}

\section{Misspecified model}
\label{sec:misspecified}

In this section, we consider a misspecified model, in which the regression
function is still linear, but we observe only a subset of the features. Such a
setting provides another potential motivation for interpolation: in
many problems, we do not know the form of the regression function, and we
generate features in order to improve our approximation capacity.  Increasing
the number of features past the point of interpolation (increasing $\gamma$ past
1) can now decrease \emph{both} bias and variance.

As such, the misspecified model setting also yields further interesting
asymptotic comparisons between the $\gamma<1$ and $\gamma>1$ regimes.
Recall the isotropic features model of Section \ref{sec:risk_iso}: the risk function in \eqref{eq:risk_iso} is
globally minimized at $\gamma = 0$. This is a consequence of the fact that, in a well-specified linear model
at $\gamma=0$ there is no bias and no variance, and hence no risk.  In a misspecified model, we will see that
the story can be quite different, and the asymptotic risk can actually attain
its \emph{global} minimum on $(1,\infty)$.

\subsection{Data model and risk}

Consider, instead of \eqref{eq:data_x}, \eqref{eq:data_y}, a data model
\begin{gather}
\label{eq:data_x_mis}
\big((x_i, w_i), \epsilon_i\big) \sim P_{x,w} \times P_\epsilon, \quad
i=1,\ldots,n, \\
\label{eq:data_y_mis}
y_i = x_i^T \beta + w_i^T \theta + \epsilon_i, \quad i=1,\ldots,n,
\end{gather}
where as before the random draws across $i=1,\ldots,n$ are independent. Here,
we partition the features according to $(x_i,w_i) \in \mathbb{R}^{p+q}$, $i=1,\ldots,n$,
where the joint distribution $P_{x,w}$ is such that $\mathbb{E}((x_i,w_i))=0$ and
$$
\mathrm{Cov}\big((x_i,w_i)\big) = \Sigma =
\begin{bmatrix}
\Sigma_x & \Sigma_{xw} \\
\Sigma_{xw}^T & \Sigma_w
\end{bmatrix}.
$$
We collect the features in a block matrix $[\,X \; W\,] \in \mathbb{R}^{n \times (p+q)}$
(which has rows $(x_i,w_i) \in \mathbb{R}^{p+q}$, $i=1,\ldots,n$).  We presume that $X$
is observed but $W$ is unobserved, and focus on the min-norm least squares
estimator exactly as before in \eqref{eq:min_ls}, from the regression of $y$ on
$X$ (not the full feature matrix $[\,X \; W\,]$).

Given a test point $(x_0,w_0) \sim P_{x,w}$, and an estimator \smash{$\hat{\beta}$}
(fit using $X,y$ only, and not $W$), we define its out-of-sample
prediction risk as
$$
R_X(\hat{\beta};\beta,\theta) =
\mathbb{E} \big[ \big( x_0^T \hat{\beta} - \mathbb{E}(y_0|x_0,w_0) \big)^2 \,|\, X \big] =
\mathbb{E} \big[ \big( x_0^T \hat{\beta} -x_0^T \beta - w_0^T \theta \big)^2 \,|\, X \big].
$$
Note that this definition is conditional on $X$, and we are integrating over the
randomness not only in $\epsilon$ (the training errors), but in the unobserved
features $W$, as well.  The next lemma decomposes this notion of risk in
a useful way.

\begin{lemma}
\label{lem:risk_mis_decomp}
Under the misspecified model \eqref{eq:data_x_mis}, \eqref{eq:data_y_mis}, for
any estimator \smash{$\hat{\beta}$}, we have
$$
R_X(\hat{\beta}; \beta, \theta) =
\underbrace{\mathbb{E} \big[ \big( x_0^T \hat{\beta} - \mathbb{E}(y_0|x_0) \big)^2 \,|\, X
  \big]}_{R^*_X(\hat{\beta}; \beta, \theta)} +
\underbrace{\mathbb{E} \big[ \big( \mathbb{E}(y_0|x_0) - \mathbb{E}(y_0|x_0,w_0) \big)^2
  \big]}_{M(\beta,\theta)}.
$$
\end{lemma}

\begin{proof}
Simply add an subtract $\mathbb{E}(y_0|x_0)$ inside the square in the definition of
\smash{$R_X(\hat{\beta}; \beta, \theta)$}, then expand, and note that the cross term
vanishes because
$\mathbb{E} \big[\big( \mathbb{E}(y_0|x_0) - \mathbb{E}(y_0|x_0,w_0) \big) \,|\, x_0 \big] = 0$.
\end{proof}

The first term \smash{$R^*_X(\hat{\beta}; \beta, \theta)$} in the decomposition in
Lemma \ref{lem:risk_mis_decomp} is precisely the risk that we studied
previously in the well-specified case, except that the response distribution
has changed (due to the presence of the middle term in \eqref{eq:data_y_mis}).
We call the second term \smash{$M(\beta, \theta)$} in Lemma
\ref{lem:risk_mis_decomp} the {\it misspecification bias}.

\begin{remark}
If $(x_i,w_i)$ are jointly Gaussian, then the above expressions simplify and  Theorem \ref{thm:risk_gen}
can be used to characterize the risk \smash{$R_X(\hat{\beta}; \beta, \theta)$}.
In particular, the conditional distribution of $w$ given $x$ is $P_{w|x} = N(\Sigma_{wx}\Sigma_{x}^{-1}x,\Sigma_{w|x})$
where $\Sigma_{wx}=\Sigma_{xw}^T$, and $\Sigma_{w|x} =\Sigma_w-\Sigma_{wx}\Sigma_x^{-1}\Sigma_{wx}^T$.
Further, $y= \tilde{\beta}^Tx +\tilde{\epsilon}$, where $\tilde\beta = \beta+\Sigma_x^{-1}\Sigma_{xw}\theta$,
and $\tilde{{\varepsilon}}\sim N(0,\tilde\sigma^2)$, $\tilde\sigma^2=\sigma^2+\theta^T\Sigma_{w|x}\theta$.
It is then easy to show that the misspecification bias is $M(\beta,\theta) = \theta^T\Sigma_{w|x}\theta$
and the term $R^*_X(\hat{\beta}; \beta, \theta)$ can be approximated using Theorem \ref{thm:risk_gen}.
\end{remark}

In order to discuss some qualitative features, we focus on the simplest possible model by assuming independent
covariates.

\subsection{Isotropic features}
\label{sec:risk_mis_iso}

Here, we make the additional
simplifying assumption that $(x,w) \sim P_{x,w}$ has i.i.d.\ entries with unit
variance, which implies that $\Sigma=I$.  (The case of independent features but
general covariances $\Sigma_x,\Sigma_w$ is similar, and we omit the details.)
Therefore, we may write the response distribution in \eqref{eq:data_y_mis} as
$$
y_i = x_i^T \beta + \delta_i, \quad i=1,\ldots,n,
$$
where $\delta_i$ is independent of $x_i$, having mean zero and variance
$\sigma^2+\|\theta\|_2^2$, for $i=1,\ldots,n$.  Denote the
total signal by $r^2=\|\beta\|_2^2+\|\theta\|_2^2$, and the fraction of the
signal captured by the observed features by $\kappa=\|\beta\|_2^2/r^2$. Then
\smash{$R_X^*(\hat{\beta};\beta,\theta)$} behaves exactly as we computed previously,
for isotropic features in the well-specified setting (Theorem \ref{thm:risk_lo}
for $\gamma < 1$, and Theorem \ref{thm:risk_iso} for $\gamma>1$), after we make
the substitutions:
\begin{equation}
\label{eq:subs}
r^2 \mapsto r^2 \kappa \quad \text{and} \quad
\sigma^2 \mapsto \sigma^2 + r^2 (1-\kappa).
\end{equation}
Furthermore, we can easily calculate the misspecification bias:
$$
M(\beta,\theta) = \mathbb{E}(w_0^T \theta)^2 = r^2 (1-\kappa).
$$
Putting these results together leads to the next conclusion.

\begin{theorem}
\label{thm:risk_mis_iso}
Assume the misspecified model \eqref{eq:data_x_mis}, \eqref{eq:data_y_mis}, and
assume $(x,w) \sim P_{x,w}$ has i.i.d.\ entries with zero mean, unit variance,
and a finite moment of order $8+\eta$, for some $\eta>0$.  Also assume that
$\|\beta\|_2^2+\|\theta\|_2^2=r^2$ and $\|\beta\|_2^2/r^2 = \kappa$ for all
$n,p$. Then for the min-norm least squares estimator \smash{$\hat{\beta}$} in
\eqref{eq:min_ls}, as $n,p \to \infty$, with $p/n \to \gamma$, it holds almost
surely that
$$
R_X(\hat{\beta};\beta,\theta) \to
\begin{cases}
r^2(1-\kappa) + \big(r^2(1-\kappa) + \sigma^2\big)
\frac{\gamma}{1-\gamma} & \text{for $\gamma < 1$}, \\
r^2(1-\kappa) + r^2 \kappa \big(1-\frac{1}{\gamma}\big)
+ \big(r^2(1-\kappa) + \sigma^2\big) \frac{1}{\gamma-1} &
\text{for $\gamma > 1$}.
\end{cases}
$$
\end{theorem}

We remark that, in the independence setting considered in Theorem
\ref{thm:risk_mis_iso}, the dimension $q$ of the unobserved feature space does
not play any role: we may equally well take $q=\infty$ for all $n,p$
(i.e., infinitely many unobserved features).

The components of the limiting risk from Theorem \ref{thm:risk_mis_iso} are
intuitive and can be interpreted as follows.  The first term $r^2(1-\kappa)$ is
the misspecification bias (irreducible).  The second term, which we deem as 0
for $\gamma<1$ and $r^2 \kappa (1-1/\gamma)$ for $\gamma>1$, is the bias.  The
third term, $r^2(1-\kappa) \gamma/(1-\gamma)$ for $\gamma<1$ and $r^2(1-\kappa)
/(\gamma-1)$ for $\gamma>1$, is what we call the {\it misspecification variance}:
the inflation in variance due to unobserved features, when we take
$\mathbb{E}(y_0|x_0)$ to be the target of estimation.  The last term, $\sigma^2
\gamma/(1-\gamma)$ for $\gamma<1$ and $\sigma^2/(\gamma-1)$ for
$\gamma>1$, is the variance itself.

\subsection{Polynomial approximation bias}
\label{sec:poly_bias}

Since adding features should generally improve our approximation capacity, it is
reasonable to model $\kappa=\kappa(\gamma)$ as an increasing function of
$\gamma$. To get an idea of the possible shapes taken by the asymptotic risk
curve from Theorem \ref{thm:risk_mis_iso}, we consider the example of a  \emph{polynomial decay} for
the approximation bias,
\begin{equation}
\label{eq:poly_decay}
1-\kappa(\gamma) = (1+\gamma)^{-a},
\end{equation}
for some $a > 0$.  In this case, the limiting risk in the isotropic setting,
from Theorem \ref{thm:risk_mis_iso}, becomes
\begin{equation}
\label{eq:risk_mis_iso}
R_a(\gamma) = \begin{cases}
r^2(1+\gamma)^{-a} + (r^2(1+\gamma)^{-a} + \sigma^2)
\frac{\gamma}{1-\gamma} & \text{for $\gamma < 1$}, \\
r^2(1+\gamma)^{-a}  + r^2 \big(1-(1+\gamma)^{-a}\big)
\big(1-\frac{1}{\gamma}\big) + (r^2(1+\gamma)^{-a}  + \sigma^2)
\frac{1}{\gamma-1} & \text{for $\gamma > 1$}.
\end{cases}
\end{equation}
We next summarize some interesting features of these risk curves, and Figures \ref{fig:risk_mis_lo} and
\ref{fig:risk_mis_hi} give accompanying plots for $\mathrm{SNR}=1$ and $5$,
respectively.  Recall that the null risk is $r^2$, which comes from predicting
with the null estimator \smash{$\tilde\beta=0$}.

\begin{figure}[p]
\vspace{-40pt}
\centering
\includegraphics[width=0.725\textwidth]{fig/risk_ab_lo_snr.pdf}
\caption{\footnotesize Asymptotic risk curves in \eqref{eq:risk_mis_iso} for the
  min-norm least squares estimator in the misspecified case, when the
  approximation bias has polynomial decay as in \eqref{eq:poly_decay}, as $a$
  varies from 0.5 to 5.  Here $r^2=1$ and $\sigma^2=1$, so $\mathrm{SNR}=1$. The null
  risk $r^2=5$ is marked as a dotted black line. The points denote finite-sample
  risks, with $n=200$, $p=[\gamma n]$, across various values of $\gamma$,
  computed from features $X$ having i.i.d.\ $N(0,1)$ entries.}
\label{fig:risk_mis_lo}

\bigskip
\includegraphics[width=0.725\textwidth]{fig/risk_ab_hi_snr.pdf}
\caption{\footnotesize Asymptotic risk curves in \eqref{eq:risk_mis_iso} for the
  min-norm least squares estimator in the misspecified case, when the
  approximation bias has polynomial decay as in \eqref{eq:poly_decay}, as $a$
  varies from 0.5 to 5.  Here $r^2=5$ and $\sigma^2=1$, so $\mathrm{SNR}=5$. The null
  risk $r^2=5$ is marked as a  dotted black line. The points are again
  finite-sample risks, with $n=200$, $p=[\gamma n]$, across various values of
  $\gamma$.}
\label{fig:risk_mis_hi}
\end{figure}

\begin{enumerate}
\item On $(0,1)$, the least squares risk $R_a(\gamma)$ can only be better than
  the null risk if \smash{$a > 1+\frac{1}{\mathrm{SNR}}$}. Further, in this case, we
  have $R_a(\gamma) < r^2$ if and only if $\gamma < \gamma_0$, where $\gamma_0$
  is the unique zero of the function
  $$
  (1+x)^{-a} + \Big(1+\frac{1}{\mathrm{SNR}}\Big) x - 1
  $$
  that lies in \smash{$(0,\frac{\mathrm{SNR}}{\mathrm{SNR}+1})$}.  Finally, on
  \smash{$(\frac{\mathrm{SNR}}{\mathrm{SNR}+1},1)$}, the least squares risk $R_a(\gamma)$ is
  always worse than the null risk, regardless of $a>0$, and it is monotonically
  increasing.

\item On $(1,\infty)$, when $\mathrm{SNR} \leq 1$, the min-norm least squares risk
  $R_a(\gamma)$ is always worse than the null risk.  Moreover, it is
  monotonically decreasing, and approaches the null risk (from above) as $\gamma
  \to \infty$.

\item On $(1,\infty)$, when $\mathrm{SNR} > 1$, the min-norm least squares risk
  $R_a(\gamma)$ can be better than the null risk for any $a>0$, and in
  particular we have $R_a(\gamma) < r^2$ if and only if $\gamma <  \gamma_0$,
  where $\gamma_0$ is the unique zero of the function
  $$
  (1+x)^{-a} (2x-1) + 1 - \Big(1-\frac{1}{\mathrm{SNR}}\Big) x
  $$
  lying in \smash{$(\frac{\mathrm{SNR}}{\mathrm{SNR}-1},\infty)$}.  Indeed, on
  \smash{$(1,\frac{\mathrm{SNR}}{\mathrm{SNR}-1})$}, the min-norm least squares risk
  $R_a(\gamma)$ is always worse than the null risk (regardless of $a>0$), and it
  is monotonically decreasing.

\item When $\mathrm{SNR} > 1$, for small enough $a>0$, the global minimum of the
  min-norm least squares risk $R_a(\gamma)$ occurs after $\gamma=1$.  A
  sufficient but not necessary condition is \smash{$a \leq 1 +
    \frac{1}{\mathrm{SNR}}$} (because, due to points 1 and 3 above, we see that in this
  case $R_a(\gamma)$ is always worse than null risk for $\gamma < 1$, but will
  be better than the null risk at some $\gamma > 1$).
\end{enumerate}

\subsection{A latent space model}
\label{sec:LatentSpace}

\begin{figure}[ht]
\vspace{-40pt}
\centering
\includegraphics[width=0.725\textwidth]{fig/lowrank_d20_s0_nvar.pdf}
\caption{\footnotesize Latent space model of Section \ref{sec:LatentSpace}: test error $R_X(\hat{\beta};\beta)$ of minimum norm regression as a function of the overparametrization ratio $\gamma$. Here
  $d=20$, $r=1$, $\sigma_{\xi}=0$ and $n$ varies across various curves. Symbols are averages over $100$
 realizations; continuous lines report the analytical prediction of Corollary \ref{coro:Latent}.}
\label{fig:risk_latent_space_n}
\end{figure}

\begin{figure}[ht]
\vspace{-40pt}
\centering
\includegraphics[width=0.725\textwidth]{fig/lowrank_d20_r1_svar.pdf}
\caption{\footnotesize Latent space model of Section \ref{sec:LatentSpace}: test error $R_X(\hat{\beta};\beta)$ of minimum norm regression as a function of the overparametrization ratio $\gamma$. Here
  $d=20$, $r=1$, $n=400$, and the noise variance $\sigma_{\xi}$
  varies across different curves. Symbols are averages over $100$
 realizations; continuous lines report the analytical prediction of Corollary \ref{coro:Latent}.}
\label{fig: risk_latent_space_sigma}
\end{figure}

We next consider an example in which $\beta$ is aligned with the top eigenvectors of $\Sigma$.
To motivate this example, assume that the responses $y_i$ are linear in the latent features vectors
$z_i\in{\mathbb R}^d$. We do not observe this latent vector, but rather observe $p\ge d$ covariates
$x_i:=(x_{i1},\dots,x_{ip})$ that also linear in the latent vector $z_i$:
\begin{align}
  %
  y_i = \theta^Tz_i+\xi_i\, ,\;\;\;\;\;\; x_{ij} = w_j^Tz_i+ u_{ij}\, .\label{eq:LS_First}

\end{align}
Here $(\xi_i)_{i\le n}$, $(u_{ij})_{i\le n,j\le p}$ are noise variables that are mutually independent,
and independent of $z_i$, with $\xi_i\sim{ N}(0,\sigma_{\xi}^2)$, $u_{ij}\sim{ N}(0,1)$.
The features matrix takes the form $X = ZW^T+U$ and therefore, for $p>n$, min-norm regression amounts to
\begin{align}
  \hat{\beta} = \arg\min \Big\{ \|b\|_2 :  ZW^Tb +Ub = 0\Big\}\, .\label{eq:MinNormLatent}
  \end{align}

Apart from its intrinsic interest, this latent-space model is directly connected to nonlinear random features models, as
the ones studied in Section \ref{sec:nonlinear}. Indeed, in  non-linear random features model we have
$x_{ij} = \varphi(w_j^Tz_i)$. We can decompose this as $x_{ij} = a_0+a_1w_j^Tz_i+ \tilde{\varphi}(w_j^Tz_i)$,
where $\tilde{\varphi}$ is such that $\tilde\varphi(w_j^Tz_i)$ has zero mean and is uncorrelated with $w_j^Tz_i$
when $z_j\sim{ N}(0,{{I}}_d)$. Equation \eqref{eq:LS_First} then corresponds to replacing the uncorrelated random variable $\tilde\varphi(w_j^Tz_i)$
by the independent Gaussians $u_{ij}$.

We assume the latent vector to be isotropic $z_i\sim{ N}(0,I_d)$.
It is then easy to see that the model is equivalent
to (a special case) model studied in Section \ref{sec:correlated}, namely $x_i\sim{ N}(0,\Sigma)$, $y_i = \beta^Tx_i+{\varepsilon}_i$.
Indeed (considering the case of Gaussian ${\varepsilon}_i$), in both cases $(y_i, x_i)\in{\mathbb R}^{p+1}$ is a centered Gaussian vector.
By matching their covariances we obtain (defining $\psi = d/p$):
\begin{align}
  &\Sigma = I_p+WW^T\, ,\;\;\;\;\; \beta =  W(I+W^TW)^{-1}\theta\, ,\\
  &\epsilon_i  \sim N(0,\sigma^2)\, ,\;\;\;\;\;\; \sigma^2 = \sigma_{\xi}^2+\theta^T(I+W^TW)^{-1}\theta\, .
\end{align}
                                %
Here $W\in{\mathbb R}^{p\times d}$ is the matrix with rows $(w_i)_{i\le p}$. In what follows $r_{\theta}^2:=\|\theta\|_2^2$.
As anticipated, the coefficients vector is aligned with the top eigenspace of $\Sigma$ (the span of the columns of
$W$).

Notice that the signal-to-noise ratio in the features $x_{ij}$ is equal to $\|w_i\|_2^2$. To be definite, we'll fix
$\|w_i\|_2=1$, which implies in particular $\|W\|_F^2=p$. In order to simplify our calculations,
we assume all the non-zero singular values of $W$ to be equal, whence $W^TW = (p/d)I_d$. It is not hard to check that this
model satisfies the assumptions of  Theorem \ref{thm:risk_gen}, with
\begin{align}
  H(s) & = (1-\psi){\boldsymbol 1}(s\ge 1)+\psi{\boldsymbol 1}(s\ge 1+\psi^{-1})\, ,\\
  G(s) &={\boldsymbol 1}(s\ge 1+\psi^{-1})\, ,\;\;\;\; \|\beta\|^2_2 = \frac{\psi r^2_{\theta}}{(1+\psi)^2}
\end{align}
Using Theorem \ref{thm:risk_gen}, we get the following explicit expressions.
\begin{corollary}\label{coro:Latent}
  Consider the latent space model described above, and assume $d/p\to \psi\in (1,\infty)$,
  $p/n\to \gamma\in (0,\infty)$. Then, almost surely
   %
    \begin{align}
      %
      R_X(\hat{\beta};\beta) &\to  \mathscrsfs{B}_{\mbox{\tiny \rm lat}}(\psi,\gamma) + \mathscrsfs{V}_{\mbox{\tiny \rm lat}}(\psi,\gamma) \, ,\\
      \mathscrsfs{B}_{\mbox{\tiny \rm lat}}(\psi,\gamma)  & :=\left\{1+\gamma c_0
                                   \frac{  \mathscrsfs{E}_1(\psi,\gamma) }{  \mathscrsfs{E}_2(\psi,\gamma) }\right\}\cdot\frac{r^2_{\theta}}{(1+\psi)
                                   (1+c_0\gamma(1+\psi^{-1}))^2}\, ,\\
     \mathscrsfs{V}_{\mbox{\tiny \rm lat}}(\psi,\gamma)  & :=\sigma^2\gamma c_0
                                   \frac{  \mathscrsfs{E}_1(\psi,\gamma) }{  \mathscrsfs{E}_2(\psi,\gamma) }\, ,\\
     \mathscrsfs{E}_1(\psi,\gamma) & := \frac{1-\psi}{(1+c_0\gamma)^2}+ \frac{\psi(1+\psi^{-1})^2}{(1+c_0(1+\psi^{-1})\gamma)^2}\, ,\\  \mathscrsfs{E}_2(\psi,\gamma) &:= \frac{1-\psi}{(1+c_0\gamma)^2}+ \frac{1+\psi}{(1+c_0(1+\psi^{-1})\gamma)^2}\, .
      %
    \end{align}
    %
    where $\sigma^2=\sigma_{\xi}^2+\psi r_{\theta}^2/(1+\psi)$, and $c_0 =c_0(\psi,\gamma)\ge 0$ is the unique non-negative solution of the following second order equation
    %
    \begin{align}
      1-\frac{1}{\gamma} = \frac{1-\psi}{1+c_0\gamma}+ \frac{\psi}{1+c_0(1+\psi^{-1})\gamma}\, .
      \end{align}
\end{corollary}

Figures \ref{fig:risk_latent_space_n} and \ref{fig: risk_latent_space_sigma} illustrate this corollary
by comparing analytical predictions to numerical simulations. We observe that the prediction risk is monotone decreasing in the overparametrization ratio for $\gamma>1$, and reaches its global minimum asymptotically as $\gamma\to\infty$ (after $p,n,d\to\infty$). To understand why this happens,
notice  that each feature vector $x_{i}$ can be viewed as a noisy measurement of the latent
covariates $z_i$. If the noise $u_{ij}$ was absent, then performing min-norm regression with respect to
$(x_i)_{i\le n}$ amounts to minimize would be equivalent to min-norm regression with respect to $(z_i)_{i\le n}$.
To see this, consider again Eq.~\eqref{eq:MinNormLatent}. If we drop
the noise $U$, we are minimizing $\| b\|_2$
subject to $Z(W^Tb) = 0$, and the regression function is $\hat{f}(z) = x^T\hat{\beta} =  z^T(W^T\hat{\beta})$.
Since $W$ is orthogonal, this is equivalent to computing $\hat{\theta} = \arg\min\{\|t\|_2:$ subject to $Zt = 0\}$.

In presence of noise $u_{ij}$, the latent features cannot be estimated exactly. However as $p$ gets larger,
the noise is effectively `averaged out' and we approach the idealized situation in which the $z_i$'s are observed.

\section{Ridge regularization}
\label{sec:ridge}

We generalize the formulas of Section \ref{sec:correlated} to non-vanishing ridge regularization.
We work under the same assumptions of that section.
In particular, recall that $\widehat{H}_n(s) = p^{-1}\sum_{i=1}^p 1_{\{s\ge s_i\}}$ is the empirical distribution of eigenvalue of
$\Sigma$, and $\widehat{G}_n(s)  =\sum_{i=1}^p \<\beta,v_i\>^2 1_{\{s\ge s_i\}}/\|\beta\|^2$ the same empirical distribution, reweighted
by the projection of $\beta$ onto the eigenvector $v_i$ of the covariance $\Sigma$.
(Recall the eigenvalue decomposition $\Sigma=\sum_{i=1}^ps_iv_iv_i^T$.)
\begin{definition}[Predicted bias and variance: ridge regression]\label{def:LimitBiasVar-lambda}
  Let $\widehat{H}_n$ be the empirical distribution of eigenvalues of $\Sigma$,
  and $\widehat{G}_n$ the reweighted distribution as per Eq.~\eqref{eq:HGdef}. For $\gamma\in{\mathbb R}_{>0}$, and $z\in {\mathbb C}_+$
  (the set of complex numbers with $\Im(z)>0$),
  define $m_n(z) = m(z;\widehat{H}_n,\gamma)$ as the unique solution of
  %
  \begin{align}
   m_n(z) = \int \frac{1}{s[1-\gamma-\gamma zm_n(z)] -z}\, d\widehat{H}_n(s)\, .\label{eq:m0def}
  \end{align}
  %
  Further define $m_{n,1}(z) = m_{n,1}(z;\widehat{H}_n,\gamma)$ via
  \begin{align}
    m_{n,1}(z) := \frac{\int \frac{s^2[1-\gamma-\gamma zm_n(z)]}{[s[1-\gamma-\gamma zm_n(z)] -z]^2}\, d\widehat{H}_n(s)}
{1-\gamma \int \frac{zs}{[s[1-\gamma-\gamma zm_n(z)] -z]^2}\, d\widehat{H}_n(s)}\, .\label{eq:m1def}
  \end{align}
  %
  These definitions are extended analytically to $\Im(z) = 0$ whenever possible.
  We then define the predicted bias and variance by
  %
  \begin{align}
    \mathscrsfs{B}(\lambda;\widehat{H}_n,\widehat{G}_n,\gamma) &:=\lambda^2\|\beta\|^2 \big(1+\gamma m_{n,1}(-\lambda)\big) \int \frac{s }{[\lambda +(1-\gamma+\gamma\lambda m_n(-\lambda))s]^2} d\widehat{G}_n(s)\, ,\label{eq:BiasRidge}\\
    \mathscrsfs{V}(\lambda;\widehat{H}_n,\gamma) &:= \sigma^2\gamma  \int \frac{s^2(1-\gamma+\gamma\lambda^2 m'_n(-\lambda))}{[\lambda+s(1-\gamma+\gamma \lambda m_n(-\lambda))]^2} d\widehat{H}_n(s)\, . \label{eq:VarianceRidge}
  \end{align}
\end{definition}

 We next state our deterministic approximation of the risk.
\begin{theorem}\label{thm:risk_ridge}
  Let $M^{-1} \le p/n \le M$,  and Assumption \ref{ass:CovariatesAssumption} hold.
  Further assume $\lambda\vee \lambda_{\min}(\Sigma)>1/M$ and $n^{-2/3+1/M}<\lambda<M$.
  Let $\hat{\beta}_{\lambda}$ be the ridge estimator of Eq.~\eqref{eq:ridge}.

  Then for any constants $D>0$ (arbitrarily large) and ${\varepsilon}>0$ (arbitrarily small),
  there exist $C=C(M,D)$  such that, with probability at least $1-Cn^{-D}$ the following hold
\begin{align}
  &R_X(\hat{\beta} _{\lambda};\beta)  =B_X(\hat{\beta} _{\lambda};\beta)  +V_X(\hat{\beta} _{\lambda};\beta) \, ,\\
  &\big|B_X(\hat{\beta}_{\lambda};\beta)-\mathscrsfs{B}(\lambda;\widehat{H}_n,\widehat{G}_n,\gamma) \big|\le \frac{C\|\beta\|_2^2}{\lambda n^{(1-{\varepsilon})/2}}\, ,\\
  &\big|V_X(\hat{\beta}_{\lambda};\beta)-\mathscrsfs{V}(\lambda;\widehat{H}_n,\gamma) \big|\le \frac{C}{\lambda^2 n^{(1-{\varepsilon})/2}}\, ,
\end{align}
  where $\mathscrsfs{B}$ and $\mathscrsfs{V}$ are given in Definition \ref{def:LimitBiasVar-lambda}.
\end{theorem}
The proof of this theorem is deferred to Appendix \ref{sec:ProofGeneralFormula}.
 As for theorem \ref{thm:risk_gen}, similar results were proved in
 \cite{wu2020optimal,richards2020asymptotics}, subsequently to a first version of this manuscript that only focused on
 random $\beta$. The same comparison of Remark \ref{rmk:Comparison} applies here.

 In particular,  Theorem \ref{thm:risk_ridge} establishes  non-asymptotic deterministic approximations for the bias
  $B_X(\hat{\beta}_{\lambda};\beta)$ and variance $V_X(\hat{\beta};\beta)$. The error terms are uniform
  over the covariance matrix, and have nearly optimal dependence upon
  the sample size $n$. Indeed, a  central-limit theorem
  heuristics suggests indeed fluctuations of order $n^{-1/2}$.

  As for the case of min-norm regression, it is easy to extract asymptotic prediction under minimal assumptions.
  %
  \begin{theorem}\label{thm:risk_ridge_asymp}
      Consider the setting of Theorem \ref{thm:risk_ridge} but, instead of Assumption \ref{ass:CovariatesAssumption} $(a)$,
  assume that $(z_i)_{i\le p}$ are identically distributed and satisfy the conditions
  $\mathbb{E} z_i= 0$, $\mathbb{E}(z_i^2) = 1$, $\mathbb{E}(|z_i|^{4+\eta})\le C<\infty$ for some $\eta>0$.
  Further assume $p/n\to \gamma\in (0,\infty)$, $\widehat{H}_n\Rightarrow H$, $\widehat{G}_n\Rightarrow G$.
  Then, for any $\lambda>0$,
  almost surely $B_X(\hat{\beta}_{\lambda};\beta)/\|\beta\|^2_2\to \mathscrsfs{B}(\lambda;H,G,\gamma)$,
  $V_X(\hat{\beta}_{\lambda};\beta)\to \mathscrsfs{V}(\lambda;H,\gamma)$.
\end{theorem}

\subsection{Isotropic features}

As a special case, we can consider the simple isotropic model that was already studied in Section \ref{sec:isotropic}.
Very similar (though not
identical) results can be found in \citet{dicker2016ridge,dobriban2018high}.
\begin{corollary}
\label{coro:risk_ridge_iso}
Assume the conditions of Theorem \ref{thm:risk_iso} (well-specified model,
isotropic features). Then for ridge regression estimator in \eqref{eq:ridge}
as $n,p \to \infty$, such that $p/n \to \gamma \in
(0,\infty)$, it holds almost surely that
\begin{align}
R_X(\hat{\beta}_\lambda;\beta) \to r^2\lambda^2m'(-\lambda)+\sigma^2 \gamma \big( m(-\lambda) - \lambda m'(-\lambda) \big)\ . \label{eq:RiskIsoRidge}
\end{align}
Here $m(z)$ is given by Eq.~\eqref{eq:m0def} which in this case has the explicit solution
$m(z) = [1-\gamma-z-\sqrt{(1-\gamma-z)^2-4\gamma z}]/(2\gamma z)$.

Furthermore, the limiting ridge risk is
minimized at $\lambda^*=\sigma^2\gamma/r^2$,  in which case we have the simpler expression
$R_X(\hat{\beta}_\lambda;\beta) \to \sigma^2\gamma m(-\lambda^*)$.

Under the conditions of Theorem \ref{thm:risk_mis_iso} (misspecified model,
isotropic features), the limiting risk of ridge regression is as in the first
two displays, and the optimal limiting risk is as in the third, after we make the
substitutions in \eqref{eq:subs} and add $r^2(1-\kappa)$, to each expression.
\end{corollary}
It is easy to recover the formulas in Theorem \ref{thm:risk_iso} as a limiting case of Eq.~\eqref{eq:RiskIsoRidge},
by using the $z\to 0$ asymptotics $m(z) = (1-\gamma)^{-1}+O(z)$ for $\gamma<1$ and $m(z) = (1-\gamma)^{-1}+O(z)$ for $\gamma<1$ and
$m(z) = -(\gamma-1)/(\gamma z)+[(\gamma-1)\gamma]^{-1} +O(z)$ for $\gamma>1$.

\begin{figure}[p]
\vspace{-30pt}
\centering
\includegraphics[width=0.725\textwidth]{fig/risk_rg.pdf}
\caption{\footnotesize  Asymptotic risk curves for the min-norm least squares
  estimator in \eqref{eq:risk_iso} as solid lines, and optimally-tuned ridge
  regression (from Theorem \ref{thm:risk_ridge}) as dashed lines. Here $r^2$
  varies from 1 to 5, and $\sigma^2=1$.  The null risks are marked by the dotted
  lines.}
\label{fig:risk_rg}

\bigskip
\includegraphics[width=0.725\textwidth]{fig/risk_rg_ab.pdf}
\caption{\footnotesize Asymptotic risk curves for the min-norm least squares
  estimator in \eqref{eq:risk_mis_iso} as solid lines, and optimally-tuned ridge
  regression (from Theorem \ref{thm:risk_ridge}) as dashed lines, in the
  misspecified case, when the approximation bias has polynomial decay as in
  \eqref{eq:poly_decay}, with $a=2$.  Here $r^2$ varies from 1 to 5, and
  $\sigma^2=1$.  The null risks are marked by the dotted lines.}
\label{fig:risk_rg_ab}
\end{figure}

Figures \ref{fig:risk_rg} and \ref{fig:risk_rg_ab} compare the risk curves of
min-norm least squares to those from optimally-tuned ridge regression, in the
well-specified and misspecified settings, respectively.  There are two important
points to make.  The first is that optimally-tuned ridge regression is seen to
have strictly better asymptotic risk throughout, regardless of $r^2$, $\gamma$,
$\kappa$.  This should not be a surprise, as by definition optimal
tuning should yield better risk than min-norm least squares, which is the
special case given by $\lambda \to 0^+$.

The second point is that, in this example, the limiting risk of
optimally-tuned ridge regression appears to have a minimum around $\gamma=1$,
and this occurs closer and closer to $\gamma=1$ as $\mathrm{SNR}$ grows.  This
behavior is interesting, especially because it is  antipodal to that
of the min-norm least squares risk, and leads us to very different suggestions
for practical usage for feature generators: in settings where we apply
substantial $\ell_2$ regularization (say, using CV tuning to mimic
optimal tuning, which the next section shows to be asymptotically equivalent),
it seems we want the complexity of the feature space to put us as close to the
interpolation boundary ($\gamma=1$) as possible.

As we will see, the behavior is rather different in the latent space model.

\subsection{Latent space model}
\label{sec:LatentRidge}

\begin{figure}[p]
\vspace{-30pt}
\centering
\includegraphics[width=0.725\textwidth]{fig/lowrank_ridge_delta20.pdf}
\caption{\footnotesize Asymptotic risk as a function of the overparametrization ratio $\gamma = p/n$,
  for ridge regression in the latent space model of Section \ref{sec:LatentRidge}. Here
  $\delta=n/d=20$, $r_{\theta}=1$, $\sigma_{\xi}=0$, and each curve corresponds to a different value of the
  regularization $\lambda$. The dashed curve correspond to the min-norm interpolator
  (which coincides with the $\lambda\to 0$ limit of ridge regression).}
\label{fig:lowrank_ridge}

\bigskip
\includegraphics[width=0.725\textwidth]{fig/lowrank_ridge_delta20_lamb_s01.pdf}
\caption{\footnotesize Asymptotic risk as a function of the regularization parameter $\lambda$,
  for ridge regression in the latent space model of Section \ref{sec:LatentRidge}. Here
  $\delta=n/d=20$, $r_{\theta}=1$, $\sigma_{\xi}=0.1$, and each curve corresponds to a different value of the overparametrization ratio $\gamma=p/n$.}
\label{fig:lowrank_ridge_lamb}
\end{figure}

As a special application, we consider the latent space model of Section \ref{sec:LatentSpace}.
It is immediate to specialize Eqs.~\eqref{eq:BiasRidge} and \eqref{eq:VarianceRidge}
to this case. We omit giving giving explicit formulas for brevity, and
instead plot the resulting curves for the prediction risk (test error).

In Figure \ref{fig:lowrank_ridge} we plot the risk as a function of
the overparametrization ration $\gamma=p/n$ for several values of the regularization parameter $\lambda$
(included the ridgeless limit $\lambda\to 0$). The setting here is analogous to the one of Fig.~\ref{fig:risk_latent_space_n}.
We observe several interesting phenomena:
\begin{enumerate}
\item Independently of $\lambda$ in the probed range, the risk is minimized at large overparametrization $\gamma\gg 1$.
\item As expected, the divergence of the risk at the interpolation threshold $\gamma=1$ is smoothed out by regularization,
  and the risk becomes a monotone decreasing function of $\gamma$ when $\lambda$ is large enough. Crucially, the optimal amount of regularization
  (corresponding to the lower envelope of these curves) results in a monotonically decreasing risk.
\item At large overparametrization, the optimal value of the regularization parameter is $\lambda\to 0$.
\end{enumerate}
We confirm the last finding in Fig.~\ref{fig:lowrank_ridge_lamb} which plots the risk as a function of $\lambda$:
the optimal regularization is $\lambda\to 0$. Notice that this is the case despite the fact that the observations are noisy,
namely $\sigma_{\xi}>0$ strictly.

The optimality of $\lambda\to 0$ has a known intuitive explanation that is worth recalling here.
Recall that ridge predictor at point $x_0$ takes the form
\begin{align}
  \hat{f}_{\lambda}(x_0) = \<x_0,\hat{\beta}_{\lambda}\> = K(x_0,X)\big(K(X,X)+\lambda I\big)^{-1}y\, ,
\end{align}
where we introduced the kernel matrix $K(X,X) = XX^T/n$, and the vector $K(x_0,X) = x_0^TX/n$.
In the present case $X$ can ve viewed as a noisy version of $\overline{X}=ZW^T$, namely
Consider the case in which the covariates $X$ contain noise, as is our case,
$X=\overline{X}+\eta U$ where $u_{ij}\sim N(0,1)$. (While we are considering $\eta=1$, it is instructive to
regard the noise standard deviation as a parameter.)
Then, we might expect
$K(X,X) \approx K(\overline{X},\overline{X})+\eta^2UU^T\approx K(\overline{X},\overline{X})+\eta^2 I_n$. If this approximations hold, the noise acts as an
extra ridge term, which can be sufficient to regularize the problem.

To the best of our knowledge, this argument was first presented by \citet{webb1994functional}
and, more explicitly, by \citet{bishop1995training}. Recently, \citet{kobak2020optimal} elucidated
its role in linear regression, establishing several of its consequences. In a parallel line of work,
a closely related idea has recently emerged  in the analysis of kernel methods in high dimension
\cite{el2010spectrum,liang2018just}.

\section{Cross-validation}
\label{sec:cv}

We analyze the effect of using cross-validation to choose the tuning parameter
in ridge regression.  In short, we find that choosing the ridge tuning parameter
to minimize the leave-one-out cross-validation error leads to the same
asymptotic risk as the optimally-tuned ridge estimator.  The next subsection
gives the details; the following subsection presents a new ``shortcut formula''
for leave-one-out cross-validation in the overparametrized regime, for min-norm
least squares, akin to the well-known formula for underparametrized least
squares and ridge regression.    We refer to   \cite{Craven:79,golub1979gcv} for background on
CV and GCV, and to \cite{arlot2010survey} for a more recent review.

\subsection{Limiting behavior of CV tuning}

Given the ridge regression solution \smash{$\hat{\beta}_\lambda$} in
\eqref{eq:ridge}, trained on $(x_i,y_i)$, $i=1,\ldots,n$, denote by
\smash{$\hat{f}_\lambda$} the corresponding ridge predictor, defined as
\smash{$\hat{f}_\lambda(x) = x^T \hat{\beta}_\lambda$} for $x \in \mathbb{R}^p$.  Additionally,
for each $i=1,\ldots,n$, denote by \smash{$\hat{f}^{-i}_\lambda$} the ridge
predictor trained on all but $i$th data point $(x_i,y_i)$.\footnote{To be
  precise, this is \smash{$\hat{f}^{-i}(x) = x^T (X_{-i}^T X_{-i} + n\lambda I)^{-1}
    X_{-i}^T y_{-i}$}, where $X_{-i}$ denotes $X$ with the $i$th row removed,
  and $y_{-i}$ denotes $y$ with the $i$th component removed. Arguably, it may
  seem more natural to replace the factor of $n$ here by a factor of $n-1$; we
  leave the factor of $n$ as is because it simplifies the presentation in what
  follows, but we remark that the same asymptotic results would hold with $n-1$
  in place of $n$.} Recall that the {\it leave-one-out cross-validation}
(leave-one-out CV, or simply CV) error of the ridge solution at a tuning
parameter value $\lambda$ is
\begin{equation}
\label{eq:cv}
\mathrm{CV}_n(\lambda) =
\frac{1}{n} \sum_{i=1}^n \big( y_i - \hat{f}^{-i}_\lambda(x_i) \big)^2.
\end{equation}
We typically view this as an estimate of the out-of-sample prediction
error \smash{$\mathbb{E}(y_0 - x_0^T \hat{\beta}_\lambda)^2$}, where the expectation is taken
over everything that is random: the training data $(x_i,y_i)$, $i=1,\ldots,n$
used to fit \smash{$\hat{\beta}_\lambda$}, as well as the independent test point
$(x_0,y_0)$. Note also that, when we observe training data from the model
\eqref{eq:data_x}, \eqref{eq:data_y}, and when $(x_0,y_0)$ is drawn
independently according to the same process, we have the relationship
$$
\mathbb{E}(y_0 - x_0^T \hat{\beta}_\lambda)^2
= \sigma^2 + \mathbb{E}(x_0^T \beta - x_0^T \hat{\beta}_\lambda)^2
= \sigma^2 + \mathbb{E}[R_X(\hat{\beta}_\lambda; \beta)],
$$
where \smash{$R_X(\hat{\beta}_\lambda; \beta) ] = \mathbb{E}[(x_0^T \beta - x_0^T
  \hat{\beta}_\lambda)^2 \,|\, X]$} is the conditional prediction risk, which has
been our focus throughout.

Recomputing the leave-one-out predictors \smash{$\hat{f}^{-i}_\lambda$},
$i=1,\ldots,n$ can be burdensome, especially for large $n$.  Importantly, there
is a well-known ``shortcut formula'' that allows us to express the leave-one-out
CV error \eqref{eq:cv} as a weighted average of the training errors,
\begin{equation}
\label{eq:cv_shortcut}
\mathrm{CV}_n(\lambda) =
\frac{1}{n} \sum_{i=1}^n \bigg( \frac{y_i - \hat{f}_\lambda(x_i)}
{1-(S_\lambda)_{ii}} \bigg)^2,
\end{equation}
where $S_\lambda=X(X^T X + n\lambda)^{-1} X^T$ is the ridge smoother matrix.
This identity is an immediate consequence of the Sherman-Morrison-Woodbury formula.
In the next subsection, we give an extension to
the case $\lambda=0$ and $\mathrm{rank}(X)=n$, i.e., to min-norm least squares.

The next result shows that, for isotropic features, the CV error of a ridge
estimator converges almost surely to its prediction error.  The focus on
isotropic features is only for simplicity: a more general analysis
is possible but is not pursued here. The proof, given in Appendix
\ref{app:risk_cv}, relies on the shortcut formula \eqref{eq:cv_shortcut}.  In
the proof, we actually first analyze generalized cross-validation (GCV), which
turns out to be somewhat of an easier calculation (see the proof for details on
the precise form of GCV), and then relate leave-one-out CV to GCV.

\begin{theorem}
\label{thm:risk_cv}
Assume the a isotropic prior, namely $\mathbb{E}(\beta)=0$, $Cov(\beta)=r^2{{I}}_p/p$,
and the data model \eqref{eq:data_x},
\eqref{eq:data_y}.  Assume that $x \sim P_x$ has i.i.d.\ entries with zero mean,
unit variance, and a finite moment of order $4+\eta$, for some $\eta>0$. Then
for the CV error \eqref{eq:cv} of the ridge estimator in \eqref{eq:ridge} with
tuning parameter $\lambda > 0$, as $n,p \to \infty$, with $p/n \to \gamma \in
(0,\infty)$, it holds almost surely that
$$
\mathrm{CV}_n(\lambda) - \sigma^2 \to \sigma^2 \gamma \big( m(-\lambda) -
\lambda (1 - \alpha \lambda) m'(-\lambda) \big),
$$
where \smash{$m(z)$} denotes the Stieltjes transform of the
Marchenko-Pastur law $F_\gamma$ (as in Corollary \ref{coro:risk_ridge_iso}), and $\alpha = r^2/(\sigma^2 \gamma)$.
Observe that the right-hand side is the asymptotic risk of ridge regression from
Theorem \ref{thm:risk_ridge}.  Moreover, the above convergence is uniform
over compacts intervals excluding zero.  Thus if $\lambda_1,\lambda_2$ are
constants with $0 < \lambda_1 \leq \lambda^* \leq \lambda_2 < \infty$, where
$\lambda^*=1/\alpha$ is the asymptotically optimal ridge tuning parameter value,
and we define \smash{$\lambda_n = \arg\min_{\lambda \in [\lambda_1,\lambda_2]}
\mathrm{CV}_n(\lambda)$}, then the expected risk of the CV-tuned ridge estimator $R_X(\hat{\beta}):=\mathbb{E}_\beta R_X(\hat{\beta};\beta)$
\smash{$\hat{\beta}_{\lambda_n}$} satisfies, almost surely,
$$
R_X(\hat{\beta}_{\lambda_n}) \to \sigma^2 \gamma m(-1/\alpha),
$$
with the right-hand side above being the asymptotic risk of optimally-tuned
ridge regression. Further, the exact same set of results holds for GCV.
\end{theorem}

Similar results were obtained  for various linear smoothers in
\citet{li1986asymptotic,li1987asymptotic},
for the lasso in the high-dimensional (proportional) asymptotics in \citet{miolane2018distribution}, and for
general smooth penalized estimators in \citet{xu2019consistent}. The latter
paper covers ridge regression as a special case, and gives more precise results
(convergence rates), but assumes more restrictive conditions.

The key implication of Theorem \ref{thm:risk_cv}, in the context of the current
paper and its central focus, is that the CV-tuned or GCV-tuned ridge
estimator has the same asymptotic performance as the optimally-tuned
ridge estimator. In other words, the ridge curves in Figures
\ref{fig:summary}, \ref{fig:risk_rg}, and \ref{fig:risk_rg_ab} can be
alternatively viewed as the asymptotic risk of ridge under CV tuning.

\subsection{Shortcut formula for ridgeless CV}

We extend the leave-one-out CV shortcut formula \eqref{eq:cv_shortcut} to work
when $p>n$ and $\lambda=0+$, i.e., for min-norm least squares.  In this case,
both the numerator and denominator are zero in each summand of
\eqref{eq:cv_shortcut}. To circumvent this, we can use the so-called ``kernel
trick'' to rewrite the ridge regression solution \eqref{eq:ridge} with
$\lambda>0$ as
\begin{equation}
\label{eq:ridge_kernel}
\hat{\beta}_\lambda = X^T (XX^T + n\lambda I)^{-1} y.
\end{equation}
Using this expression, the shortcut formula for leave-one-out CV in \eqref{eq:cv_shortcut} can be
rewritten as
$$
\mathrm{CV}_n(\lambda) = \frac{1}{n} \sum_{i=1}^n
\frac{[(XX^T + n\lambda I)^{-1} y]^2_i}
{[(XX^T + n\lambda I)^{-1}]^2_{ii}}\, .
$$
Taking $\lambda \to 0^+$ yields the shortcut formula for leave-one-out CV in
min-norm least squares (assuming without a loss of generality that
$\mathrm{rank}(X)=n$),
\begin{equation}
\label{eq:cv_shortcut_mn}
\mathrm{CV}_n(0) = \frac{1}{n} \sum_{i=1}^n
\frac{[(XX^T)^{-1} y]_i^2}{[(XX^T)^{-1}]_{ii}^2},
\end{equation}
In fact, the exact same arguments given here still apply when we replace $XX^T$
by a positive definite kernel matrix $K$ (i.e., $K_{ij}=k(x_i,x_j)$ for each
$i,j=1,\ldots,n$, where $k$ is a positive definite kernel function), in which
case \eqref{eq:cv_shortcut_mn} gives a shortcut formula for leave-one-out CV in
kernel ridgeless regression (the limit in kernel ridge regression as $\lambda
\to 0^+$).  We also remark that, when we include an unpenalized intercept in the
model, in either the linear or kernelized setting, the shortcut formula
\eqref{eq:cv_shortcut_mn} still applies with $XX^T$ or $K$ replaced by their
doubly-centered (row- and column-centered) versions, and the matrix inverses
replaced by pseudoinverses.

\section{Nonlinear model}
\label{sec:nonlinear}

Our analysis in the previous sections assumed $x_i = \Sigma^{1/2}z_i$, with $z_i$ a vector with
i.i.d. entries. As discussed in the introduction, we expect our results to have implications on certain neural
networks in the lazy training regime, via the universality hypothesis. Roughly speaking, if
covariates are obtained by applying the featurization map $z_i\mapsto x_i=\nabla f(z_i;\theta_0)$ to
data $z_i$, the asymptotic behavior of ridge regression is expected to be close to the one of an equivalent
Gaussian model, whose second-order statistics match the ones of $\nabla f(z_i;\theta_0)$.

In this section, we test universality on one simple example.
We observe data as in \eqref{eq:data_x}, \eqref{eq:data_y},
but now $x_i = \varphi(W z_i) \in \mathbb{R}^p$, where $z_i \in \mathbb{R}^d$ has i.i.d.\
entries from $N(0,1)$, for $i=1,\ldots,n$.  Also, $W \in \mathbb{R}^{p \times d}$ has
i.i.d.\ entries from $N(0,1/d)$. Finally, $\varphi:{\mathbb R}\to{\mathbb R}$ is an activation function acting
componentwise on vectors.

We focus on a simple case in which the second order statistics match the ones of the isotropic model.

\subsection{Limiting risk}

Notice that,  conditionally on $W$, the vectors  $x_i = \varphi(W z_i)$, $i\le n$ are independent. However, they do not have independent
coordinates: intuitively if $d$ is significantly smaller than $p$, $x_i$ contains much less randomness than a vector of $p$ independent
entries\footnote{For instance, if $\varphi(t) = at^2+b$,  we can reconstruct  $z_i$ from the first $2d$ coordinates of $x_i$, and
  therefore the remaining  $p-2d$ coordinates of $x_i$ are a function of the first $2d$.}.

Nevertheless, the next theorem shows that if $\varphi$ is purely nonlinear (in the sense that $\mathbb{E}\{\varphi(G)\} =\mathbb{E}\{G\varphi(G)\} = 0$),
then the feature matrix $X$ behaves ``as if'' it has
i.i.d.\ entries, in that the asymptotic bias and variance are exactly as in the
linear isotropic case, recall Eq.~\eqref{eq:risk_iso}.  In other words, this theorem provides a rigorous confirmation of the universality hypothesis stated in the introduction.
\begin{theorem}
  \label{cor:purely_nonlinear}
  Assume the model \eqref{eq:data_x}, \eqref{eq:data_y}, where each
$x_i = \varphi(W z_i) \in \mathbb{R}^p$, for $z_i \in \mathbb{R}^d$ having i.i.d.\
entries from $N(0,1)$, $W \in \mathbb{R}^{p \times d}$ having i.i.d.\ entries from
$N(0,1/d)$ (with $W$ independent of $z_i$), and is $\varphi$ an activation
function that acts componentwise.  Assume that $|\varphi(x)|\le
c_0(1+|x|)^{c_0}$ for a constant $c_0>0$. Also, for $G \sim N(0,1)$, assume that
the standardization conditions hold: $\mathbb{E}[\varphi(G)]=0$ and
$\mathbb{E}[\varphi(G)^2]=1$, $\mathbb{E}[G\varphi(G)]=0$.

Then for $\gamma>1$, the variance
satisfies, almost surely,
$$
\lim_{\lambda \to 0^+} \, \lim_{n,p,d \to \infty} \,
V_X(\hat{\beta}_\lambda; \beta) = \frac{\sigma^2}{\gamma-1},
$$
which is precisely as in the case of linear isotropic features, recall Theorem
\ref{thm:risk_iso}. Also, under a isotropic prior, namely $\mathbb{E}(\beta)=0$, $Cov(\beta)=r^2{{I}}_p/p$), the Bayes bias
$B_X(\hat{\beta}_{\lambda}) := \mathbb{E}_{\beta}B_X(\hat{\beta}_{\lambda};\beta)$
satisfies, almost surely
$$
\lim_{\lambda \to 0^+} \, \lim_{n,p,d \to \infty} \,
B_X(\hat{\beta}_\lambda) =
\begin{cases}
0 & \text{for $\gamma < 1$}, \\
r^2(1-1/\gamma) & \text{for $\gamma > 1$},
\end{cases}
$$
which is again as in the case of linear isotropic features, recall Theorem
 \ref{thm:risk_iso}.
\end{theorem}
The proof of Theorem \ref{cor:purely_nonlinear} is lengthy and will be sketched
shortly.

\begin{figure}[p]
\vspace{-40pt}
\centering
\includegraphics[width=0.725\textwidth]{fig/var_nl2.pdf}
\caption{\footnotesize Asymptotic variance curves for the min-norm least squares
  estimator in the nonlinear feature model (from Theorem
  \ref{cor:purely_nonlinear}), for the purely nonlinear activation
  $\varphi_{\mbox{\tiny \rm abs}}$.  Here $\sigma^2=1$, and the points are finite-sample
  risks, with $n=200$, $p=[\gamma n]$, over various values of $\gamma$, and
  varying input dimensions: $d=100$ in black, $d=200$ in red, $d=300$ in green,
  and $d=400$ in black.  As before, the features used for finite-sample
  calculations are $X=\varphi(ZW^T)$, where $Z$ has i.i.d.\ $N(0,1)$ entries and
  $W$ has i.i.d.\ $N(0,1/d)$ entries.}
\label{fig:purely_nonlinear}
\end{figure}

The origin of the conditions $\mathbb{E}\{\varphi(G)\}=\mathbb{E}\{G\varphi(G)\}=0$ can be easily explained
(throughout $G\sim N(0,1)$).
In summary, these conditions ensure that the first and second order statistics of $x_i=\varphi(Wz_i)$ approximately match
those of the isotropic model. To illustrate this point, let $i\neq j$, and
assume that the corresponding rows of $W$ (denoted by $w_i^T$ and $w_j^T$) have unit norm (this will only be approximately true,
but simplifies our explanation). We then have $\mathbb{E}\{\varphi(w_i^Tz_1)\} =\mathbb{E}\{\varphi(w_j^Tz_1)\} =\mathbb{E}\{\varphi(G)\}=0$. Further
\begin{align}
  \mathbb{E}\{x_{1,i}x_{1,j}|W\} = \mathbb{E}\{\varphi(w_i^Tz_1)\varphi(w_{j}^Tz_1)|W\} =  \mathbb{E}\{\varphi(G_1)\varphi(G_2)\}\, ,
\end{align}
where $G_1,G_2$ are jointly Gaussian with unit variance and covariance $w_i^Tw_j$. Denoting by $\varphi(x) = \sum_{k\ge 0}\lambda_k(\varphi)\,  h_k(x)$
the decomposition of $\varphi$ into orthonormal Hermite polynomials, we thus obtain
\begin{align}
  \mathbb{E}\{x_{1,i}x_{1,j}|W\} = \sum_{k=0}^{\infty}\lambda_k^2(\varphi)(w_i^Tw_j)^k\, ,
\end{align}
Since $\lambda_0(\varphi)=\mathbb{E}\{\varphi(G)\}=0$, $\lambda_1(\varphi) = \mathbb{E}\{G\varphi(G)\}=0$, we have
$\mathbb{E}\{x_{1,i}x_{1,j}|W\} = O((w_i^Tw_j)^2)=O(1/d)$. In other words,  the population covariance $\mathbb{E}\{x_{1}x_{1}^T|W\}$ has small entries out-of diagonal, and in fact
$\|\mathbb{E}\{x_{1}x_{1}^T|W\} -I_p\|_{\mbox{\tiny\rm op}}=o_P(1)$ \cite{el2010spectrum}.

Despite the nonlinear model $x_i = \varphi(Wz_i)$ matches the second-order
\emph{population} statistics of
the isotropic model, it is far from obvious that the resulting least norm regression risk is the same.
Indeed the coordinates of vector $x_i$ are highly dependent, as it can be seen by noting
that they are a function of only $d\ll p$ independent random variables.
Theorem \ref{cor:purely_nonlinear} confirms that ---despite dependence--- the risk is asymptotically the same, thus providing a concrete example of the general universality
phenomenon.

Figure \ref{fig:purely_nonlinear} compares the asymptotic risk curve from
Theorem  \ref{cor:purely_nonlinear} to that computed by simulation, using an
activation function $\varphi_{\mbox{\tiny \rm abs}}(t) = a(|t|-b)$, where
\smash{$a=\sqrt{\pi/(\pi-2)}$} and \smash{$b=\sqrt{2/\pi}$} are chosen to meet
the standardization conditions. This activation function is purely nonlinear,
i.e., it satisfies \smash{$\mathbb{E}[G\varphi_{\mbox{\tiny \rm abs}}(G)]=0$} for $G \sim N(0,1)$, by
symmetry. Again, the agreement between finite-sample and asymptotic risks is
excellent. Notice in particular that, as predicted by Theorem  \ref{cor:purely_nonlinear}, the risk
depends only on $p/n$ and not on $d/n$.

\subsection{Proof outline for Theorem \ref{cor:purely_nonlinear}}
\label{sec:nonlinear_proof}

We define $\gamma_n= p/n$ and $\psi_n= d/p$.  Recall that as $n,p,d \to
\infty$, we have $\gamma_n \to \gamma$ and $\psi_n \to \psi$.  To reduce
notational overhead, we will generally drop the subscripts from
$\gamma_n,\psi_n$, writing these simply as $\gamma,\psi$, since their meanings
should be clear from the context.
Let $N=p+n$ and define the symmetric matrix ${A}(s) \in {\mathbb R}^{N\times N}$,
for $s  \geq 0$, with the block structure:
\begin{equation}
{A}(s) = \begin{bmatrix}
s{{I}}_p & \frac{1}{\sqrt{n}} {X}^{{T}} \\
\frac{1}{\sqrt{n}} {X} & 0_n
\end{bmatrix},\label{eq:aden}
\end{equation}
where $I_p \in \mathbb{R}^{p\times p}$ and $0_n \in \mathbb{R}^{n\times n}$ are the identity and zero matrix, respectively. As we will see this matrix
allows to construct the traces of interest by taking suitable derivatives of its resolvent.

We introduce the following resolvents (as usual, these are defined for
$\Im(\xi)>0$ and by analytic continuation, whenever possible, for $\Im(\xi) =
0$):
\begin{align*}
m_{1,n}(\xi,s) &=
\mathbb{E}\Big\{\big({A}(s) - \xi{{I}}_{N}\big)^{-1}_{1,1}\Big\} =
\mathbb{E} M_{1,n}(\xi,s),\\
 M_{1,n}(\xi,s) &=
\frac{1}{p}{\sf Tr}_{[1,p]}\Big\{\big({A}(s) - \xi{{I}}_{N}\big)^{-1}\Big\},\\
m_{2,n}(\xi,s) &=
\mathbb{E}\Big\{\big({A}(s) -\xi{{I}}_{N}\big)^{-1}_{p+1,p+1}\Big\} =
\mathbb{E} M_{2,n}(\xi,s),\\
 M_{2,n}(\xi,s) &=
\frac{1}{n}{\sf Tr}_{[p+1,p+n]}\Big\{\big({A}(s) - \xi{{I}}_{N}\big)^{-1}\Big\}.
\end{align*}
Here and henceforth, we write $[i,j]=\{i+1,\ldots,i+j\}$ for integers $i,j$.
We also write \smash{$M^{-1}_{ij}= (M^{-1})_{ij}$} for a matrix $M$, and
\smash{${\sf Tr}_S(M) = \sum_{i\in S} M_{ii}$} for a subset $S$.  The equalities in
the first and third lines above follow by invariance of the distribution of
${A}(s)$ under permutations of $[1,p]$ and $[p+1,p+n]$. Whenever clear from
the context, we will omit the arguments from block matrix and resolvents, and
write ${A}={A}(s)$, $m_{1,n} =m_{1,n}(\xi,s)$, and $m_{2,n}
=m_{2,n}(\xi,s)$.

The next lemma characterizes the asymptotics of $m_{1,n}$, $m_{2,n}$.
\begin{lemma}\label{thm:ResolventKernel}
Assume the conditions of Theorem \ref{cor:purely_nonlinear}.  Consider $\Im(\xi)>0$
or $\Im(\xi)=0$, $\Re(\xi)<0$, with $s\ge t\ge 0$. Let $m_1$ and $m_2$ be the
unique solutions of the following quadratic equations:
\begin{align}
m_2= \big(-\xi-\gamma m_1\big)^{-1},\;\;\;\;\;\;\;\;
m_1=  \big(-\xi-s-m_2 \big)^{-1},\label{eq:M1}
\end{align}
subject to the condition of being analytic functions for $\Im(z)>0$, and
satisfying $|m_1(z,s)|, |m_2(z,s)| \le 1/\Im(z)$ for $\Im(z)>C$ (with $C$ a
sufficiently large constant).  Then, as $n,p,d \to \infty$, such that $p/n \to
\gamma$ and $d/p \to \psi$, we have almost surely (and in $L^1$),
\begin{align}
\lim_{n,p,d\to\infty} M_{1,n}(\xi,s) &= m_1(\xi,s),\label{eq:AS-1}\\
\lim_{n,p,d\to\infty} M_{2,n}(\xi,s) &= m_2(\xi,s).\label{eq:AS-2}
\end{align}
\end{lemma}

The proof of this lemma is given in Appendix \ref{app:ResolventKernel}.
As a corollary of the above, we obtain that the asymptotical empirical spectral distribution of the empirical covariance
\smash{$\hat\Sigma=X^T X/n$} matches the one for the independent entries model, and is hence given by the Marchenko-Pastur law
(a result already obtained in \citet{pennington2017nonlinear}).
We state this formally using the Stieltjes transform
\begin{equation}
R_n(z) = \frac{1}{p}{\sf Tr}\big((\hat\Sigma-z{{I}}_p)^{-1}\big),\label{eq:Sn}
\end{equation}

\begin{corollary}\label{coro:Stieltjes}
Assume the conditions of Theorem \ref{cor:purely_nonlinear}.  Consider
$\Im(z)>0$. As $n,p,d \to \infty$, with $p/n \to \gamma$ and $d/p \to \psi$,
we have (almost surely and in $L^1$) $R_{n}(\xi)\to r(\xi)$
where $r$ is  nonrandom and coincides with the Stieltjes transform of the Marchenko-Pastur law, namely
\begin{align}
  r(z) = \frac{1-\gamma-z -\sqrt{(1-\gamma-z)^2-4\gamma z}}{2\gamma z}\, \label{eq:coroS1}.
\end{align}
\end{corollary}

We refer to Appendix \ref{app:CoroStieltjes} for a proof of this corollary.
The next lemma connects the above resolvents computed in Lemma \ref{thm:ResolventKernel} to the variance of min-norm
least squares, hence finishing our proof outline.
\begin{lemma}\label{lemma:ResolventToVar}
Assume the conditions of Theorem \ref{cor:purely_nonlinear}.
Let $m_1$, $m_2$ be the asymptotic resolvents given in Lemma
\ref{thm:ResolventKernel}. Define
$$
m(\xi,s) = \gamma m_1(\xi,s) + m_2(\xi,s)\, .
$$
Then for $\gamma\neq 1$, $\partial_x m(\xi,x)\big|_{x=0}$ as a simple pole at $\xi=0$,
and hence admits a Taylor-Laurent expansion around
$\xi=0$, whose coefficients will be denoted by $D_{-1}$, $D_0$
\begin{align}
-\partial_x m(\xi,x)\big|_{x=0}
= \frac{D_{-1}}{\xi^2}+D_0+O(\xi^2).\label{eq:Laurent}
\end{align}
Here each coefficient is a function of $\gamma,\psi$:
$D_{-1} = D_{-1}(\gamma,\psi)$, $D_{0} = D_{0}(\gamma,\psi)$. Furthermore, for the ridge regression
estimator \smash{$\hat{\beta}_\lambda$} in \eqref{eq:ridge}, as $n,p,d \to \infty$,
such that $p/n \to \gamma \in (0,\infty)$, $d/p \to \psi \in (0,1)$, the
following ridgeless limit holds almost surely:
$$
\lim_{\lambda \to 0^+} \, \lim_{n,p,d\to\infty} \,
V_X(\hat{\beta}_\lambda;\beta) = D_0.
$$
\end{lemma}

The proof of this lemma can be found in Appendix
\ref{app:ResolventToVar}. Theorem \ref{cor:purely_nonlinear} follows by evaluating
the formula in Lemma \ref{lemma:ResolventToVar}, by using the
result of Lemma \ref{thm:ResolventKernel}. We refer to the appendices material for details.

\subsection*{Acknowledgements}

The authors are grateful to Brad Efron, Rob Tibshirani, and Larry Wasserman for
inspiring us to work on this in the first place.  RT sincerely thanks Edgar
Dobriban for many helpful conversations about the random matrix theory
literature, in particular, the literature trail leading up to Proposition
\ref{thm:risk_lo}, and the reference to \citet{rubio2011spectral}. We are also grateful to Dmitry Kobak
for bringing to our attention the \cite{kobak2020optimal} and clarifying the implications
of this work on our setting.

TH was partially supported by grants NSF DMS-1407548, NSF IIS-1837931, and NIH
5R01-EB001988-21.  AM was partially supported by NSF DMS-1613091, NSF
CCF-1714305, NSF IIS-1741162, and ONR N00014-18-1-2729.  RT was partially
supported by NSF DMS-1554123.

Statistics Theory (math.ST); Machine Learning (cs.LG); Machine Learning (stat.ML)