Random perturbation of low rank matrices: Improving classical bounds
Matrix perturbation inequalities, such as Weyl's theorem (concerning the singular values) and the Davis-Kahan theorem (concerning the singular vectors), play essential roles in quantitative science; in particular, these bounds have found application in data analysis as well as related areas of engineering and computer science. In many situations, the perturbation is assumed to be random, and the original matrix has certain structural properties (such as having low rank). We show that, in this scenario, classical perturbation results, such as Weyl and Davis-Kahan, can be improved significantly. We believe many of our new bounds are close to optimal and also discuss some applications.
\section{Introduction}

The singular value decomposition of a real $m \times n$ matrix $A$ is a factorization of the form $A = U \Sigma V^\mathrm{T}$, where $U$ is a $m \times m$ orthogonal matrix, $\Sigma$ is a $m \times n$ rectangular diagonal matrix with non-negative real numbers on the diagonal, and $V^\mathrm{T}$ is an $n \times n$ orthogonal matrix.  The diagonal entries of $\Sigma$ are known as the \emph{singular values} of $A$.  The $m$ columns of $U$ are the \emph{left-singular vectors} of $A$, while the $n$ columns of $V$ are the \emph{right-singular vectors} of $A$.  If $A$ is symmetric, the singular values are given by the absolute value of the eigenvalues, and the singular vectors are just the eigenvectors of $A$.  Here, and in the sequel, whenever we write \emph{singular vectors}, the reader is free to interpret this as left-singular vectors or right-singular vectors provided the same choice is made throughout the paper.  

Consider a real (deterministic) $m \times n$ matrix $A$ with singular values 
$$\sigma_1 \geq \sigma_2  \geq \cdots \geq \sigma_{\min\{m,n\}} \geq 0$$
and  corresponding  singular vectors  $v_1, v_2, \ldots, v_{\min\{m,n\}}.$ We will call $A$ the data matrix.  In general, the vector $v_i$ is not unique.  However, if $\sigma_i$ has multiplicity one, then $v_i$ is determined up to sign.  

An important problem in statistics and numerical analysis is to compute the first $k$ singular values and vectors of $A$.  In particular, the largest few singular values and corresponding singular vectors are typically the most important.  Among others, this problem lies at the heart of Principal Component Analysis (PCA), which has a very wide range of applications (for many examples, see \cite{KVbook, LR} and the references therein) and in the closely related low rank approximation procedure often used in theoretical computer science and combinatorics.   In application, $m,n$ are typically large and $k$ is small, often a fixed constant. 
 
A problem of fundamental importance in quantitative science (including pure  and applied mathematics, statistics, engineering, and computer science) is to estimate how a small perturbation to the data effects the spectrum. This problem has been discussed  in virtually every  text book on quantitative linear algebra and numerical analysis  (see, for instance,  \cite{BT, Hig1, Hig2, SS}). 

A basic model is as follows. Instead of $A$, one needs to work with  $A+E$, where $E$ represents the perturbation matrix.  Let 
$$ \sigma_1' \geq \cdots \geq \sigma_{\min\{m,n\}}' \geq 0 $$ 
denote the singular values of $A+E$ with corresponding singular vectors $v_1', \ldots, v_{\min\{m,n\}}'$.  We consider the following natural questions.  

\begin{question}
	Is $v_i'$ a good approximation of $v_i$?
\end{question}  
\begin{question} \label{quest:weyl}
	Is $\sigma_i'$ a good approximation of $\sigma_i$?  
\end{question}

These two questions are addressed by the Davis-Kahan-Wedin sine theorem and Weyl's inequality.  

Let us begin with the first question in the case when $i=1$.  A canonical  way (coming from the numerical analysis literature; see for instance  \cite{GVL})  
to measure the distance between two unit vectors $v$ and $v'$ is to look at $ \sin \angle(v,v')$,
where $\angle(v,v')$ is the angle between $v$ and $v'$ taken in $[0,\pi/2]$. 
It has been observed by numerical analysts (in the setting where $E$ is deterministic) for quite some time  that the key parameter to consider in the bound is the gap (or separation)
\begin{equation} \label{def:delta}
	\delta := \sigma_1 - \sigma_2,
\end{equation} 
between the first and second singular values of $A$.  
The first result in this direction is the famous Davis-Kahan sine $\theta$ theorem \cite{DK} for Hermitian matrices. 
The non-Hermitian version was proved later by Wedin \cite{W}.  

Throughout the paper, we use $\|M\|$ to denote the spectral norm of a matrix $M$.  That is, $\|M\|$ is the largest singular value of $M$. 

\begin{theorem}[Davis-Kahan, Wedin; sine theorem] \label{wedin}
$$ \sin \angle(v_1, v_1') \leq 2 \frac{\|E\|}{\delta}. $$
\end{theorem} 

\begin{remark}
Theorem \ref{wedin} is trivially true when $\delta \leq 2 \|E\|$ since sine is always bounded above by one.  In other words, even if the vector $v_1'$ is not uniquely determined, the bound is still true for any choice of $v_1'$.  On the other hand, when $\delta > 2 \|E\|$, the proof of Theorem \ref{wedin} reveals that the vector $v_1'$ is uniquely determined up to sign. 
\end{remark}

Theorem \ref{wedin} is a simple corollary of \cite[Theorem V.4.4]{SS} which is originally due to Wedin \cite{W}; we present a proof below for completeness.  

More generally, one can consider approximating the $i$-th singular vector $v_i$ or the space spanned by the first $i$ singular vectors $\mathrm{Span}\{v_1, \ldots, v_i \}$.  Naturally, in these cases, one must consider the gaps
$$ \delta_i := \sigma_i  - \sigma_{i+1}. $$

Question \ref{quest:weyl} is addressed by Weyl's inequality.  In particular, Weyl's perturbation theorem \cite{Wy} gives the following deterministic bound for the singular values (see \cite[Theorem IV.4.11]{SS} for a more general perturbation bound due to Mirsky \cite{M}).  

\begin{theorem} [Weyl's bound]  \label{theorem:Weyl} 
\begin{equation*}  
	\max_{1 \leq i\leq \min\{m,n\}} | \sigma_i -\sigma_i'| \le \|E \|. 
\end{equation*} 
\end{theorem} 

For more discussions concerning general perturbation bounds, we refer the reader to \cite{B, SS} and references therein.  We now pause for a moment to prove Theorem \ref{wedin}.  

\begin{proof}[Proof of Theorem \ref{wedin}]
If $\delta \leq 2 \|E\|$, the theorem is trivially true since sine is always bounded above by one.  Thus, assume $\delta > 2 \|E\|$.  By Theorem \ref{theorem:Weyl}, we have
$$ \sigma_1' - \sigma_2' \geq \delta - 2 \|E\| > 0, $$
and hence the singular vectors $v_1$ and $v_1'$ are uniquely determined up to sign.  By another application of Theorem \ref{theorem:Weyl}, we obtain
$$ \delta = \sigma_1 - \sigma_2 \leq \sigma_1 - \sigma_2' + \|E\|. $$
Rearranging the inequalities, we have
$$ \sigma_1 - \sigma_2' \geq \delta - \|E\| \geq \frac{1}{2} \delta > 0. $$
Therefore, by \cite[Theorem V.4.4]{SS}, we conclude that
$$ \sin \angle(v_1, v_1') \leq \frac{\|E\|}{ \sigma_1 - \sigma_2'} \leq 2 \frac{\|E\|}{ \delta}, $$
and the proof is complete.  
\end{proof}

Let us now focus on the matrices  $A$ and $E$. 
It has become common practice to assume that the perturbation matrix $E$ is random. 
Furthermore,  researchers have observed that 
data matrices are usually  not arbitrary.  They  often possess  certain  structural properties. Among these properties, one of the most frequently seen is having low rank (see, for instance, \cite{CP, CR, CRT, CS, TK} and references therein).  

The goal in this paper is to show that in this situation, one can significantly improve classical results like Theorems \ref{wedin} and \ref{theorem:Weyl}.
To give a quick example, let us assume  that  $A$ and $E$ are $n \times n$ matrices and that the entries of $E$ are independent and identically distributed (iid) random variables with zero mean, unit variance (which is just matter of normalization), and bounded fourth moment.   It is well known that in this case 
 $\|E\|= (2+o(1)) \sqrt n $ with high probability\footnote{We use asymptotic notation under the assumption that $n \to \infty$.  Here we use $o(1)$ to denote a term which tends to zero as $n$ tends to infinity.} \cite[Chapter 5]{BS}.  Thus, the above two theorems imply
\begin{corollary} \label{wedin-cor}
For any $\eta > 0$, with probability $1-o(1)$,
\begin{equation*}  |\sigma_1  -\sigma _1'| \le (2 + \eta) \sqrt n, \end{equation*} 
and
\begin{equation} \label{bound0}  \sin \angle(v_1, v_1') \leq 2 (2+\eta) \frac{\sqrt n }{\delta}. \end{equation}
\end{corollary}


Among others, this shows that  if one wants accuracy $\varepsilon$ in the first  singular vector computation, $A$ needs to satisfy
\begin{equation}  \label{bound1} \delta  \ge  2 (2 + \eta) \varepsilon^{-1} \sqrt n. \end{equation} 

We present the results of a numerical simulation for $A$ being a $n \times n$ matrix of rank 2 when $n=400$, $\delta=8$, and where $E$ is a random Bernoulli matrix (its entries are iid random variables that take values $\pm 1$ with probability $1/2$). 
The results, shown in Figure \ref{young}, turn out to be very different from what \eqref{bound1} predicts. 
It is easy to see that for the parameters $n=400$ and $\delta =8$,   Corollary \ref{wedin-cor} does not give  a useful bound
(since $\frac{\sqrt{n}}{\delta} = 2.5 >1$). However, Figure \ref{young} shows that,  with high probability, $\sin \angle(v_1,v_1') \leq 0.2$, which means 
$v_1'$ approximates $v_1$ with a relatively small error. 


\begin{figure}[!Ht]
 \begin{center}
   \includegraphics[width=12cm]{noise_n=400r=2.pdf}
   \includegraphics[width=12cm]{noise_n=1000r=2.pdf}
   \caption{The cumulative distribution functions of $\sin \angle(v_1, v_1')$ where $A$ is a $n \times n$ deterministic matrix with rank $2$ ($n=400$ for the figure on top and $n=1000$ for the one below) and the noise $E$ is a Bernoulli random matrix, evaluated from $400$ samples (top figure) and $300$ samples (bottom figure). In both figures, the largest singular value of $A$ is taken to be $200$.}
   \label{young}
 \end{center}
\end{figure}



\section{ The \emph{real dimension} and new results} 

Trying to explain the inefficiency of the Davis-Kahan-Wedin bound in the above example,  the second author was led to the following intuition.

\begin{quote}
If $A$ has rank $r$, all actions of $A$ focus on an $r$ dimensional subspace; intuitively then, $E$ must act like an  $r$ dimensional random matrix rather than an $n$ dimensional one. 
\end{quote}

  This means that  the {\it real dimension} of the problem  is $r$, not $n$. 
While it is clear that one cannot automatically  ignore the (rather wild) action of $E$ outside the range of $A$, this intuition, if true, would show that 
what really matters in \eqref{bound0} or \eqref{bound1} is $r$, the rank of $A$, rather than its size $n$. 
If this is indeed the case, one may hope to obtain a bound of the form 
\begin{equation} \label{bound2}  \sin \angle(v_1, v_1') \leq C \frac{\sqrt r}{\delta}, \end{equation}  for some constant $C$ 
(with some possible corrections). 
This is  much  better than \eqref{bound0} when $A$ has low rank  and  explains the phenomenon arising from Figure \ref{young}. 


In \cite{V}, the second author managed to prove 
\begin{equation*}  
	\sin ^2 \angle(v_1, v'_1)  \le  C \frac{ \sqrt{r \log n} }{\delta } 
\end{equation*}  
under certain conditions.  While the right-hand side is quite close to the  optimal form 
 in \eqref{bound2}, the main problem here is that in  the left-hand side  one needs to square the sine function. The bound for 
 $\sin \angle (v_i, v_i') $ with $i \ge 2$ was done by an inductive argument and was rather complicated.  Finally, the problem of estimating the singular values was not addressed at all in  \cite{V}.  
 
 
 
  In this paper, by using an entirely different (and simpler)  argument, we are going to remove the unwanted 
  squaring effect. This enables us to obtain a near optimal improvement of the Davis-Kahan-Wedin theorem. One can easily extend the proof to give 
 a  (again near optimal) bound on the angle between two subspaces spanned by the first few singular vectors of $A$ and their counterparts of $A+E$. 
 (This is the space one often actually cares about in PCA and low rank approximation procedures.) 
  Finally, as a co-product, we obtain an improved version of Weyl's bound, which also  supports our {\it real dimension}  intuition.  Our results hold under very mild assumptions on $A$ and $E$.
 As a matter of fact, in the strongest results, we will not even need the entries of $E$ to be independent. 
   
  
  
As an illustration, 
let us first state a result  in the case that $A$ is a $n \times n$ matrix and 
$E$ is a Bernoulli matrix (the entries are iid Bernoulli random variables, taking values $\pm 1$ with probability $1/2$).  
 
 
 \begin{theorem} \label{theorem:main1} Let $ E$ be a $n \times n$ Bernoulli random matrix and fix $\varepsilon > 0$.  Then there exists constants $C_0, \delta_0 > 0$ (depending only on $\varepsilon$) such that the following holds.  
 Let $A$ be a $n \times n$ matrix with rank $r$ satisfying  $\delta \geq \delta_0$ and $\sigma_1 \geq \max\{n,\sqrt{n}\delta\}$. Then, with probability at least $1-\varepsilon$,
\begin{equation*} \sin \angle (v_1, v_1') \leq C \frac{\sqrt{r}} {\delta}  . \end{equation*}  
 \end{theorem}
 
 Notice that the assumptions on $E$ are normalized (as we assume that the variance of the entries in $E$ is one). If the error entries have variance $\sigma^2$, then 
 we need to scale accordingly by replacing $A +E$ by $\frac{1}{\sigma} A + \frac{1}{\sigma} E $; thus, the assumptions become weaker as $\sigma$ decreases. 
 
 For the singular values, a good toy result  is the following

\begin{theorem} \label{thm:probweyl0}
 Let $E$ be an $n \times n$  Bernoulli random matrix and  fix $ \varepsilon > 0$. Then there exists a constant $C_0>0$ (depending only on $\varepsilon$) 
such that the following holds. 
 Let  $A$ be an $n \times n$ matrix with rank $r$ satisfying  $ \sigma_1 \geq n$. Then  with probability at least $1-\varepsilon$ 
\begin{equation*} \label{eq:probweylbnd0}
\sigma_1  -  C  \leq \sigma_1'  \leq  \sigma_1  +  C  \sqrt{r}.
\end{equation*}
\end{theorem}

It may be useful for the reader  to compare these new bounds  with the bounds obtained directly from the Davis-Kahan-Wedin  sine theorem and Weyl's inequality (see Corollary \ref{wedin-cor}).


Both theorems above are corollaries of much more general statements, which we describe in the next sections. 
 
 
 
 \section{Models of random noise} 

 

In the literature, there  are many models of random matrices.   We can capture almost all natural  models   by focusing on a common  property. 


\begin{definition} \label{def:concentration}
We say the  $m \times n$  random matrix $E$ is $(C_1,c_1, \gamma)$-concentrated if for all unit vectors $u \in \mathbb{R}^m, v \in \mathbb{R}^n$, and every $t>0$, 
\begin{equation} \label{eq:concentration}
	\mathbb{P}( |u^T E v| > t ) \leq C_1 \exp(-c_1 t^\gamma).
\end{equation}	
\end{definition}





The key parameter is $\gamma$. It is easy to verify the following fact, which asserts that the concentration property is closed under addition. 
\begin{fact}  \label{fact1}  If $E_1$ is $(C_1,c_1, \gamma)$-concentrated and $E_2$ is $(C_2,c_2, \gamma)$-concentrated, then 
$E_3 =E_1+E_2$ is $(C_3, c_3, \gamma)$-concentrated for some $C_3, c_3$ depending on $C_1,c_1, C_2, c_2$. \end{fact}

Furthermore, the concentration property guarantees a bound on $\| E \|$.  A standard net argument (see Lemma \ref{lemma:net}) shows


\begin{fact}  \label{fact2}  If $E$ is $(C_1,c_1, \gamma)$-concentrated  then there are constants $C',  c' >0$ such that $\mathbb{P} (\| E \| \ge C' n^{1/\gamma} )  \le C_1 \exp (-c'n) $. \end{fact}


For readers not familiar with random matrix theory, let us point out why the concentration property is expected to hold for any natural model. If $E$ is random and $v$ is fixed, then the vector $Ev$ must look random. It is well known that in a high dimensional space, a random vector, with very high probability, is nearly orthogonal to any fixed vector. Thus, one expects that very likely, the inner product of $u$ and $E v$ is small. Definition \ref{def:concentration} is a way to express this observation quantitatively. 


It turns out that all random matrices with independent entries satisfying a mild condition have the concentration property. 
This class covers virtually all examples one sees in practice. 
 In particular, Lemma \ref{lemma:bernoulli} shows that if $E$ is a $n \times n$ Bernoulli random 
 matrix, then $E$ is $\left(2, \frac{1}{2}, 2 \right)$-concentrated, and $\|E\| \leq 3 \sqrt{n}$ with high probability \cite{V,Vnorm}. 
 A convenient feature of  the definition is that   independence  between the entries is not a requirement.
  For instance, it is easy to show that a random orthogonal matrix 
 satisfies the concentration property.  We continue the discussion of the $(C_1,c_1,\gamma)$-concentration property (Definition \ref{def:concentration}) in Section \ref{sec:concentration}. 



\vskip2mm

Let us  state an extension of   Theorem \ref{theorem:main1}. 



\begin{theorem} \label{thm:main}
Assume that $E$ is $(C_1, c_1, \gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$, and suppose $A$ has rank $r$.  Then, for any $t>0$, 
$$ \sin \angle(v_1,v_1') \leq  4 \sqrt{2} \left( \frac{t r^{1/\gamma}} {\delta}  +     \frac{ \|E\|}{\sigma_1} +  \frac{ \|E\|^2}{\sigma_1 \delta} \right) $$
with probability at least 
$$ 1 - 54 C_1 \exp\left(-c_1\frac{\delta^\gamma}{8^\gamma} \right) - 2C_1 9^{2r} \exp \left( -c_1 r \frac{t^\gamma}{4^{\gamma}} \right). $$
\end{theorem}


\begin{remark} \label{remark:boundonE}
Using  Fact \ref{fact2}, one can replace $\| E\|$ on the right-hand side by $C' n^{1/\gamma }$, which yields that  
$$ \sin \angle(v_1,v_1') \leq 4 \sqrt{2}  \left( \frac{t r^{1/\gamma}} {\delta} +   \frac{ C' n^{1/\gamma} }{\sigma_1} +  \frac{ C'^2 n^{2/\gamma} }{\sigma_1 \delta} \right)$$
with probability at least 
$$ 1 - 54 C_1 \exp\left(-c_1\frac{\delta^\gamma}{8^\gamma} \right) - 2C_1 9^{2r} \exp \left( -c_1 r \frac{t^\gamma}{4^{\gamma}} \right) - C_1 \exp( -c' n). $$  
However, we prefer to state our theorems in the form of Theorem \ref{thm:main}, as the bound $C' n^{1/\gamma}$, in many cases, may not be optimal. 
\end{remark} 


\begin{remark}
Another useful  corollary of Theorem \ref{thm:main} is  the following. For  any constant $\varepsilon >0$ there are constants $C_0 =  C_0(\varepsilon, C_1, c_1, \gamma) >0$ and $\delta_0 = \delta_0(\varepsilon, C_1, c_1, \gamma)$ such that if $\delta \geq \delta_0$, then 
\begin{equation*} 
\sin \angle(v_1,v_1') \leq  C_0  \left(  \frac{ r^{1/\gamma}} {\delta}  +     \frac{ \|E\|}{\sigma_1}  + \frac{ \|E\|^2}{\sigma_1 \delta} \right)
\end{equation*} 
with probability at least $1-\varepsilon$.  

The first term $\frac{r ^{1/\gamma}} {\delta} $  on the right-hand side  corresponds to the conjectured optimal bound \eqref{bound2}. The second term $\frac{\|E\|}{ \sigma_1}$ is necessary. 
If $\|E \| \gg \sigma_1$, then  the intensity of the noise is much stronger than the strongest  signal in the data matrix, so $E$ would corrupt  $A$ completely. 
Thus in order to retain crucial information about $A$, it seems  necessary  to assume $\|E\| < \sigma_1$.  We are not absolutely sure about the necessity of the third term 
$ \frac{ \|E\|^2}{\sigma_1 \delta}$, but under the condition $\|E\| \ll \sigma_1 $, this term is superior to the Davis-Kahan-Wedin bound $\frac{\|E\| }{ \delta}$.  \end{remark}

We are able to  extend Theorem \ref{thm:main} in two different ways. First, we can bound the angle between $v_j$ and $v_j'$ for any index $j$. Second,  and more importantly, we can bound the 
angle between the subspaces spanned by $\{v_1, \dots, v_j \}$ and $\{v_1', \dots, v_j' \}$, respectively. As the projection onto the subspaces spanned by the first few singular vectors  (i.e.  low rank approximation)  plays an important role in  a vast collection of problems,  this result potentially has a large number of applications.  We are going to present these two results in the next section. 

To conclude this section, let us mention that related results have been obtained in the case where the random matrix $E$ contains Gaussian entries.  In \cite{RRW}, R.~Wang estimates the non-asymptotic distribution of the singular vectors when the entries of $E$ are iid standard normal random variables.  Recently, Allez and Bouchaud have studied the eigenvector dynamics of $A+E$ when $A$ is a real symmetric matrix and $E$ is a symmetric Brownian motion (that is, $E$ is a diffusive matrix process constructed from a family of independent real Brownian motions) \cite{AB}.  Our results also seems to have a close tie to 
the study of spiked covariance matrices, where a different kind of perturbation has been considered; see \cite{Ma, John,Nadler} for details. It would be  interesting to 
find a common generalization for these problems. 

\section{General theorems} 
 

First, we consider the problem of approximating the $j$-th singular vector $v_j$ for any $j$.  In light of the Davis-Kahan-Wedin result and Theorem \ref{thm:main}, it is natural to consider the gap 
$$ \delta_j := \sigma_j - \sigma_{j+1}. $$


\begin{theorem} \label{thm:general}
Assume that $E$ is $(C_1, c_1, \gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$.  Suppose $A$ has rank $r$, and let $1 \leq j \leq r$ be an integer.  Then, for any $t>0$,
$$ \sin \angle (v_j, v_j') \leq 4 \sqrt{2} \left( \left( \sum_{i=1}^{j-1} \sin^2 \angle (v_i, v_i') \right)^{1/2} + \frac{t r^{1/\gamma}}{\delta_j} + \frac{\|E\|^2}{\sigma_j \delta_j} + \frac{\|E\|}{\sigma_j} \right) $$
with probability at least 
$$ 1 - 6C_1 9^j \exp \left( -c_1 \frac{\delta_j^\gamma}{8^\gamma} \right) - 2C_1 9^{2r} \exp \left( -c_1 r \frac{t^\gamma}{4^\gamma} \right). $$ 
\end{theorem}

In the next theorem, we bound the largest principal angle between
\begin{equation} \label{eq:uspan}
	V := \mathrm{Span}\{v_1, \ldots, v_j\} \quad \text{and}\quad V' := \mathrm{Span}\{v_1', \ldots, v_j'\}
\end{equation}
for some integer $1 \leq j \leq r$, where $r$ is the rank of $A$.  


Let us recall that  if $U$ and $V$ are two subspaces of the same dimension, then the (principal) angle between them is defined as
\begin{equation} \label{eq:ssad}
	\sin \angle(U,V) := \max_{u \in U; u \neq 0} \min_{v \in V; v \neq 0} \sin \angle(u,v) = \|P_U - P_V \| = \|P_{U^\perp} P_{V} \|, 
\end{equation}
where $P_W$ denotes the orthogonal projection onto subspace $W$. 



\begin{theorem} \label{thm:subspace}
Assume that $E$ is $(C_1, c_1, \gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$.  Suppose $A$ has rank $r$, and let $1 \leq j \leq r$ be an integer.  Then, for any $t>0$,
$$ \sin \angle(V,V') \leq 4 \sqrt{2j} \left( \frac{t r^{1/\gamma}}{\delta_j} + \frac{\|E\|^2}{\sigma_j \delta_j} + \frac{ \|E\|}{\sigma_j} \right), $$
with probability at least 
$$ 1 - 6C_1 9^j \exp \left( -c_1 \frac{\delta_j^\gamma}{8^\gamma} \right) - 2C_1 9^{2r} \exp \left( -c_1 r \frac{t^\gamma}{4^\gamma} \right), $$ 
where $V$ and $V'$ are the $j$-dimensional subspaces defined in \eqref{eq:uspan}.
\end{theorem}

 It remains an open question 
to give an efficient bound for subspaces corresponding to an arbitrary set of singular values.  However, we can use Theorem \ref{thm:subspace} repeatedly to obtain bounds for the case when one considers a few intervals of singular values.  For instance, by applying Theorem \ref{thm:subspace} twice, we  obtain

\begin{corollary}
Assume that $E$ is $(C_1, c_1, \gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$.  Suppose $A$ has rank $r$, and let $1 < j \leq l \leq r$ be integers.  Then, for any $t>0$,
$$ \sin \angle(V,V') \leq 8 \sqrt{2l} \left( \frac{t r^{1/\gamma}}{\delta_{j-1}} + \frac{t r^{1/\gamma}}{\delta_l} + \frac{\|E\|^2}{\sigma_{j-1} \delta_{j-1}} + \frac{\|E\|^2}{\sigma_l \delta_l} +\frac{ \|E\|}{\sigma_l} \right), $$
with probability at least 
$$ 1 - 6C_1 9^{j-1} \exp \left( -c_1 \frac{\delta_{j-1}^\gamma}{8^\gamma} \right) - 6C_1  9^l \exp \left( -c_1 \frac{\delta_l^\gamma}{8^\gamma} \right) - 4C_1 9^{2r} \exp \left( -c_1 r \frac{t^\gamma}{4^\gamma} \right), $$ 
where
$$ V:= \mathrm{Span}\{v_j,\ldots, v_l\} \quad \text{and}\quad V':=\mathrm{Span}\{v_j',\ldots,v_l'\}. $$
\end{corollary}
\begin{proof}
Let
\begin{align*}
	V_1 &:= \mathrm{Span}\{v_1,\ldots,v_l\}, \quad V_1' := \mathrm{Span}\{v_1',\ldots,v_l'\}, \\
	V_2 &:= \mathrm{Span}\{v_1,\ldots,v_{j-1}\}, \quad V_2' := \mathrm{Span}\{v_1',\ldots,v_{j-1}'\}.
\end{align*}
For any subspace $W$, let $P_W$ denote the orthogonal projection onto $W$.  It follows that $P_{W^\perp} = I - P_{W}$, where $I$ denotes the identity matrix.  By definition of the subspaces $V,V'$, we have
$$ P_V = P_{V_1} P_{V_2^\perp} \quad \text{and}\quad P_{V'} = P_{V_1'} P_{V_2'^\perp}. $$
Thus, by \eqref{eq:ssad}, we obtain
\begin{align*}
	\sin \angle(V,V') &= \| P_{V_1} P_{V_2^\perp} - P_{V_1'} P_{V_2'^\perp} \| \\
		&\leq \| P_{V_1} P_{V_2^\perp} - P_{V_1'} P_{V_2^\perp} \| + \| P_{V_1'} P_{V_2^\perp} - P_{V_1'} P_{V_2'^\perp} \| \\
		&\leq \| P_{V_1} - P_{V_1'} \| + \| P_{V_2} - P_{V_2'} \| \\
		&= \sin \angle (V_1,V_1') + \sin \angle(V_2,V_2').
\end{align*}
Theorem \ref{thm:subspace} can now be invoked to bound $\sin\angle(V_1,V_1')$ and $\sin\angle(V_2,V_2')$, and the claim follows.  
\end{proof}


\noindent Finally, let us present the general form of Theorem \ref{thm:probweyl0} for singular values. 

\begin{theorem} \label{thm:probweyl}
Assume that $E$ is $(C_1, c_1, \gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$.  Suppose $A$ has rank $r$, and let $1 \leq j \leq r$ be an integer.  Then, for any $t>0$, 
\begin{equation} \label{eq:probweylbndlower}
	\sigma_j' \geq \sigma_j - t
\end{equation} 
with probability at least 
$$ 1 - 2C_1 9^j \exp \left( -c_1 \frac{t^\gamma}{4^\gamma} \right), $$
and
\begin{equation} \label{eq:probweylbndupper}
	\sigma_j' \leq \sigma_j + t r^{1/\gamma} + 2\sqrt{j} \frac{ \|E\|^2}{\sigma_j'} + j \frac{\|E\|^3}{{\sigma_j'}^2} 
\end{equation}
with probability at least 
$$ 1 - 2C_1 9^{2r} \exp \left( -c_1 r \frac{t^\gamma}{4^\gamma} \right). $$
\end{theorem}

\begin{remark}
Notice that the  upper bound for $\sigma_j'$ given in \eqref{eq:probweylbndupper} involves 
$1/\sigma_j'$.  In many situations, the lower bound in \eqref{eq:probweylbndlower} can be used to provide an upper bound for $1/\sigma_j'$.  
\end{remark}

\section{Overview and outline}

We now briefly give an overview of the paper and discuss some of the key ideas behind the proof of our main results.  For simplicity, let us assume that $A$ and $E$ are $n \times n$ real symmetric matrices.  (In fact, we will symmetrize the problem in Section \ref{sec:prelim} below.)  Let $\sigma_1 \geq \cdots \geq \sigma_n$ be the eigenvalues of $A$ with corresponding (orthonormal) eigenvectors $v_1, \ldots, v_n$.  Let $\sigma_1'$ be the largest eigenvalue of $A+E$ with corresponding (unit) eigenvector $v_1'$.

Suppose we wish to bound $\sin \angle(v_1, v_1')$ (from Theorem \ref{thm:main}).  Since 
$$ \sin^2 \angle (v_1, v_1') = 1- \cos^2 \angle(v_1, v_1') = \sum_{k=2}^n | v_k \cdot v_1' |^2, $$
it suffices to bound $|v_k \cdot v_1'|$ for $k=2, \ldots, n$.  Let us consider the case when $k=2, \ldots, r$.  In this case, we have
$$ v_k^\mathrm{T} (A+E) v_1' - v_k^\mathrm{T} A v_1' = v_k^\mathrm{T} E v_1'. $$
Since $(A+E) v_1' = \sigma_1' v_1'$ and $v_k^\mathrm{T} A = \sigma_k v_k$, we obtain
$$ |\sigma_1' - \sigma_k| |v_k \cdot v_1'| \leq | v_k^\mathrm{T}E v_1' |. $$

Thus, the problem of bounding $|v_k \cdot v_1'|$ reduces to obtaining an upper bound for $| v_k^\mathrm{T}E v_1' |$ and a lower bound for the gap $|\sigma_1' - \sigma_k|$.  We will obtain bounds for both of these terms by using the concentration property (Definition \ref{def:concentration}).  

More generally, in Section \ref{sec:prelim}, we will apply the concentration property to obtain lower bounds for the gaps $\sigma_j' - \sigma_k$ when $j < k$, which will hold with high probability.  Let us illustrate this by now considering the gap $\sigma_1' - \sigma_2$.  Indeed, we note that
$$ \sigma_1' = \| A + E \| \geq v_1^\mathrm{T} (A+E) v_1 = \sigma_1 + v_1^\mathrm{T} E v_1. $$
Applying the concentration property \eqref{eq:concentration}, we see that $\sigma_1' > \sigma_1 - t$ with probability at least $1 - C_1 \exp(- c_1 t^{\gamma})$.  As $\delta := \sigma_1 - \sigma_2$, we in fact observe that
$$ \sigma_1' - \sigma_2 = \sigma_1' - \sigma_1 + \delta > \delta - t. $$
Thus, if $\delta$ is sufficiently large, we have (say) $\sigma_1' - \sigma_2 \geq \delta/2$ with high probability.  

In Section \ref{sec:proof}, we will again apply the concentration property to obtain upper bounds for terms of the form $v_k E v_j'$.  At the end of Section \ref{sec:proof}, we combine these bounds to complete the proof of Theorems  \ref{thm:main}, \ref{thm:general}, \ref{thm:subspace} and \ref{thm:probweyl}.  In Section \ref{sec:concentration}, we discuss the $(C_1,c_1,\gamma)$-concentration property (Definition \ref{def:concentration}).  In particular, we generalize some previous results obtained by the second author in \cite{V}.  Finally, in Section \ref{section:app}, we present some applications of our main results. 


 Among others, our results seem useful for matrix recovery problems. The general matrix recovery problem  is the following.  $A$ is a large matrix.  However, the matrix $A$ is unknown to us.  We can only observe its noisy perturbation $A+E$, or in some cases just a small portion of the perturbation.  Our goal is to reconstruct $A$  or estimate an  important parameter
as accurately as possible from this observation. 
Furthermore,  several problems from combinatorics and theoretical computer science  can also be formulated in this setting. 
Special instances  of the matrix recovery problem have been investigated by many researchers using  spectral techniques  and combinatorial arguments in ingenious ways \cite{AM, AK, AKS,AzarMc,CCS,CP,CR,CRT,CT,Cest, DGP, KMO,KMO2,Krank,KLT,Kucera,MHT,Mc,NW,RVsamp}.  

We propose   the following simple analysis:   if  $A$ has rank $r$ and $1 \le j \le r$, then the projection of $A+E$ on the subspace  $V'$ spanned by the first $j$ singular vectors of $A+E$ is close to the projection of $A+E$ onto  the subspace  $V$ spanned by the first $j$ singular vectors of $A$, as our new results show that $V$ and $V'$ are very  close. Moreover, we can also show that the projection of $E$  onto $V$ is typically small.  Thus,  by projecting $A+E$ onto $V'$, we obtain  a good approximation of the rank $j$ approximation of $A$.   In certain cases, we  can repeat the above  operation a few times to obtain sufficient 
information to recover $A$ completely or to estimate the required parameter  with high accuracy and certainty.  



\section{Preliminary tools} \label{sec:prelim}

In this section, we present some of the preliminary tools we will need to prove Theorems \ref{thm:main}, \ref{thm:general}, \ref{thm:subspace}, and \ref{thm:probweyl}.  

To begin, we define the $(m+n) \times (m+n)$ symmetric block matrices
\begin{equation} \label{eq:def:tildeA}
	\tilde{A} := \begin{bmatrix} 0 & A \\ A^\mathrm{T} & 0 \end{bmatrix} 
\end{equation}
and 
$$ \tilde{E} := \begin{bmatrix} 0 & E \\ E^\mathrm{T} & 0 \end{bmatrix}. $$

We will work with the matrices $\tilde{A}$ and $\tilde{E}$ instead of $A$ and $E$.  In particular, the non-zero eigenvalues of $\tilde{A}$ are $\pm \sigma_1, \ldots, \pm \sigma_r$ and the eigenvectors are formed from the left and right singular vectors of $A$.  Similarly, the non-trivial eigenvalues of $\tilde{A} + \tilde{E}$ are $\pm \sigma_1', \ldots, \pm \sigma_{\min\{m,n\}}'$ (some of which may be zero) and the eigenvectors are formed from the left and right singular vectors of $A+E$.  

Along these lines, we introduce the following notation, which differs from the notation used above.  The non-zero eigenvalues of $\tilde{A}$ will be denoted by $\pm \sigma_1, \ldots, \pm \sigma_r$ with orthonormal eigenvectors $u_k$, $k=\pm 1, \ldots, \pm r$ such that
$$ \tilde{A} u_k = \sigma_k u_k, \qquad \tilde{A} u_{-k} = - \sigma_k u_{-k}, \qquad k = 1, \ldots, r. $$
Let $v_1, \ldots, v_j$ be the orthonormal eigenvectors of $\tilde{A}+\tilde{E}$ corresponding to the $j$-largest eigenvalues $\lambda_1 \geq \cdots \geq \lambda_j$.

In order to prove Theorems \ref{thm:main}, \ref{thm:general}, \ref{thm:subspace}, and \ref{thm:probweyl}, it suffices to work with the eigenvectors and eigenvalues of the matrices $\tilde{A}$ and $\tilde{A}+\tilde{E}$.  Indeed, Proposition \ref{prop:sine} will bound the angle between the singular vectors of $A$ and $A+E$ by the angle between the corresponding eigenvectors of $\tilde{A}$ and $\tilde{A} + \tilde{E}$.  

\begin{proposition} \label{prop:sine}
Let $u_1, v_1 \in \mathbb{R}^m$ and $u_2, v_2 \in \mathbb{R}^n$ be unit vectors.  Let $u, v \in \mathbb{R}^{m+n}$ be given by
$$ u = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix}, \quad v = \begin{bmatrix} v_1 \\ v_2 \end{bmatrix}.  $$
Then
$$ \sin^2 \angle(u_1, v_1) + \sin^2 \angle(u_2, v_2) \leq 2 \sin^2 \angle(u, v). $$
\end{proposition}
\begin{proof}
Since $\|u\|^2 = \|v\|^2 = 2$, we have
\begin{align*}
	\cos^2 \angle(u,v) = \frac{1}{4} |u \cdot v|^2 \leq \frac{1}{2} |u_1 \cdot v_1|^2 + \frac{1}{2} |u_2 \cdot v_2|^2.
\end{align*}
Thus, 
\begin{align*}
	\sin^2 \angle(u,v) = 1 - \cos^2 \angle(u,v) \geq \frac{1}{2} \sin^2 \angle(u_1, v_1) + \frac{1}{2} \sin^2\angle(u_2, v_2),
\end{align*}
and the claim follows.
\end{proof}

We now introduce some useful lemmas.  The first lemma below, states that if $E$ is $(C_1,c_1,\gamma)$-concentrated, then $\tilde{E}$ is $(\tilde{C}_1,\tilde{c}_1,\gamma)$-concentrated, for some new constants $\tilde{C}_1 := 2C_1$ and $\tilde{c}_1:= c_1/2^{\gamma}$. 

\begin{lemma} \label{lemma:tilde}
Assume that $E$ is $(C_1, c_1,\gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$.  Let $\tilde{C}_1 := 2C_1$ and $\tilde{c}_1:= c_1/2^{\gamma}$.  Then for all unit vectors $u,v \in \mathbb{R}^{n+m}$, and every $t>0$,
\begin{equation} \label{eq:tilde-concentration}
	\mathbb{P}( |u^t \tilde{E} v| > t ) \leq \tilde{C}_1 \exp(-\tilde{c}_1 t^\gamma ).
\end{equation}	
\end{lemma}
\begin{proof}
Let 
$$ u = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix}, \quad v = \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} $$
be unit vectors in $\mathbb{R}^{m+n}$, where $u_1, v_1 \in \mathbb{R}^m$ and $u_2,v_2 \in \mathbb{R}^n$.  We note that
$$  u^{\mathrm{T}} \tilde{E} v  = u_1^\mathrm{T} E v_2 + u_2^\mathrm{T} E^\mathrm{T} v_1. $$
Thus, if any of the vectors $u_1, u_2, v_1, v_2$ are zero, \eqref{eq:tilde-concentration} follows immediately from \eqref{eq:concentration}.  Assume all the vectors $u_1, u_2, v_1, v_2$ are nonzero.  Then
$$ | u^{\mathrm{T}} \tilde{E} v | = |u_1^\mathrm{T} E v_2 + u_2^\mathrm{T} E^\mathrm{T} v_1| \leq \frac{|u_1^\mathrm{T} E v_2|}{\|u_1\| \|v_2\|} + \frac{|v_1^\mathrm{T} E u_2|}{\|u_2\|\|v_1\|}. $$
Thus, by \eqref{eq:concentration}, we have
\begin{align*}
	\mathbb{P}( |u^\mathrm{T} \tilde{E} v| > t) &\leq \mathbb{P} \left( \frac{|u_1^\mathrm{T} E v_2|}{\|u_1\| \|v_2\|}  > \frac{t}{2} \right) + \mathbb{P} \left( \frac{|v_1^\mathrm{T} E u_2|}{\|u_2\|\|v_1\|} > \frac{t}{2} \right) \\
	&\leq 2C_1 \exp \left( -c_1 \frac{ t^\gamma }{ 2^\gamma } \right),
\end{align*}
and the proof of the lemma is complete.  
\end{proof}

We will also consider the spectral norm of $\tilde{E}$.  Since $\tilde{E}$ is a symmetric matrix whose eigenvalues in absolute value are given by the singular values of $E$, it follows that
\begin{equation} \label{eq:normE}
	\| \tilde{E} \| = \|E \|. 
\end{equation}

We introduce $\varepsilon$-nets as a convenient way to discretize a compact set.  Let $\varepsilon > 0$.  A set $X$ is an $\varepsilon$-net of a set $Y$ if for any $y \in Y$, there exists $x \in X$ such that $\|x-y\| \leq \varepsilon$.  The following estimate for the maximum size of an $\varepsilon$-net of a sphere is well-known (see for instance \cite{RV}).

\begin{lemma} \label{lemma:net}
A unit sphere in $d$ dimensions admits an $\varepsilon$-net of size at most 
$$ \left(1+\frac{2}{\varepsilon} \right)^d.$$
\end{lemma}

Lemmas \ref{lemma:r-norm}, \ref{lemma:largest}, and \ref{lemma:j-largest} below are consequences of  the concentration property \eqref{eq:tilde-concentration}.  

\begin{lemma} \label{lemma:r-norm}
Assume that $E$ is $(C_1, c_1, \gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$.  Let $A$ be a $m \times n$ matrix with rank $r$.  Let $U$ be the $(m+n) \times 2r$ matrix whose columns are the vectors $u_1, \ldots, u_r, u_{-1}, \ldots, u_{-r}$.  Then, for any $t > 0$,
$$ \mathbb{P} \left( \|U^\mathrm{T} \tilde{E}U\| >  t r^{1/\gamma} \right) \leq \tilde{C}_1 9^{2r} \exp\left( - \tilde{c}_1 r \frac{t^\gamma}{2^\gamma} \right). $$
\end{lemma}
\begin{proof}
Clearly $U^\mathrm{T} \tilde{E} U$ is a symmetric  $2r \times 2r$ matrix.  Let $S$ be the unit sphere in $\mathbb{R}^{2r}$.  Let $\mathcal{N}$ be a $1/4$-net of $S$.  It is easy to verify (see for instance \cite{RV}) that for any $2r \times 2r$ symmetric matrix $B$,
$$ \|B\| \leq 2 \max_{x \in \mathcal{N}} |x^\ast B x| . $$

For any fixed $x \in \mathcal{N}$, we have
$$ \mathbb{P}( |x^\mathrm{T} U^\mathrm{T} \tilde{E} U x | > t) \leq \tilde{C}_1 \exp(-\tilde{c}_1 t^\gamma) $$
by Lemma \ref{lemma:tilde}.  Since $|\mathcal{N}| \leq 9^{2r}$, we obtain
\begin{align*}
	\mathbb{P} ( \| U^\mathrm{T} \tilde{E} U \| > t r^{1/\gamma}) &\leq \sum_{x \in \mathcal{N}} \mathbb{P}\left( |x^\mathrm{T} U^\mathrm{T} \tilde{E} U x | > \frac{1}{2} t r^{1/\gamma} \right) \\
	& \leq \tilde{C}_1 9^{2r} \exp\left(-\tilde{c}_1 r \frac{t^\gamma}{2^{\gamma}} \right).
\end{align*}
\end{proof}

\begin{lemma} \label{lemma:largest}
Assume that $E$ is $(C_1, c_1, \gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$.  Suppose $A$ has rank $r$.  Then, for any $t > 0$, 
\begin{equation} \label{eq:lambda1bnd}
	\lambda_1 \geq \sigma_1 - t 
\end{equation}
with probability at least $1 - \tilde{C}_1 \exp(-\tilde{c}_1 t^{\gamma})$.  

In particular, if $\sigma_1 > 0$, then $\lambda_1 \geq \frac{\sigma_1}{2}$ with probability at least $1 - \tilde{C}_1 \exp \left( -\tilde{c}_1 \frac{\sigma_1^\gamma}{2^\gamma} \right)$.  If, in addition, $\delta > 0$, then 
$$ \lambda_1 - \sigma_k \geq \frac{1}{2} \delta $$
for $k = 2,\ldots, r $ with probability at least $1 - \tilde{C}_1 \exp \left( -\tilde{c}_1 \frac{\delta^\gamma}{2^\gamma} \right)$.
\end{lemma}
\begin{proof}
We observe that
$$ \lambda_1 = \|\tilde{A} + \tilde{E}\| \geq u_1^\mathrm{T} (\tilde{A} + \tilde{E}) u_1 = \sigma_1 + u_1^\mathrm{T} \tilde{E} u_1. $$
By Lemma \ref{lemma:tilde}, we have 
$$ \mathbb{P}( |u_1^\mathrm{T} \tilde{E} u_1| > t) \leq \tilde{C}_1 \exp( -\tilde{c}_1 t^\gamma) $$
for every $t > 0$, and \eqref{eq:lambda1bnd} follows.  

If $\sigma_1 > 0$, then the bound $\lambda_1 \geq \frac{\sigma_1}{2}$ can be obtained by taking $t = \sigma_1/2$ in \eqref{eq:lambda1bnd}.  Assume $\delta > 0$.  Taking $t=\delta/2$ in \eqref{eq:lambda1bnd} yields 
$$ \lambda_1 - \sigma_k \geq \lambda_1 - \sigma_2 = \lambda_1 - \sigma_1 + \delta \geq \frac{\delta}{2} $$
for $k=2, \ldots, r$ with probability at least $1 - \tilde{C}_1 \exp \left( -\tilde{c}_1 \frac{\delta^\gamma}{2^\gamma} \right)$.
\end{proof}

Using the Courant minimax principle, Lemma \ref{lemma:largest} can be generalized to the following.

\begin{lemma} \label{lemma:j-largest}
Assume that $E$ is $(C_1, c_1, \gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$.  Suppose $A$ has rank $r$, and let $1 \leq j \leq r$ be an integer.  Then, for any $t > 0$, 
\begin{equation} \label{eq:lambdajbnd1}
	\lambda_j \geq \sigma_j - t
\end{equation}
with probability at least $1 - \tilde{C}_1 9^j \exp\left( - \tilde{c}_1 \frac{t^\gamma}{2^\gamma} \right)$.  

In particular, $\lambda_j \geq \frac{\sigma_j}{2}$ with probability at least $1 - \tilde{C}_1 9^j \exp \left( - \tilde{c}_1 \frac{\sigma_j^\gamma}{4^\gamma} \right)$. In addition, if $\delta_j > 0$, then 
\begin{equation} \label{eq:lambdajbnd2}
	\lambda_j - \sigma_k \geq \frac{\delta_j}{2}
\end{equation}
for $k=j+1, \ldots, r$ with probability at least $1 - \tilde{C}_1 9^j \exp \left( - \tilde{c}_1 \frac{\delta_j^\gamma}{4^\gamma} \right)$. 
\end{lemma}
\begin{proof}
It suffices to prove \eqref{eq:lambdajbnd1}.  Indeed, the bound $\lambda_j \geq \frac{\sigma_j}{2}$ follows from \eqref{eq:lambdajbnd1} by taking $t = \sigma_j/2$, and \eqref{eq:lambdajbnd2} follows by taking $t = \delta_j/2$.

Let $S$ be the unit sphere in $\mathrm{Span}\{u_1,\ldots,u_j\}$.  By the Courant minimax principle,
\begin{align*}
	\lambda_j &= \max_{\dim(V)=j} \min_{\|v\|=1;v\in V} v^\mathrm{T} (\tilde{A}+\tilde{E})v \\
		& \geq \min_{v \in S} v^\mathrm{T} (\tilde{A}+\tilde{E})v \\
		& \geq \sigma_j + \min_{v \in S} v^\mathrm{T} \tilde{E} v.
\end{align*}
Thus, it suffices to show
$$ \mathbb{P}\left( \sup_{v \in S} |v^\mathrm{T} \tilde{E} v| > t \right) \leq \tilde{C}_1 9^j \exp\left( -\tilde{c}_1 \frac{t^\gamma}{2^\gamma} \right) $$
for all $t > 0$.  

Let $\mathcal{N}$ be a $1/4$-net of $S$.  By Lemma \ref{lemma:net}, $|\mathcal{N}| \leq 9^{j}$.  We now claim that 
\begin{equation} \label{eq:supmaxnet}
	T := \sup_{v \in S} | v^\mathrm{T} \tilde{E} v| \leq 2 \max_{ u \in \mathcal{N}} |u^\mathrm{T} \tilde{E} u|. 
\end{equation}
Indeed, fix a realization of $\tilde{E}$.  Since $S$ is compact, there exists $v \in S$ such that $T = |v^\mathrm{T} \tilde{E} v|$.  Moreover, there exists $x \in \mathcal{N}$ such that $\|x - v\| \leq 1/4$.  Clearly the claim is true when $x = v$; assume $x \neq v$.  Then, by the triangle inequality, we have
\begin{align*}
	T &\leq |v^\mathrm{T} \tilde{E} v - v^\mathrm{T} \tilde{E} x| + |v^\mathrm{T} \tilde{E} x - x^\mathrm{T} \tilde{E} x| + |x^\mathrm{T} \tilde{E} x| \\
		&\leq \frac{1}{4} \frac{ |v^\mathrm{T} \tilde{E} (v-x)| }{\| v-x\|} + \frac{1}{4} \frac{ |(v-x)^\mathrm{T} \tilde{E} x |}{\|v-x\| } + \sup_{u \in \mathcal{N}} |u^\mathrm{T} \tilde{E} u| \\
		& \leq \frac{T}{2} + \sup_{u \in \mathcal{N}} |u^\mathrm{T} \tilde{E} u|,
\end{align*}
and \eqref{eq:supmaxnet} follows.  

Applying \eqref{eq:supmaxnet} and Lemma \ref{lemma:tilde}, we have
\begin{align*}
	\mathbb{P} \left( \sup_{v \in S} |v^\mathrm{T} \tilde{E} v| > t \right) &\leq \sum_{u \in \mathcal{N}} \mathbb{P} \left( |u^\mathrm{T} \tilde{E} u| > \frac{t}{2} \right) \leq 9^j \tilde{C}_1 \exp \left( -\tilde{c}_1 \frac{t^\gamma}{2^\gamma} \right),
\end{align*}
and the proof of the lemma is complete.  
\end{proof}

We will continually make use of the following simple fact:
\begin{equation} \label{eq:aea}
	(\tilde{A} + \tilde{E}) - \tilde{A} = \tilde{E}.
\end{equation}


\section{Proof of Theorems \ref{thm:main}, \ref{thm:general}, \ref{thm:subspace}, and \ref{thm:probweyl}} \label{sec:proof}

This section is devoted to Theorems \ref{thm:main}, \ref{thm:general}, \ref{thm:subspace}, and \ref{thm:probweyl}.  To begin, define the subspace
$$ W:= \mathrm{Span}\{u_1, \ldots, u_r, u_{-1}, \ldots, u_{-r} \}.$$
Let $P$ be the orthogonal projection onto $W^\perp$.  

\begin{lemma} \label{lemma:proj_bound}
Assume that $E$ is $(C_1, c_1, \gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$.  Suppose $A$ has rank $r$, and let $1 \leq j \leq r$ be an integer.  Then 
\begin{equation} \label{eq:suppvi}
	\sup_{1 \leq i \leq j} \|P v_i \| \leq 2 \frac{\|E\|}{\sigma_j}
\end{equation}
with probability at least $1 - \tilde{C}_1 9^j \exp \left( - \tilde{c}_1 \frac{\sigma_j^\gamma}{4^\gamma} \right)$.  
\end{lemma}
\begin{proof}
Consider the event 
$$ \Omega_j := \left\{ \lambda_j \geq \frac{1}{2} \sigma_j \right\}. $$
By Lemma \ref{lemma:j-largest} (or Lemma \ref{lemma:largest} in the case $j=1$), $\Omega_j$ holds with probability at least $1 - \tilde{C}_1 9^j \exp \left( - \tilde{c}_1 \frac{\sigma_j^\gamma}{4^\gamma} \right)$.    

Fix $1 \leq i \leq j$.  By multiplying \eqref{eq:aea} on the left by $(P v_i)^\mathrm{T}$ and on the right by $v_i$, we obtain
$$ | \lambda_i (P v_i)^\mathrm{T} v_i | \leq \| P v_i\| \|\tilde{E}\| $$
since $(P v_i)^\mathrm{T} \tilde{A} = 0$.  Thus, on the event $\Omega_j$, we have
$$ \| P v_i \|^2 = |(P v_i)^\mathrm{T} v_i | \leq \frac{1}{\lambda_j}\|P v_i\| \|\tilde{E}\| \leq \frac{2}{\sigma_j} \| P v_i \|\|\tilde{E}\|. $$
We conclude that, on the event $\Omega_j$, 
$$ \sup_{1 \leq i \leq j} \|P v_i \| \leq 2 \frac{\|E\|}{\sigma_j}, $$
and the proof is complete.  
\end{proof}

\begin{lemma} \label{lemma:uproj}
Assume that $E$ is $(C_1, c_1, \gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$.  Suppose $A$ has rank $r$, and let $1 \leq j \leq r$ be an integer.  Define $U_j$ to be the $(m+n) \times (2r-j)$ matrix with columns $u_{j+1}, \ldots, u_r, u_{-1}, \ldots, u_{-r}$.  Then, for any $t>0$,
\begin{equation} \label{eq:sujtv}
	\sup_{1 \leq i \leq j} \| U_j^\mathrm{T} v_i \| \leq 4 \left( \frac{t r^{1/\gamma}}{\delta_j} + \frac{\|E\|^2}{\delta_j \sigma_j} \right)
\end{equation}
with probability at least 
$$ 1 - 2\tilde{C}_1 9^j \exp \left( - \tilde{c}_1 \frac{\delta_j^\gamma}{4^\gamma}\right) - \tilde{C}_1 9^{2r} \exp \left( - \tilde{c}_1r \frac{t^\gamma}{2^\gamma} \right). $$
\end{lemma}
\begin{proof}
Define the event
\begin{align*}
\Omega_j &:= \left\{ \sup_{1 \leq i \leq j} \|P v_i \| \leq 2 \frac{\|E\|}{\sigma_j} \right\} \bigcap \left\{ \| U^\mathrm{T} \tilde{E} U \| \leq t r^{1/\gamma} \right\} \bigcap \left\{  \lambda_j - \sigma_{j+1} \geq \frac{\delta_j}{2} \right\}. 
\end{align*}
By Lemmas \ref{lemma:r-norm}, \ref{lemma:j-largest}, and \ref{lemma:proj_bound}, it follows that
$$ \mathbb{P}( \Omega_j ) \geq 1 - 2\tilde{C}_1 9^j \exp \left( - \tilde{c}_1 \frac{\delta_j^\gamma}{4^\gamma}\right) - \tilde{C}_1 9^{2r} \exp \left( - \tilde{c}_1r \frac{t^\gamma}{2^\gamma} \right). $$

Fix $1 \leq i \leq j$.  We multiply \eqref{eq:aea} on the left by $U_j^\mathrm{T}$ and on the right by $v_i$ to obtain
\begin{equation} \label{eq:ujae}
	U_j^\mathrm{T} (\tilde{A} + \tilde{E}) v_i - U_j^\mathrm{T} \tilde{A} v_i = U_j^\mathrm{T} \tilde{E} v_i. 
\end{equation}
We note that
$$ U_j^\mathrm{T} (\tilde{A} + \tilde{E}) v_i = \lambda_i U_j^\mathrm{T} v_i $$
and
$$ U_j^\mathrm{T} \tilde{A} v_i = D_j U_j^\mathrm{T} v_i, $$
where $D_j$ is the diagonal matrix with the values $\sigma_{j+1}, \ldots, \sigma_r, -\sigma_{1}, \ldots, -\sigma_{r}$ on the diagonal.  

For the right-hand side of \eqref{eq:ujae}, we write $v_i = U U^\mathrm{T} v_i + P v_i$, where $U$ is the matrix with columns $u_1, \ldots, u_r, u_{-1}, \ldots, u_{-r}$ and $P$ is the orthogonal projection onto $W^\perp$.  Thus, on the event $\Omega_j$, we have
\begin{align*}
	\|U_j^\mathrm{T} \tilde{E} v_i\| \leq \|U_j^\mathrm{T} \tilde{E} U\| + \|\tilde{E}\| \|P v_i\| \leq t r^{1/\gamma} + 2 \frac{\|E\|^2}{\sigma_j}.
\end{align*}
Here we used the fact that $U_j^\mathrm{T} \tilde{E} U$ is a sub-matrix of $U^\mathrm{T} \tilde{E} U$ and hence 
$$ \|U_j^\mathrm{T} \tilde{E} U\| \leq \| U^\mathrm{T} \tilde{E} U\|. $$


Combining the above computations and bound yields
$$ \| (\lambda_i I - D_j) U_j^\mathrm{T} v_i \| \leq 2 \left( t r^{1/\gamma} + \frac{\|E\|^2}{\sigma_j} \right) $$
on the event $\Omega_j$.    

We now consider the entries of the diagonal matrix $\lambda_i I - D_j$.  On $\Omega_j$, we have that, for any $k \geq j+1$,
$$ \lambda_i - \sigma_k \geq \lambda_j - \sigma_{j+1} \geq \frac{\delta_j}{2}. $$
By writing the elements of the vector $U_j^\mathrm{T} v_i$ in component form, it follows that
$$ \|(\lambda_i I - D_j) U_j^\mathrm{T} v_i \| \geq \frac{\delta_j}{2} \| U_j^\mathrm{T} v_i \| $$
and hence
$$  \| U_j^\mathrm{T} v_i \| \leq 4 \left( \frac{t r^{1/\gamma}}{\delta_j} + \frac{\|E\|^2}{\sigma_j \delta_j} \right) $$
on the event $\Omega_j$.  Since this holds for each $1 \leq i \leq j$, the proof is complete.  
\end{proof}

With Lemmas \ref{lemma:proj_bound} and \ref{lemma:uproj} in hand, we now prove Theorems \ref{thm:main}, \ref{thm:general}, \ref{thm:subspace}, and \ref{thm:probweyl}.  By Proposition \ref{prop:sine}, in order to prove Theorems \ref{thm:main} and \ref{thm:general}, it suffices to bound $\sin \angle (u_j, v_j)$  because $u_j, v_j$ are formed from the left and right singular vectors of $A$ and $A+E$.  

\begin{proof}[Proof of Theorem \ref{thm:main}]
We write 
$$ v_1 = \sum_{k=1}^r \alpha_k u_k + \sum_{k=1}^r \alpha_{-k} u_{-k} + P v_1, $$  
where $P$ is the orthogonal projection onto $W^\perp$.  Then
$$ \sin^2 \angle (u_1, v_1) = 1- \cos^2 \angle (u_1, v_1) = \sum_{k=2}^r |\alpha_k|^2 + \sum_{k=1}^r |\alpha_{-k}|^2 + \| P v_1 \|^2. $$
Applying the bounds obtained from Lemmas \ref{lemma:proj_bound} and \ref{lemma:uproj} (with $j=1$), we obtain
$$ \sin^2 \angle (u_1, v_1) \leq 16 \left( \frac{t r^{1/\gamma}}{\delta} + \frac{\|E\|^2}{\sigma_1 \delta} \right)^2 + 4 \frac{\|E\|^2}{ \sigma_1^2} $$
with probability at least 
\begin{equation} \label{eq:probholdmain}
	1 - 27 \tilde{C}_1 \exp \left( -\tilde{c}_1 \frac{\delta^\gamma}{4^\gamma} \right) - \tilde{C}_1 9^{2r} \exp \left(- \tilde{c}_1 r \frac{t^\gamma}{2^\gamma} \right). 
\end{equation}
We now note that
\begin{align*}
	16 \left( \frac{t r^{1/\gamma}}{\delta} + \frac{\|E\|^2}{\sigma_1 \delta} \right)^2 + 4 \frac{\|E\|^2}{ \sigma_1^2}  &\leq 16  \left( \frac{t r^{1/\gamma}}{\delta} + \frac{\|E\|^2}{\sigma_1 \delta} + \frac{\|E\|}{ \sigma_1} \right)^2.
\end{align*}
The correct absolute constant in front can now be deduced from the bound above and Proposition \ref{prop:sine}.  The lower bound on the probability given in \eqref{eq:probholdmain} can be written in terms of the constants $C_1, c_1, \gamma$ by recalling the definitions of $\tilde{C}_1$ and $\tilde{c}_1$ given in Lemma \ref{lemma:tilde}.   
\end{proof}

\begin{proof}[Proof of Theorem \ref{thm:general}]
We again write 
\begin{equation} \label{eq:vjproj}
	v_j = \sum_{k=1}^r \alpha_k u_k + \sum_{k=1}^r \alpha_{-k} u_{-k} + P v_j, 
\end{equation}
where $P$ is the orthogonal projection onto $W^\perp$.  Then we have that
\begin{align*}
	\sin^2 \angle (u_j, v_j) &= 1- \cos^2 \angle (u_j, v_j) \\
		&= \sum_{k=1}^{j-1} |\alpha_k|^2 + \sum_{k=j+1}^r |\alpha_k|^2 + \sum_{k=1}^r |\alpha_{-k}|^2 + \|P v_j\|^2. 
\end{align*}
For any $1 \leq k \leq j-1$, we have that
$$ |\alpha_k|^2 = | v_j \cdot (u_k - v_k) |^2 \leq \|v_k - u_k\|^2 \leq 2 (1 - \cos \angle (v_k,u_k)) \leq 2 \sin^2 \angle(v_k,u_k). $$
Moreover, from Lemmas \ref{lemma:proj_bound} and \ref{lemma:uproj}, we have
$$ \sum_{k=j+1}^r |\alpha_k|^2 + \sum_{k=1}^r |\alpha_{-k}|^2 \leq 16 \left( \frac{t r^{1/\gamma}}{\delta_j} + \frac{\|E\|^2}{\sigma_j \delta_j} \right)^2 $$
with probability at least 
$$ 1 - 2\tilde{C}_1 9^j \exp \left( - \tilde{c}_1 \frac{\delta_j^\gamma}{4^\gamma}\right) - \tilde{C}_1 9^{2r} \exp \left( - \tilde{c}_1r \frac{t^\gamma}{2^\gamma} \right). $$ 
and
$$ \| P v_j \|^2 \leq 4  \frac{\|E\|^2}{\sigma_j^2} $$
with probability at least $1 - \tilde{C}_1 9^j \exp \left( - \tilde{c}_1 \frac{\sigma_j^\gamma}{4^\gamma} \right)$.  The proof of Theorem \ref{thm:general} is complete by combining the bounds above\footnote{Here the bounds are given in terms of $\sin^2 \angle(v_k, u_k)$ for $1 \leq k \leq j-1$.  However, $u_k$ and $v_k$ are formed from the left and right singular vectors of $A$ and $A+E$.  To avoid the dependence on both the left and right singular vectors, one can begin with \eqref{eq:vjproj} and consider only the coordinates of $v_j$ which correspond to the left (alternatively right) singular vectors.  By then following the proof for only these coordinates, one can bound the left (right) singular vectors by terms which only depend on the previous left (right) singular vectors.}.  As in the proof of Theorem \ref{thm:main}, the correct constant factor in front can be deduced from Proposition \ref{prop:sine}.  
\end{proof}

\begin{proof}[Proof of Theorem \ref{thm:subspace}]
Define the subspaces
$$ \tilde{U}:= \mathrm{Span}\{u_1, \ldots, u_j \} \quad \text{and} \quad \tilde{V}:= \mathrm{Span}\{v_1, \ldots, v_j\}. $$
By Proposition \ref{prop:sine}, it suffices to bound $\sin \angle(\tilde{U}, \tilde{V})$.  

Let $Q$ be the orthogonal projection onto $\tilde{U}^\perp$.  By Lemmas \ref{lemma:proj_bound} and \ref{lemma:uproj}, it follows that
\begin{equation} \label{eq:supqvi}
	\sup_{1 \leq i \leq j} \|Q v_i \| \leq 4 \left( \frac{t r^{1/\gamma} }{\delta_j} + \frac{\|E\|^2}{\sigma_j \delta_j} + \frac{\|E\|}{\sigma_j} \right) 
\end{equation}
with probability at least 
$$ 1 - 3\tilde{C}_1 9^j \exp \left( - \tilde{c}_1 \frac{\delta_j^\gamma}{4^\gamma}\right) - \tilde{C}_1 9^{2r} \exp \left( - \tilde{c}_1r \frac{t^\gamma}{2^\gamma} \right). $$
On the event where \eqref{eq:supqvi} holds, we have
$$ \sup_{v \in \tilde{V}, \|v\| = 1} \| Q v \| \leq 4 \sqrt{j} \left( \frac{t r^{1/\gamma} }{\delta_j} + \frac{\|E\|^2}{\sigma_j \delta_j} + \frac{\|E\|}{\sigma_j} \right)  $$
by the triangle inequality and the Cauchy-Schwarz inequality.  Thus, by \eqref{eq:ssad}, we conclude that
\begin{align*}
	\sin \angle(\tilde{U}, \tilde{V}) &\leq 4 \sqrt{j} \left( \frac{t r^{1/\gamma} }{\delta_j} + \frac{\|E\|^2}{\sigma_j \delta_j} + \frac{\|E\|}{\sigma_j} \right)
\end{align*}
on the event where \eqref{eq:supqvi} holds.  The claim now follows from Proposition \ref{prop:sine}.  
\end{proof}

\begin{proof}[Proof of Theorem \ref{thm:probweyl}]
The lower bound \eqref{eq:probweylbndlower} follows from Lemma \ref{lemma:j-largest}; it remains to prove \eqref{eq:probweylbndupper}.  Let $U$ be the $(m+n) \times 2r$ matrix whose columns are given by the vectors $u_1, \ldots, u_r, u_{-1}, \ldots, u_{-r}$, and recall that $P$ is the orthogonal projection onto $W^\perp$.  

Let $S$ denote the unit sphere in $\mathrm{Span}\{v_1, \ldots, v_j\}$.  Then for $1 \leq i \leq j$, we multiply \eqref{eq:aea} on the left by $v_i^\mathrm{T} P$ and on the right by $v_i$ to obtain
$$ \lambda_i \|P v_i \|^2 \leq \| v_i^\mathrm{T} P \tilde{E} v_i \| \leq \|P v_i \| \|E\|. $$
Here we used \eqref{eq:normE} and the fact that $P \tilde{A} = 0$.  Therefore, we have the deterministic bound
$$ \sup_{1 \leq i \leq j} \| P v_i \| \leq \frac{ \| E\|}{\lambda_j}. $$
By the Cauchy-Schwarz inequality, it follows that
\begin{equation} \label{eq:detsupbnd}
	\sup_{v \in S} \| P v \| \leq \sqrt{j} \frac{ \|E \| }{\lambda_j}. 
\end{equation}

By the Courant minimax principle, we have
\begin{align*}
	\sigma_j = \max_{\dim(V)=j} \min_{v\in V,\|v\|=1} v^\mathrm{T} \tilde{A}v \geq \min_{v \in S} v^\mathrm{T} \tilde{A} v \geq \lambda_j - \max_{v \in S} |v^\mathrm{T} \tilde{E} v|. 
\end{align*}
Thus, it suffices to show that
\begin{equation*}
	\max_{v \in S} |v^\mathrm{T} \tilde{E} v| \leq t r^{1/\gamma} + 2\sqrt{j} \frac{ \|E\|^2}{\lambda_j} + j \frac{\|E\|^3}{{\lambda_j}^2} 
\end{equation*}
with probability at least $1-\tilde{C}_1 9^{2r} \exp \left( - \tilde{c}_1 r \frac{t^\gamma}{2^\gamma} \right)$.  

We decompose $v = Pv + U U^\mathrm{T} v$ and obtain
\begin{align*}
	 \max_{v \in S} |v^\mathrm{T} \tilde{E} v| \leq \max_{v \in S} \|Pv\|^2 \|\tilde{E}\| + 2 \max_{v \in S} \| Pv \| \|\tilde{E}\| + \| U^\mathrm{T} \tilde{E} U \|.
\end{align*}
Thus, by Lemma \ref{lemma:r-norm} and \eqref{eq:detsupbnd}, we have
$$ \max_{v \in S} |v^\mathrm{T} \tilde{E} v| \leq j \frac{ \|E\|^3}{\lambda_j^2} + 2 \sqrt{j} \frac{ \|E\|^2}{\lambda_j} + t r^{1/\gamma} $$
with probability at least $1-\tilde{C}_1 9^{2r} \exp \left( - \tilde{c}_1 r \frac{t^\gamma}{2^\gamma} \right)$, and the proof is complete.   
\end{proof}


\section{The concentration property} \label{sec:concentration}

In this section, we give examples of random matrix models satisfying  Definition \ref{def:concentration}.  

\begin{lemma} \label{lemma:bernoulli}
There exists a constant $C_1$ such that the following holds.  Let $E$ be a random $n \times n$ Bernoulli matrix.  Then 
$$ \mathbb{P} ( \|E\| > 3 \sqrt{n} ) \leq \exp(-C_1 n), $$
and for any fixed unit vectors $u,v$ and positive number $t$,
$$ \mathbb{P} (|u^\mathrm{T} E v| \geq t ) \leq 2 \exp(-t^2/2). $$
\end{lemma}

The bounds in Lemma \ref{lemma:bernoulli} also hold for the case where the noise is Gaussian (instead of Bernoulli).  Indeed, when the entries of $E$ are iid standard normal random variables, $u^{\mathrm{T}} E v$ has the standard normal distribution. The first bound is a corollary of a general concentration result from \cite{V}. It can also be proved directly using a net argument. The second bound follows from martingale different sequence inequality \cite{McDiarmid}; see also \cite{V} for a direct proof with a more generous constant. 



We now verify the $(C_1,c_1,\gamma)$-concentration property for slightly more general random matrix models.  We will discuss these matrix models further in Section \ref{section:app}.  In the lemmas below, we consider both the case where $E$ is a real symmetric random matrix with independent entries and when $E$ is a non-symmetric random matrix with independent entries.  

\begin{lemma} \label{lemma:conc-sym}
Let $E = (\xi_{ij})_{i,j=1}^n$ be a $n \times n$ real symmetric random matrix where 
$$ \{ \xi_{ij} : 1 \leq i \leq j \leq n\} $$ 
is a collection of independent random variables each with mean zero.  Further assume
$$ \sup_{1 \leq i \leq j \leq n} |\xi_{ij}| \leq K $$  
with probability $1$, for some $K \geq 1$.  Then for any fixed unit vectors $u,v$ and every $t > 0$
$$ \mathbb{P} (|u^\mathrm{T} E v| \geq t) \leq 2 \exp \left( \frac{-t^2}{8 K^2 } \right). $$
\end{lemma}
\begin{proof}
We write
$$ u^\mathrm{T} E v = \sum_{1 \leq i < j \leq n} (u_i v_j + v_i u_j) \xi_{ij} + \sum_{i=1}^n u_i v_i \xi_{ii}. $$
As the right side is a sum of independent, bounded random variables, we apply Hoeffding's inequality (\cite[Theorem 2]{H}) to obtain 
$$ \mathbb{P} (|u^\mathrm{T} E v - \mathbb{E} u^\mathrm{T} E v| \geq t) \leq 2 \exp \left( \frac{-t^2}{8 K^2 } \right). $$
Here we used the fact that
$$ \sum_{1 \leq i < j \leq n} (|u_i| |v_j| + |v_i| |u_j|)^2 + \sum_{i=1}^n |u_i|^2 |v_i|^2 \leq 4 \sum_{i,j=1}^n |u_i|^2 |v_j|^2 \leq 4 $$
because $u,v$ are unit vectors.  Since each $\xi_{ij}$ has mean zero, it follows that $\mathbb{E} u^\mathrm{T} E v = 0$, and the proof is complete.  
\end{proof}

\begin{lemma} \label{lemma:conc-nonsym}
Let $E = (\xi_{ij})_{1 \leq i \leq m, 1 \leq j \leq n}$ be a $m \times n$ real random matrix where
$$ \{ \xi_{ij} : 1 \leq i \leq m, 1 \leq j \leq n \} $$
is a collection of independent random variables each with mean zero.  Further assume
$$ \sup_{1 \leq i \leq m, 1 \leq j \leq n} |\xi_{ij}| \leq K $$
with probability $1$, for some $K \geq 1$.  Then for any fixed unit vectors $u \in \mathbb{R}^m, v \in \mathbb{R}^n$, and every $t > 0$
\begin{equation} \label{eq:concprop-nonsym}
	\mathbb{P}( |u^\mathrm{T} E v| \geq t) \leq 2 \exp \left( \frac{-t^2}{2 K^2} \right). 
\end{equation}
\end{lemma}

The proof of Lemma \ref{lemma:conc-nonsym} is nearly identical to the proof of lemma \ref{lemma:conc-sym}.  Indeed, \eqref{eq:concprop-nonsym} follows from Hoeffding's inequality since $u^\mathrm{T}E v$ can be the written as the sum of independent random variables; we omit the details.  

Many other models of random matrices satisfy Definition \ref{def:concentration}.  If the entries of $E$ are independent and have a rapidly decaying tail, then $E$ will be $(C_1,c_1,\gamma)$-concentrated for some constants $C_1,c_1,\gamma>0$.  One can achieve this by standard truncation arguments.  
For many arguments of this type, see for instance \cite{VW}.  As an example, we present a concentration result from \cite{RV} when the entries of $E$ are iid sub-exponential random variables.

\begin{lemma}[Proposition 5.16 of \cite{RV}] \label{lemma:conc-sub}
Let $E = (\xi_{ij})_{1 \leq i \leq m, 1 \leq j \leq n}$ be a $m \times n$ real random matrix whose entries $\xi_{ij}  $ are iid copies of a sub-exponential random variable $\xi$ with constant $K$, i.e. $\mathbb{P}(|\xi| > t) \le \exp(1-t/K)$ for all $t>0$.  Assume $\xi$ has mean 0 and variance 1. Then there are constants $C_1, c_1>0$ (depending only on $K$) such that for any fixed unit vectors $u \in \mathbb{R}^m, v \in \mathbb{R}^n$ and any $t > 0$, one has
\begin{equation*}
	\mathbb{P}( |u^\mathrm{T} E v| \geq t) \leq C_1 \exp \left( -c_1 t \right). 
\end{equation*}

\end{lemma}



Finally, let us point out that  the assumption that the entries are independent is not necessary. As an example, we mention random orthogonal matrices. 
For another example, one can consider the elliptic ensembles;  this can be verified using standard truncation and concentration results, see for instance \cite{KS, LT, McDiarmid, RV} and \cite[Chapter 5]{BS}.  


\section{An application: The matrix recovery problem}  \label{section:app} 


 The matrix recovery problem  is the following: $A$ is a large unknown matrix.  We can only observe its noisy image  $A+E$, or in some cases just a small part of it.  We would like to reconstruct $A$  or estimate an important parameter as accurately as possible from this observation.  

 Consider a deterministic $m \times n$ matrix $$A = (a_{ij})_{1 \leq i \leq m, 1 \leq j \leq n.}$$  
Let $Z$ be a random matrix of the same size whose entries
 $\{z_{ij} : 1 \leq i \leq m, 1 \leq j \leq n \}$  are independent random variables with mean zero and unit variance.   For convenience, we will assume that $\| Z \| _{\infty} := \max_{i,j} |z_{ij}| \le K$, for some fixed $K >0$, with probability $1$.
 
 
Suppose  that  we have only partial access to the noisy data $A+Z$. Each entry of this matrix is observed with probability $p$ and 
 unobserved with probability $1-p$ for some small $p$.  We will write $0$ if the entry is not observed.  Given this sparse observable data matrix $B$, the  task is to reconstruct $A$. 
 
 The matrix completion problem is a central one in data analysis, and there is a large collection of literature focusing on the low rank case; see  \cite{AM,CCS,CP,CR,CRT,CT,Cest,KMO,KMO2,Krank,KLT,MHT,NW,RVsamp} and references therein.   A representative example here is the Netflix problem, where $A$ is the matrix of ratings (the rows are viewers, the columns are movie titles, and entries are ratings). 
 
 In this section, we are going to use our new results to study this problem.  The main novel feature here is  that our analysis allows us to approximate  {\it any given column (or row)}  with high probability.  For instance, in the  Netflix problem, 
 one can figure out the ratings of any given individual, or any given movie. 
 
  In earlier algorithms we know of, the approximation was mostly  done for the Frobenius norm of the whole matrix.  Such a result is equivalent to saying that a {\it random} row or column is well approximated, but 
  cannot guarantee anything about a specific row or column.  
  
  Finally, let us mention that there are   algorithms which can recover $A$ precisely, but these work only if $A$ satisfies certain structural assumptions \cite{CCS,CP,CR,CRT,CT}. 
 
Without loss of generality,  we assume $A$ is a square  $n \times n$  matrix.  The rectangular case follows by applying the analysis below 
to the matrix $\tilde{A}$ defined in \eqref{eq:def:tildeA}.  We assume that $n$ is large and  
asymptotic notation such as $o, O, \Omega, \Theta$ will be used under the assumption that $n \rightarrow \infty$. 


  Let $A$ be a  $n \times n$ deterministic matrix with rank $r$ where  $\sigma_1 \geq \cdots \geq \sigma_r > 0$ are the singular values  with corresponding singular vectors $u_1, \ldots, u_r$.  
  Let $\chi_{ij}$ be iid indicator random variables with $\mathbb{P} (\chi_{ij}=1)=p$.  The entries of the sparse matrix $B$ can be written as 
  \begin{equation*} b_{ij} = (a_{ij } +z_{ij} ) \chi_{ij} = p a_{ij} + a_{ij} (\chi_{ij} -p) + z_{ij} \chi_{ij} = pa_{ij} + f_{ij}, \end{equation*}  where 
  \begin{equation*} f_{ij} :=  a_{ij} (\chi_{ij} -p) + z_{ij} \chi_{ij} . \end{equation*} It is clear that the $f_{ij}$ are independent random variables with mean 0 and variance 
  $\sigma_{ij}^2 =  a_{ij}^2 p(1-p) + p $.  This way, we can write $\frac{1}{p} B$ in the form $A + E$, where  $E$ is the  random matrix with independent entries $e_{ij} := p^{-1} f_{ij}$.  We assume  $p \le 1/2$; in fact, our result 
  works for $p$ being a negative power of $n$.
  
  
  
  
  
 Let $1 \le j \le r$ and consider  the subspace $U$ spanned by $u_1, \dots, u_j$ and $V$ spanned by $v_1, \dots, v_j$, where $u_i$ (alternatively $v_i)$  is the $i$-th singular vector of 
 $A$ (alternatively $B$).
 Fix any $1 \le m  \le n$ and 
  consider the $m$-th columns
  of $A$ and $A+E$. Denote them by 
  $x$ and $\tilde x $, respectively.  We have 
  \begin{equation*}  \| x- P_{V} \tilde x\| \le \| x - P_U x\| + \| P_U x - P_U \tilde x\| + \| P_U \tilde x - P_V \tilde x \|. \end{equation*} 
  Notice that  $P_V \tilde x $  is efficiently computable given $B$ and $p$.
  (In fact,  we can estimate $p$ very well by the density of $B$, so we don't even need to know $p$.)  In the remaining part of the analysis, we 
 will estimate the three error terms on the right-hand side. 
 
 
 
  We will make use of the following lemma, which is a variant of \cite[Lemma 2.2]{TVdet}; see also \cite{VW} where results of this type are discussed in depth. 
 
 
 \begin{lemma} \label{lemma:projection} Let $X$ be a random vector in $\mathbb{R}^n$ whose coordinates $x_i, 1\le i \le n$  are independent random variables with mean 0, variance at most $\sigma^2$, and are bounded in absolute value by $1$. Let $H$ be a fixed subspace of dimension $d$ and $P_H (X)$ be the projection of $X$ onto $H$. Then 
 \begin{equation} \label{eqn:distance} \mathbb{P} \left(  \| P_H (X) \| \ge \sigma d^{1/2} +  t \right)  \le C  \exp (-c t^2 )  ,\end{equation}  
 where $c, C>0$ are absolute constants. 
 
 \end{lemma} 
 
   
  The first term $\| x -P_U  x \|$ is  bounded from above by $\sigma_{j+1}$.  The second term has the form $\| P_U X \|$, where $X:=x -\tilde x$ is the random vector
  with independent entries, which is the $m$-th column of   $E$.  Notice that entries of $X$ are bounded (in absolute value) by   $\alpha : = p^{-1} (\|x\| _{\infty}+  K)$ with probability $1$. Applying Lemma \ref{lemma:projection} (with the proper normalization),   we obtain 
\begin{equation} \label{recovery3} 
	\mathbb{P} \left(   \| P_U X \| \geq  j^{1/2} \sqrt{ \frac{ \|x\|_{\infty}^2 + 1}{p} }+ t \right)  \le C \exp( -c t^2 \alpha^{-2} ) 
\end{equation} 
since $\sigma_{im}^2 \leq p^{-1} ( \|x\|_{\infty}^2 + 1)$.  
By setting $t := c^{-1/2} \alpha \lambda $,  \eqref{recovery3} implies  that, for any $\lambda > 0$, 
\begin{equation*} \| P_U X \| \le  j^{1/2} \sqrt{ \frac{ \|x\|_{\infty}^2 + 1}{p} }+ c^{-1/2} \lambda \alpha  \end{equation*} 
with probability at least $1-   C \exp(-\lambda^2 ) $.  

To bound  $\| P_U \tilde x - P_V \tilde x \|$, we appeal to Theorem \ref{thm:subspace}. Assume for a moment that $E$ is $(C_1, c_1, \gamma)$-concentrated 
  for some constants $C_1, c_1, \gamma > 0$.  Let $\delta_j := \sigma_j - \sigma_{j+1}$.  Then it follows that, for any $\lambda > 0$, 
\begin{equation*} \| P_U - P_V\| \le C \sqrt{j} \left(\frac{ \lambda^{2/\gamma} r^{1/\gamma} }{ \delta_{j} } + \frac{\|E \| }{\sigma_j} +\frac{\| E\|^2 }{ \sigma_j \delta_j } \right), \end{equation*}  
with probability at least 
$$ 1 - 6C_1 9^j \exp \left( -c_1 \frac{\delta_j^\gamma}{8^\gamma} \right) - 2C_1 9^{2r} \exp \left( -c_1 r \frac{\lambda^2}{4^\gamma} \right), $$
where $C$ is an absolute constant.  
  
Since 
$$ \| P_U \tilde{x} - P_V \tilde{x} \| \leq \|P_U - P_{V} \| \|\tilde{x} \|, $$
it remains to bound $\| \tilde{x} \|$.  We first note that $\|\tilde{x}\| \leq \|x\| + \|X \|$.  By Talagrand's inequality (see \cite{Tconc} or \cite[Theorem 2.1.13]{Tbook}) , we have
$$ \mathbb{P} \left( \|X \| \geq \mathbb{E} \|X\| + t \right) \leq C \exp(-c t^2 \alpha^{-2}). $$
In addition, 
$$ \mathbb{E} \|X\|^2 = \frac{1}{p^2} \sum_{i=1}^n \sigma_{im}^2 \leq \frac{1}{p} \left( \|x\|^2 + n \right). $$
Thus, we conclude that
$$ \|X\| \leq \sqrt{ \frac{\|x\|^2 + n }{p} } + c^{-1/2} \lambda \alpha $$
with probability at least $1 - C \exp(-\lambda^2)$.  

Putting the bounds together, we obtain Theorem \ref{theorem:recovery} below.   
  
  
  \begin{theorem}  \label{theorem:recovery} Assume that $A$ has rank $r$ and $\| Z\|_{\infty} \le K $ with probability $1$. Assume that $E$  is 
  $(C_1, c_1, \gamma)$-concentrated for a trio of constants $C_1, c_1, \gamma >0$.  Let $m$ be an arbitrary index between $1$ and $n$, and let $x$ and $\tilde x$ be the $m$-th columns of $A$ and $\frac{1}{p} B$.  Let $1 \leq j \leq r$ be an integer, and let $V$ be the subspace spanned by the first $j$ singular vectors of $B$.  Let $\sigma_1 \ge \dots \ge \sigma_r > 0$ be the 
  singular values of $A$.   Assume $\delta_j := \sigma_j - \sigma_{j+1}$.  Then, for any $\lambda >0$,  
  \begin{equation*} 
  \| x - P_V (\tilde x) \| \le \sigma_{j+1} +  j^{1/2} \sqrt{ \frac{ \|x\|_{\infty}^2 + 1}{p} } + \mu \left( \sqrt{ \frac{ \|x\|^2 + n}{p} } + C \lambda \alpha \right) + C \lambda \alpha, 
   \end{equation*} 
with probability at least  
$$ 1 - C \exp(-\lambda^2) -  6C_1 9^j \exp \left( -c_1 \frac{\delta_j^\gamma}{8^\gamma} \right) - 2C_1 9^{2r} \exp \left( -c_1 r \frac{\lambda^2}{4^\gamma} \right), $$
  where 
  $$ \alpha := p^{-1} (\| x\| _{\infty} + K) \quad \text{and}\quad \mu:= C  \sqrt{j} \left(\frac{\lambda^{2/\gamma} r^{1/\gamma} }{ \delta_{j} } + \frac{\|E \| }{\sigma_j} +\frac{\| E\|^2 }{ \sigma_j \delta_j } \right), $$
and $C$ is an absolute constant.  
  \end{theorem} 
  
As this theorem is a bit technical, let us consider a special, simpler case. Assume that all entries of $A$ are of order $\Theta(1)$ and $p=\Theta(1)$.   Thus, any column $x$ has length $ \Theta (n^{1/2})$. Assume furthermore that $j=r=\Theta(1)$ and $\sigma_r = \Omega(n^{1/2+\varepsilon})$ for some $\varepsilon > 0$.  Then our analysis yields


\begin{corollary} 
There exists $c_0 > 0$ (depending only on $\varepsilon$) such that, for any given column $x$,  
$$ \| x - P_V (\tilde x) \|  = O(  n^{-c_0} \|x\|) $$
with probability $1-o(1)$. 
\end{corollary}   


\subsection*{Acknowledgements}
The authors would like to thank Nicholas Cook and David Renfrew for useful comments.  
     
   


2014-09-08T02:10:39.000
1311
1311.2657
en
https://arxiv.org/abs/1311.2657
Matrix perturbation inequalities, such as Weyl's theorem (concerning the singular values) and the Davis-Kahan theorem (concerning the singular vectors), play essential roles in quantitative science; in particular, these bounds have found application in data analysis as well as related areas of engineering and computer science. In many situations, the perturbation is assumed to be random, and the original matrix has certain structural properties (such as having low rank). We show that, in this scenario, classical perturbation results, such as Weyl and Davis-Kahan, can be improved significantly. We believe many of our new bounds are close to optimal and also discuss some applications.
Numerical Analysis (math.NA); Combinatorics (math.CO); Probability (math.PR); Statistics Theory (math.ST)
Random perturbation of low rank matrices: Improving classical bounds
Qwen/Qwen-72B
1. YES
2. YES