Derivatives of Schubert polynomials and proof of a determinant conjecture of Stanley

Abstract: We study the action of a differential operator on Schubert polynomials. Using this action, we first give a short new proof of an identity of I. Macdonald (1991). We then prove a determinant conjecture of R. Stanley (2017). This conjecture implies the (strong) Sperner property for the weak order on the symmetric group, a property recently established by C. Gaetz and Y. Gao (2018).

\section{Introduction}

This paper is motivated by a conjecture of R.~Stanley~\cite[Conjecture~2.2]{Stanley:shen}.
Let $\mathcal{S}_n$ be the symmetric group with its standard generating set $S = \{s_1, s_2, \dots, s_{n-1}\}$, and let $\mathcal{S}_n(\ell)$ denote the subset of those permutations of (Coxeter) length $\ell$.
For $1 \leq \ell \leq \binom{n}{2}$, let $M_{\ell}$ be the matrix with rows indexed by $\mathcal{S}_n(\ell-1)$ and columns indexed by $\mathcal{S}_n(\ell)$, where the entry in position $(u,v)$ is
\[
M_\ell [u,v] =
\begin{cases}
k, &\text{if}\enspace v=us_k \enspace\text{and}\\
0, &\text{if $u^{-1} v \notin S$}.
\end{cases}
\]

For $\ell \leq \binom{n}{2}-\ell$, the product $\widetilde{M}^{(\ell)} = M_{\ell+1} M_{\ell+2} \cdots M_{\binom{n}{2} - \ell}$ is a square matrix with rows indexed by $\mathcal{S}_n(\ell)$ and columns indexed by $\mathcal{S}_n \left(\binom{n}{2} - \ell \right)$. Stanley conjectures an explicit formula for $\det \widetilde{M}^{(\ell)}$,  which implies that $\widetilde{M}^{(\ell)}$ is invertible.
A motivation for this conjecture is that $M_{\ell(u)+1} M_{\ell(u)+2} \cdots M_{\ell(v)}[u,v]$ is nonzero if and only if $u \leq v$ in weak order.
Hence, by standard linear-algebraic arguments (cf.~\cite{Stanley:Weyl, Stanley:shen}), showing that $\det \widetilde{M}^{(\ell)}$ is nonzero implies that the weak order on the symmetric group has the (strong)  Sperner property.

Recently, C.~Gaetz and Y.~Gao \cite{Gaetz.Gao} proved the invertibility of Stanley's matrix by constructing an action of the Lie algebra $\sl_2$.
We give a new proof of invertibility by proving Stanley's determinant conjecture.
Our proof also involves an $\sl_2$-representation, but relies on a new identity for derivatives of Schubert polynomials.

The generator $s_i$ acts on $\ensuremath{\mathrm{Poly}}_n = \mathbb{C}[x_1, \ldots, x_n]$ by
\[
s_i \cdot f(x_1, \ldots, x_i, x_{i+1}, \ldots, x_n) = f(x_1, \ldots, x_{i+1}, x_{i}, \ldots, x_n).
\]
We define the Newton divided difference operators on $\ensuremath{\mathrm{Poly}}_n$ by
\[ N_i(f) = \frac{f - s_i \cdot f}{x_{i} - x_{i+1}}. \]
(We avoid the more standard notation $\partial_i$ because of potential confusion with partial derivatives.)
For $w \in \mathcal{S}_n$, the \newword{Schubert polynomials} $\mathfrak{S}_w$ are defined by the recurrence
\[ \mathfrak{S}_{s_k w} = N_k \mathfrak{S}_w \ \mbox{for} \ \ell(s_k w) < \ell(w) \] with $\mathfrak{S}_{w_0} = x_1^{n-1} x_2^{n-2} \cdots x_{n-1}$.
For background on Schubert polynomials, we refer the reader to \cite{Macdonald:notes,Manivel}.

We consider the differential operator $\nabla = \sum_{i=1}^n \der{i}$. Our key result is the following:
\begin{proposition} \label{DerIdentity}
For $w \in \mathcal{S}_n$, we have
\[ \nabla (\mathfrak{S}_w) = \sum_{\ell(w s_k) < \ell(w)} k \mathfrak{S}_{ws_k} . \]
\end{proposition}
Thus, Stanley's $M$-matrices are the matrices of the operator $\nabla$ in the basis of Schubert polynomials.

We first apply Proposition~\ref{DerIdentity} to give a short new proof of a theorem of Macdonald~\cite[(6.11)]{Macdonald:notes}.
We then use Proposition~\ref{DerIdentity} to prove Stanley's conjecture.
\begin{theorem}[{Conjectured by Stanley~\cite[Conjecture~2.2]{Stanley:shen}}] \label{MainTheorem}
For $\ell \leq \binom{n}{2}-\ell$,
\[ \det \widetilde{M}^{(\ell)}  =  \pm \prod_{k=0}^{\ell} \left( (\ell-k+1) (\ell-k+2) \cdots ( \textstyle{\binom{n}{2}} - \ell -k)   \right)^{|\mathcal{S}_n(k)| - |\mathcal{S}_n(k-1)| }. \]
\end{theorem}
We write $\pm$ because we have not specified an order on the rows and columns of each $M_i$.

\section{Proof of Proposition~\ref{DerIdentity} and a Macdonald identity}

We start with a straightforward lemma.
\begin{lemma}\label{lem:commutation}
$\nabla$ commutes with $N_i$ for all $i$. That is, for any $f \in \ensuremath{\mathrm{Poly}}_n$, we have
\[
\nabla(N_i(f)) = N_i(\nabla(f)).
\]
\end{lemma}
\begin{proof}
If $j \notin \{ i, i+1 \}$, then \[
\der{j} N_i(f) =
\frac{1}{x_i - x_{i+1}} \left( \der{j} (f) - \der{j} (s_i \cdot f) \right).
\]

If $j = i$, then
\[
\der{i} N_i(f) = \frac{1}{x_i - x_{i+1}} \left( \der{i} (f) - \der{i} (s_i \cdot f) \right) - \frac{f - s_i \cdot f}{(x_i - x_{i+1})^2}.
\]

Similarly, if $j = i+1$, then
\[
\der{i+1} N_i(f)
= \frac{1}{x_i - x_{i+1}} \left( \der{i+1} (f) - \der{i+1} (s_i \cdot f) \right) + \frac{f - s_i \cdot f}{(x_i - x_{i+1})^2}.
\]
Therefore,
\begin{align*}
\nabla(N_i(f)) &= \frac{f - s_i \cdot f}{(x_i - x_{i+1})^2} - \frac{f - s_i \cdot f}{(x_i - x_{i+1})^2} +   \sum_{j=1}^n \frac{1}{x_i - x_{i+1}} \left( \der{j} (f) - \der{j} (s_i \cdot f) \right) \\
&= \frac{\sum_{j=1}^n \der{j}(f) - \sum_{j=1}^n \der{j}(s_i \cdot f)}{x_i - x_{i+1}}  = \frac{\nabla(f) - \nabla(s_i \cdot f)}{x_i - x_{i+1}} \\
&= \frac{\nabla(f) - s_i \cdot \nabla(f)}{x_i - x_{i+1}} = N_i (\nabla (f)),
\end{align*}
as desired.
\end{proof}

\begin{proof}[Proof of Proposition~\ref{DerIdentity}]
We first verify Proposition~\ref{DerIdentity} in the case that $w = w_0$. Since $\mathfrak{S}_{w_0} = x_1^{n-1} x_2^{n-2} \cdots x_{n-1}$,  we have
\begin{align*}
\nabla(\mathfrak{S}_{w_0}) &= \sum_{j=1}^{n-1} (n-j) x_1^{n-1} x_2^{n-2} \cdots x_{j-1}^{n-j+1} x_j^{n-j-1} x_{j+1}^{n-j-1} \cdots x_{n-1} \\
&= \sum_{k=1}^{n-1} k x_1^{n-1} x_2^{n-2} \cdots x_{n-k-1}^{k+1} x_{n-k}^{k-1} x_{n-k+1}^{k-1} \cdots x_{n-1} .
\end{align*}
 But also
\[ \mathfrak{S}_{w_0 s_{k}} = \mathfrak{S}_{s_{n-k} w_0} = N_{n-k}(\mathfrak{S}_{w_0}) =  x_1^{n-1} x_2^{n-2} \cdots x_{n-k-1}^{k+1} x_{n-k}^{k-1} x_{n-k+1}^{k-1} \cdots x_{n-1} . \]
Comparing these equations gives
\[ \nabla(\mathfrak{S}_{w_0}) = \sum_{k=1}^{n-1} k \mathfrak{S}_{w_0 s_k} . \]

Consider an arbitrary permutation $w$. Let $r = \binom{n}{2} - \ell(w)$ and write $w = s_{i_1} s_{i_2} \cdots s_{i_r} w_0$.
By Lemma~\ref{lem:commutation}, we have
\begin{equation}\label{eq:nabla}\tag{$\star$}
 \nabla(\mathfrak{S}_{w}) = \nabla N_{i_1} N_{i_2} \cdots N_{i_r}(\mathfrak{S}_{w_0}) = N_{i_1} N_{i_2} \cdots N_{i_r} \nabla(\mathfrak{S}_{w_0}) =  N_{i_1} N_{i_2} \cdots N_{i_r}  \sum_{k=1}^{n-1} k \mathfrak{S}_{w_0 s_k} .
 \end{equation}
Hence, \[
N_{i_1} N_{i_2} \cdots N_{i_r} ( \mathfrak{S}_{w_0 s_k} )=
\begin{cases}
\mathfrak{S}_{s_{i_1} \cdots s_{i_r} w_0 s_k}, &\text{if $\ell(w s_k) = \binom{n}{2} - r -1 = \ell(w)-1$} \\
0, & \text{otherwise}.
\end{cases}
\]
Since $\mathfrak{S}_{s_{i_1} \cdots s_{i_r} w_0 s_k} = \mathfrak{S}_{w s_k}$, Equation~\eqref{eq:nabla} then becomes
\[ \nabla (\mathfrak{S}_{w}) = \sum_{\ell(w s_k) = \ell(w)-1} k \mathfrak{S}_{w s_k}, \]
as desired.
\end{proof}

Proposition~\ref{DerIdentity} yields a short proof of an identity of Macdonald~\cite[(6.11)]{Macdonald:notes}. Another proof of this result was given by S.~Fomin and R.~Stanley in terms of nilCoxeter algebras \cite{Fomin.Stanley}, while a bijective proof was given by S.~Billey, A.~Holroyd and B.~Young \cite{Billey.Holroyd.Young}. A \newword{reduced word} for $w \in \mathcal{S}_n$ is a tuple $a = (a_1, \dots, a_{\ell(w)})$ such that $w=s_{a_1} s_{a_2} \cdots s_{a_{\ell(w)}}$. We write $R(w)$ for the set of reduced words of $w$.

\begin{theorem}[{Macdonald~\cite[(6.11)]{Macdonald:notes}}]
Let $w \in \mathcal{S}_n$ with $\ell(w) = k$. Then
\[ \frac{1}{k! } \sum_{a \in R(w)} a_1 a_2 \cdots a_k = \mathfrak{S}_w(1,1,\ldots,1). \]
\end{theorem}

\begin{proof}
For any monomial $\mu$ of degree $k$, we have $\nabla^k (\mu) = k!$.
Since $\mathfrak{S}_w$ is homogenous of degree $k$, we then see $\nabla^k( \mathfrak{S}_w) = k! \mathfrak{S}_w(1,1,\ldots,1)$.
On the other hand, by Proposition~\ref{DerIdentity},
\[ \pushQED{\qed} \nabla^k (\mathfrak{S}_w) =  \sum_{a \in R(w)} a_1 a_2 \cdots a_k. \qedhere \popQED \] \let\qed\relax
\end{proof}

\section{A vector space of polynomials}\label{sec:vectorspace}

Let $W \subset \ensuremath{\mathrm{Poly}}_n$ be the span of the monomials of the form $x_1^{a_1} x_2^{a_2} \cdots x_n^{a_n}$ with $0 \leq a_j \leq n-j$.
(In particular, the exponent of $x_n$ is required to be $0$, so the variable $x_n$ does not occur in any polynomial in $W$.)
Let $W_{\ell}$ be the subspace of $W$ spanned by monomials of degree $\ell$. We will need the following lemma.

\begin{lemma}[{\cite[Proof of Corollary~3.9]{Bergeron.Billey}}] \label{LeadingTerm}
For $w \in \mathcal{S}_n$, the Schubert polynomial $\mathfrak{S}_w$ lies in $W$. If we choose a term order with $x_n > x_{n-1} > \cdots > x_1$, then the leading term of $\mathfrak{S}_w$ is
\[ \pushQED{\qed} \prod_{j=1}^n x_j^{\# \{ k \ : \ k>j,\ w(k) < w(j) \} }.  \qedhere \popQED \]
\end{lemma}

The list of numbers $\# \{ k \ : \ k>j,\ w(k) < w(j) \}$ is the \newword{(Lehmer) code} of $w$; taking the code is a bijection between $\mathcal{S}_n$ and $\{ (a_1, \ldots, a_n) \in \mathbb{Z}^n : 0 \leq a_j \leq n-j \}$ (see, e.g., \cite[Proposition~2.1.2]{Manivel}). Thus, Lemma~\ref{LeadingTerm} implies that the Schubert polynomials have distinct leading terms and we deduce:
\begin{cor} \label{ChangeBasis}
The Schubert polynomials $\mathfrak{S}_w$ for $w \in \mathcal{S}_n$ are a basis for $W$. The change of basis matrix between $\{ \mathfrak{S}_w : \ell(w) = k \}$ and $\{ x_1^{a_1} x_2^{a_2} \cdots x_n^{a_n} \ : \ 0 \leq a_j \leq n-j,\ \sum a_j =k \}$ has determinant $\pm 1$. \qed
\end{cor}

Proposition~\ref{DerIdentity} shows that $\nabla : W_{\ell} \to W_{\ell-1}$, in the Schubert basis, is represented by the matrix $M_{\ell}$. Therefore, to prove Theorem~\ref{MainTheorem}, we must compute
\[ \det\left( W_{\binom{n}{2} - \ell} \xrightarrow{\ \nabla^{\binom{n}{2} - 2 \ell}\ } W_{\ell} \right) \]
in the Schubert basis. By Corollary~\ref{ChangeBasis}, we may compute this determinant instead in the monomial basis.
For the remainder of this note, Schubert polynomials disappear and our goal is to compute the determinant of $\nabla^j$ acting with respect to the monomial basis.

We would prefer to have a map from a vector space to itself, so that we could speak of its determinant without any reference to bases.
There is a simple bijection between the monomial bases of $W_k$ and $W_{\binom{n}{2} - k}$, taking $\prod x_j^{a_j}$ to $\prod x_j^{n-j-a_j}$.
For reasons that will become clear in Section~\ref{sec:sl2_reps}, we prefer to twist this map by $(-1)^k$, so we define $J$ to be the linear endomorphism of $W$ with
\[ J\left( \prod x_j^{a_j} \right) = (-1)^{\sum a_j} \prod x_j^{n-j-a_j} . \]
Note that $J$ has determinant $\pm 1$ in the monomial basis and that $\nabla^{\binom{n}{2} - 2\ell} \circ J$ maps $W_{\ell}$ to itself. Thus, to finish our proof of Theorem~\ref{MainTheorem}, it remains to establish
\[ \det \left( W_{\ell} \xrightarrow{\ \nabla^{\binom{n}{2} - 2\ell} \circ J \ } W_{\ell} \right) =   \pm \prod_{k=0}^{\ell} \left( (\ell-k+1) (\ell-k+2) \cdots ( \textstyle{\binom{n}{2}} - \ell -k)   \right)^{|\mathcal{S}_n(k)| - |\mathcal{S}_n(k-1)| },\]
a statement that makes no reference to bases. We now turn to this task.

\section{$SL_2$-representations and a proof of Theorem~\ref{MainTheorem}}\label{sec:sl2_reps}

In this section, we discuss some representations of the Lie group $\mathrm{SL}_2$ and its Lie algebra $\sl_2$. We will denote group actions by variants of the letter $\rho$ and the
corresponding Lie algebra actions by variants of $\sigma$. We write the standard basis of $\sl_2$ as
\[ F = \begin{bmatrix} 0&0\\1&0 \end{bmatrix} \qquad H = \begin{bmatrix} 1&0\\0&-1 \end{bmatrix} \qquad E = \begin{bmatrix} 0&1\\0&0 \end{bmatrix} \]
and define the element
\[ J = \begin{bmatrix} 0&1 \\ -1 & 0 \end{bmatrix} \in \mathrm{SL}_2.\]

Let $V_k$ be the $(k+1)$-dimensional irreducible representation of $\mathrm{SL}_2$ and $\sl_2$; we write $\rho_k$ and $\sigma_k$ for the action maps $\rho_k : \mathrm{SL}_2 \to \mathrm{GL}(V_k)$ and $\sigma_k : \sl_2 \to \mathrm{End}(V_k)$.

One usually describes $V_k$ as the natural action on degree $k$ polynomials in two variables.
For our purposes, it is more convenient to describe $V_k$ as an action on polynomials  of degree $\leq k$ in one variable $x$.
We have
\begin{equation}\tag{$\dagger$}\label{eq:rep}
\begin{array}{rcr@{}l}
\sigma_k(F)(x^j) &=& j &x^{j-1} \\  \sigma_k(H)(x^j) &=& (2j-k) &x^j \\  \sigma_k(E) (x^j) &=& (k-j) &x^{j+1} \\  \rho_k(J)(x^j) &=& (-1)^j &x^{k-j} . \\
\end{array}
\end{equation}
In particular, for any polynomial $f$, we have $\sigma_k(F)(f) = \tfrac{d f}{d x}$.

Identify the vector space $W$ from Section~\ref{sec:vectorspace} with $V_{n-1} \otimes V_{n-2} \otimes \cdots  \otimes V_0$ by identifying $x_1^{a_1} x_2^{a_2} \cdots x_{n}^{a_{n}}$ with $x^{a_1} \otimes x^{a_2} \otimes \cdots \otimes x^{a_{n}}$.
We let $\mathrm{SL}_2$ and $\sl_2$ act on this tensor product in the standard way, and denote these actions by $\sigma_W$ and $\rho_W$.
We note that $W_{\ell}$ is the $2 \ell-\binom{n}{2}$ weight space, i.e.,\ the $2 \ell-\binom{n}{2}$ eigenspace of $\sigma_W(H)$.

We have
\[ \sigma_W(F) = \sum_{k=1}^n \mathrm{Id} \otimes \mathrm{Id} \otimes \cdots \otimes \sigma_{n-k}(F) \otimes \cdots \otimes \mathrm{Id} \]
where $\sigma_{n-k}(F)$ occurs in the $k$-th position. Therefore,
\[ \sigma_W(F) \cdot f = \sum_{k=1}^n \der{k} f  = \nabla f  .\]
Similarly,
\[ \rho_W(J) = \rho_{n-1} \otimes \rho_{n-2} \otimes \cdots \otimes \rho_0, \]
so
\[
\rho_W(J) \cdot \left( \prod_{j=1}^n x_j^{a_j} \right) = \prod_{j=1}^n (-1)^{a_j} x_j^{n-j-a_j} = (-1)^{\sum a_j} \prod_{j=1}^n  x_j^{n-j-a_j} = J \left( \prod_{j=1}^n x_j^{a_j} \right).
\]

Thus, our goal of computing $\det \left( \nabla^{\binom{n}{2} - 2 \ell} \circ J \right)$ on $V_{\ell}$ is the same as computing the determinant of $\sigma_W(F)^{\binom{n}{2} - 2 \ell} \rho_W(J)$ as a map from the $2 \ell - \binom{n}{2}$ weight space of $V_{n-1} \otimes V_{n-2} \otimes \cdots \otimes V_0$ to itself.

This is a standard computation. By comparing dimensions of weight spaces,
\[ V_{n-1} \otimes V_{n-2} \otimes \cdots \otimes V_0 \ \cong \ \bigoplus_{0 \leq k \leq \binom{n}{2} - k} V_{\binom{n}{2}- 2 k}^{\oplus \, |\mathcal{S}_n(k)| - |\mathcal{S}_n(k-1)|} . \]
Thus, Theorem~\ref{MainTheorem} comes down to showing that $\sigma_W(F)^{\binom{n}{2} - 2 \ell} \rho_W(J)$ as a map from the $2 \ell - \binom{n}{2}$ weight space of $V_{\binom{n}{2}- 2 k}$ to itself is $\pm (\ell-k+1) (\ell-k+2) \cdots ( \textstyle{\binom{n}{2}} - \ell -k)$.
Consulting the formulas from Equation~\eqref{eq:rep}, one sees that
\[ \rho_{\binom{n}{2} - 2k}(J) \ x^{\ell-k} = (-1)^{\ell-k} x^{\binom{n}{2} -2k -(\ell-k)} = (-1)^{\ell-k} x^{\binom{n}{2} - \ell - k }\]
and
\[ \sigma_{\binom{n}{2}-2k}(F)^{\binom{n}{2} - 2 \ell} x^{\binom{n}{2} - \ell - k } = \left(\tfrac{d}{dx} \right)^{\binom{n}{2}-2 \ell}  x^{\binom{n}{2} - \ell - k } = (\ell-k+1) (\ell-k+2) \cdots ( \textstyle{\binom{n}{2}} - \ell -k) x^{\ell-k} .\]
Theorem~\ref{MainTheorem} follows. \qed

\section*{Acknowledgments}
We are grateful for helpful conversations with Ben Elias, Sergey Fomin and Benjamin Young.

OP was partially supported by a Mathematical Sciences Postdoctoral Research Fellowship (\#1703696) from the National Science
Foundation.
DES was partially supported by NSF Grant DMS-1600223.

\bibliographystyle{amsalpha}


Abstract: Given a matrix of observed data, Principal Components Analysis (PCA) computes a small number of orthogonal directions that contain most of its variability. Provably accurate solutions for PCA have been in use for a long time. However, to the best of our knowledge, all existing theoretical guarantees for it assume that the data and the corrupting noise are mutually independent, or at least uncorrelated. This is valid in practice often, but not always. In this paper, we study the PCA problem in the setting where the data and noise can be correlated. Such noise is often also referred to as "data-dependent noise". We obtain a correctness result for the standard eigenvalue decomposition (EVD) based solution to PCA under simple assumptions on the data-noise correlation. We also develop and analyze a generalization of EVD, cluster-EVD, that improves upon EVD in certain regimes.

\subsubsection{#1}  \vspace{-0.1in} }
\newcommand{ \vspace{-0.02in} \item \vspace{-0.02in} } %\newcommand{\Item}{ \vspace{-0.05in} \Item }{ \vspace{-0.02in} \item \vspace{-0.02in} }

\newcommand{{\bm{I}_{\mathcal{T}_t}}}{{\bm{I}_{\mathcal{T}_t}}}
\newcommand{[(\bm{\Phi}_{t})_{\mathcal{T}_t}{}'(\bm{\Phi}_{t})_{\mathcal{T}_t}]^{-1}}{[(\bm{\Phi}_{t})_{\mathcal{T}_t}{}'(\bm{\Phi}_{t})_{\mathcal{T}_t}]^{-1}}
\newcommand{{{\tau}}}{{{\tau}}}
\renewcommand{\tt}{{{t}}}
\newcommand{\epsilon_2}{\epsilon_2}
\newcommand{\epsilon}{\epsilon}

\newcommand{\bm{M}}{\bm{M}}
\newcommand{\bm{N}}{\bm{N}}
\newcommand{\mathrm{cur}}{\mathrm{cur}}
\newcommand{\mathrm{undet}}{\mathrm{undet}}
\newcommand{\mathrm{det}}{\mathrm{det}}
\newcommand{\mathrm{rest}}{\mathrm{rest}}
\newcommand{\mathrm{term}}{\mathrm{term}}
\newcommand{\bm\Lambda_{\dett}}   %{\bm\Lambda_{t,\dett}}{\bm\Lambda_{\mathrm{det}}}
\newcommand{\bm\Lambda_{\cur}} %{\bm\Lambda_{t,\cur}}{\bm\Lambda_{\mathrm{cur}}}
\newcommand{\bm\Lambda_{\undet}} %{\bm\Lambda_{t,\undet}}{\bm\Lambda_{\mathrm{undet}}}
\newcommand{\bm{G}}{\bm{G}}
\newcommand{\bm{\hat{G}}}{\bm{\hat{G}}}
\newcommand{\mathcal{G}}{\mathcal{G}}
\newcommand{\hat{\mathcal{G}}}{\hat{\mathcal{G}}}
\newcommand{\beta}{\beta}
\newcommand{\tilde\beta}{\tilde\beta}
\newcommand{\cM}{\hat{\bm{D}}}  % {\mathcal{\bm{M}}}
\newcommand{}{}
\newcommand{m}{m}
\newcommand{\twt}{\tilde{\bm{w}_t}}

\newcommand{\Section}[1]{ \vspace{-0.15in}  \section{#1} \vspace{-0.16in} }
\newcommand{\Subsection}[1]{ \vspace{-0.15in} \subsection{#1} \vspace{-0.12in}}
\renewcommand{\subsubsection}[1]{{\bf #1. }}

\begin{document}

\title{Correlated-PCA: Principal Components' Analysis when Data and Noise are Correlated

\author{Namrata Vaswani and Han Guo \\ Iowa State University, Ames, IA, USA
\\ Email: \{namrata,hanguo\}@iastate.edu
}

\maketitle

\newcommand{\text{simple-EVD}}{\text{simple-EVD}}
\newcommand{\text{PCA}(2\alpha)}{\text{PCA}(2\alpha)}

\begin{abstract}
Given a matrix of observed data, Principal Components Analysis (PCA) computes a small number of orthogonal directions that contain most of its variability. Provably accurate solutions for PCA have been in use for a long time. However, to the best of our knowledge, all existing theoretical guarantees for it assume that the data and the corrupting noise  are mutually independent, or at least uncorrelated. This is valid in practice often, but not always. In this paper, we study the PCA problem in the setting where the data and noise can be correlated. Such noise is often also referred to as ``data-dependent noise". We obtain a correctness result for the standard eigenvalue decomposition (EVD) based solution to PCA under simple assumptions on the data-noise correlation.
We also develop and analyze a generalization of EVD, cluster-EVD, that improves upon EVD in certain regimes
\end{abstract}

\Section{Introduction}

Principal Components Analysis (PCA) is among the most frequently used tools for dimension reduction. Given a matrix of data, it computes a small number of orthogonal directions that contain all (or most) of the variability of the data. The subspace spanned by these directions is the ``principal subspace". To use PCA for dimension reduction, one projects the observed data onto this subspace.
The standard solution to PCA is to compute the reduced singular value decomposition (SVD) of the data matrix, or, equivalently, to compute the reduced eigenvalue decomposition (EVD) of the empirical covariance matrix of the data. If all eigenvalues are nonzero, a threshold is used and all eigenvectors with eigenvalues above the threshold are retained.
This solution, which we henceforth refer to as {\em simple EVD}, or just {\em EVD}, has been used for many decades and is well-studied in literature, e.g., see \cite{nadler} and references therein.
However, to the best of our knowledge, all existing results for it assume that the true data and the corrupting noise in the observed data are independent, or, at least, uncorrelated. This is valid in practice often, but not always. Here, we study the PCA problem in the setting where the data and noise vectors may be correlated (correlated-PCA). Such noise is sometimes called ``data-dependent" noise.

{\bf Contributions. }
(1) Under a boundedness assumption on the true data vectors, and some other assumptions,  for a fixed desired subspace error level, we show that the sample complexity of simple-EVD for correlated-PCA scales as $f^2 r^2 \log n$ where $n$ is the data vector length, $f$ is the condition number of the true data covariance matrix and $r$ is its rank. Here ``sample complexity" refers to the number of samples needed to get a  small enough subspace recovery error with high probability (whp).
The dependence on $f^2$ is problematic for datasets with large condition numbers, and, especially in the high dimensional setting where $n$ is large.
(2) To address this, we also develop and analyze a generalization of simple-EVD, called {\em cluster-EVD}. Under an eigenvalues' ``clustering" assumption, cluster-EVD weakens the dependence on $f$.

To our best knowledge, the correlated-PCA problem has not been explicitly studied.
We first encountered it while solving the dynamic robust PCA problem in the Recursive Projected Compressive Sensing (ReProCS) framework \cite{rrpcp_allerton,rrpcp_perf,rrpcp_isit15,rrpcp_aistats}. The version of correlated-PCA studied here is motivated by these works.
Some other somewhat related recent works include \cite{sebro_PCA,shamir_PCA} that study stochastic optimization based techniques for PCA; and \cite{onlinePCA,onlinePCA1,onlinePCA2,onlinePCA3} that study online PCA

{\bf Notation. }
We use the interval notation $[a, b]$ to mean all of the integers between $a$ and $b$, inclusive, and similarly for $[a,b)$ etc.
We use $\|\cdot \|$ to denote the $l_2$ norm of a vector or the induced $l_2$ norm of a matrix. For other $l_p$ norms, we use $\|\cdot \|_p$. For a set $\mathcal{T}$, $\bm{I}_{\mathcal{T}}$ refers to an $n \times |\mathcal{T}|$ matrix of columns of the identity matrix indexed by entries in $\mathcal{T}$. For a matrix $\bm{A}$,  $\bm{A}_{\mathcal{T}} := \bm{AI}_{\mathcal{T}}$. A tall matrix with orthonormal columns is referred to as a {\em basis matrix}. For basis matrices $\hat{\bm{P}}$ and $\P$, we  quantify the subspace error (SE) between their range spaces using
\begin{equation}
\mathrm{SE}(\hat{\bm{P}},\P):= \|(\bm{I} - \hat{\bm{P}} \hat{\bm{P}}') \P \|.
\label{SE_def}
\end{equation}

\Subsection{Correlated-PCA: Problem Definition}\label{probdef}

We are given a time sequence of data vectors, $\bm{y}_t$, that satisfy
\begin{eqnarray}  \label{yt_mod}
\bm{y}_t = \bm{\ell}_t + \bm{w}_t, \text{ with } \bm{w}_t = \bm{M}_t \bm{\ell}_t  \text{ and } \bm{\ell}_t = \P \bm{a}_t
\end{eqnarray}
where $\P$ is an $n \times r$ basis matrix with $r \ll n$. Here $\bm{\ell}_t$ is the true data vector that lies in a low dimensional subspace of $\mathbb{R}^n$, $\operatorname{range}(\P)$; $\bm{a}_t$ is its projection into this $r$-dimensional subspace; and $\bm{w}_t$ is the data-dependent noise. We refer to $\bm{M}_t$ as the correlation / data-dependency matrix.  The {\em goal} is to estimate $\operatorname{range}(\P)$. We make the following assumptions on $\bm{\ell}_t$ and $\bm{M}_t$.

\begin{sigmodel
\label{lt_mod}
The subspace projection coefficients, $\bm{a}_t$, are zero mean, mutually independent and bounded random vectors (r.v.), with a diagonal covariance matrix $\bm{\Lambda}$. Define $\lambda^- :=\lambda_{\min}(\bm{\Lambda})$, $\lambda^+:=\lambda_{\max}(\bm{\Lambda})$ and $f:=\frac{\lambda^+}{\lambda^-}$.
Since the $\bm{a}_t$'s are bounded, we can also define a finite constant
$
\eta:=\max_{j=1,2, \dots r}  \max_t \frac{( \bm{a}_t)_j^2}{\lambda_j}.
$
Thus, $(\bm{a}_t)_j^2 \le \eta \lambda_j$.
\end{sigmodel}
For most bounded distributions, $\eta$ will be a small constant more than one, e.g., if the distribution of all entries of $\bm{a}_t$ is iid zero mean uniform, then $\eta =3$.
From Assumption \ref{lt_mod}, clearly, the $\bm{\ell}_t$'s are also zero mean, bounded, and mutually independent r.v.'s with a rank $r$ covariance matrix $\bm{\Sigma} \overset{\mathrm{EVD}}{=} \P \bm{\Lambda} \P'$.
In the model, for simplicity, we assume $\bm{\Lambda}$ to be fixed. However, even if we replace $\bm{\Lambda}$ by $\bm{\Lambda}_t$ and define $\lambda^-  = \min_t \lambda_{\min}(\bm{\Lambda}_t)$ and  $\lambda^+=\lambda_{\max}(\bm{\Lambda}_t)$,  all our results will still hold.

\begin{sigmodel
\label{Mt_model}
Decompose $\bm{M}_t$ as $\bm{M}_t = \bm{M}_{2,t} \bm{M}_{1,t}$. Assume that
\begin{eqnarray}
\|{\bm{M}_{1,t}} \P \| \le q < 1, \ \|{\bm{M}_{2,t}}\| \le 1,
\label{simp_bnd}
\end{eqnarray}
and, for any sequence of positive semi-definite Hermitian matrices, $\bm{A}_t$, the following holds
\begin{eqnarray}
\text{for a $\beta < \alpha$, } \left\|\frac{1}{\alpha} \sum_{t =1}^\alpha {\bm{M}_{2,t}} \bm{A}_t {\bm{M}_{2,t}}' \right\|  \le {\frac{\beta}{\alpha}} \max_{t \in [1,\alpha]} \|\bm{A}_t\|.
\label{M2t_bnd}
\end{eqnarray}
\end{sigmodel}
We will need the above to hold for all $\alpha \ge \alpha_0$ and for all $\beta \le c_0 \alpha$ with a $c_0 \ll 1$. We set $\alpha_0$ and $c_0$ in Theorems \ref{thm1_pca} and \ref{thm1}; both will depend on $q$.
Observe that, using \eqref{simp_bnd}, $\frac{\|\bm{w}_t\|}{\|\bm{\ell}_t\|} \le q$, and so {\em $q$ is an upper bound on the signal-to-noise ratio (SNR).}

To understand the assumption on $\bm{M}_{2,t}$, notice that, if we allow $\beta = \alpha$, then \eqref{M2t_bnd} always holds and is not an assumption. Let $\bm{B}$ denote the matrix on the LHS of \eqref{M2t_bnd}. One example situation when \eqref{M2t_bnd} holds with a $\beta \ll \alpha$ is if $\bm{B}$ is block-diagonal with blocks $\bm{A}_t$. In this case, it holds with $\beta = 1$. In fact, it also holds with $\beta=1$ if $\bm{B}$ is permutation-similar to a block diagonal matrix. The matrix $\bm{B}$ will be of this form if $\bm{M}_{2,t} = \bm{I}_{\mathcal{T}_t}$ with all the sets $\mathcal{T}_t$ being mutually disjoint.
More generally, if $\bm{B}$ is permutation-similar to a block-diagonal matrix with blocks given by the summation of $\bm{A}_t$'s over at most $\beta_0 < \alpha$ time instants, then \eqref{M2t_bnd} holds with $\beta = \beta_0$.  This will happen if $\bm{M}_{2,t} = \bm{I}_{\mathcal{T}_t}$ with $\mathcal{T}_t = \mathcal{T}^{[k]}$ for at most $\beta_0$ time instants and if  sets $\mathcal{T}^{[k]}$ are mutually disjoint for different $k$. Finally, the $\mathcal{T}^{[k]}$'s need not even be mutually disjoint. As long as they are such that $\bm{B}$ is a matrix with nonzero blocks on only the main diagonal and on a few diagonals near it, e.g., if it is block tri-diagonal, it can be shown that the above assumption holds. This example is generalized in  Assumption \ref{sbyrho} given below.

\Subsection{Examples of correlated-PCA problems} \label{examples}

One key example of correlated-PCA is the {\em PCA with missing data (PCA-missing)} problem. Let $\mathcal{T}_t$ denote the set of missing entries at time $t$. Suppose, we set the missing entries of $\bm{y}_t$ to zero. Then
\begin{eqnarray}  \label{MC_mod}
\bm{y}_t = \bm{\ell}_t - \bm{I}_{\mathcal{T}_t} \bm{I}_{\mathcal{T}_t}{}' \bm{\ell}_t.
\end{eqnarray}
In this case $\bm{M}_{2,t}= \bm{I}_{\mathcal{T}_t}$ and $\bm{M}_{1,t} = -\bm{I}_{\mathcal{T}_t}{}'$. Thus, $q$ is an upper bound on $ \| \bm{I}_{\mathcal{T}_t}{}' \P \|$. Clearly, it will be small if the columns of $\P$ are dense vectors.
For the reader familiar with low-rank matrix completion (MC), e.g., \cite{matcomp_first,matcomp_candes}, {\em PCA-missing} can also be solved by first solving the low-rank matrix completion problem to recover $\L$, followed by PCA on the completed matrix. This would, of course, be much more expensive than directly solving {\em PCA-missing} and would need more assumptions.

Another example where correlated-PCA occurs is that of robust PCA (low-rank + sparse formulation) \cite{rpca,rpca2,rpca_zhang} when the sparse component's magnitude is correlated with $\bm{\ell}_t$. Let $\mathcal{T}_t$ denote the support set of $\bm{w}_t$ and let $\bm{x}_t$ be the $|\mathcal{T}_t|$-length vector of its nonzero entries. If we assume linear dependency of $\bm{x}_t$ on $\bm{\ell}_t$, we can write out $\bm{y}_t$ as
\begin{eqnarray}  \label{RPCA_mod}
\bm{y}_t = \bm{\ell}_t + \bm{I}_{\mathcal{T}_t} \bm{x}_t = \bm{\ell}_t + \bm{I}_{\mathcal{T}_t} \bm{M}_{s,t} \bm{\ell}_t.
\end{eqnarray}
Thus $\bm{M}_{2,t}= \bm{I}_{\mathcal{T}_t}$ and $\bm{M}_{1,t} = \bm{M}_{s,t}$ and so $q$ is an upper bound on $\|\bm{M}_{s,t} \P\|$.
In the rest of the paper, we refer to this problem is {\em ``PCA with sparse data-dependent corruptions (PCA-SDDC)".}
One key application where it occurs is in foreground-background separation for videos consisting of a slow changing background sequence (modeled as lying close to a low-dimensional subspace) and a sparse foreground image sequence consisting typically of one or more moving objects \cite{rpca}. The PCA-SDDC problem is to estimate the background sequence's subspace.
In this case, $\bm{\ell}_t$ is the background image at time $t$, $\mathcal{T}_t$  is the support set of the foreground image at $t$, and $\bm{x}_t$ is the difference between foreground and background intensities on $\mathcal{T}_t$.
An alternative solution approach for PCA-SDDC is to use an RPCA solution such as principal components' pursuit (PCP) \cite{rpca,rpca2} or Alternating-Minimization (Alt-Min-RPCA) \cite{robpca_nonconvex} to first recover the matrix $\L$ followed by PCA on $\L$. However, as shown in Sec. \ref{sims}, Table \ref{table_real}, this approach will be much slower; and it will work only if its required incoherence assumptions hold. For example, if the columns of $\P$ are sparse, it fails

For both problems above, a solution for PCA will work only when the {\em corrupting noise $\bm{w}_t$ is small compared to $\bm{\ell}_t$.} A sufficient condition for this is that $q$ is small.

A third example where correlated-PCA and its generalization, correlated-PCA with partial subspace knowledge, occurs is in the subspace update step of Recursive Projected Compressive Sensing (ReProCS) for dynamic robust PCA \cite{rrpcp_perf,rrpcp_aistats}. We refer the reader to \cite{long_simple_proof} to understand this application.

In all three of the above applications, the assumptions on the data-noise correlation matrix given in Assumption \ref{Mt_model} hold if there are enough changes of a certain type in the set of missing or corrupted entries, $\mathcal{T}_t$. One example where this is true is in case of a 1D object of length $s$ or less that remains static for at most $\beta$ frames at a time. When it moves, it moves by at least a certain fraction of $s$ pixels. The following assumption is inspired by the object's support.
\begin{sigmodel}
\label{sbyrho}
Let $l$ denote the number of times the set $\mathcal{T}_t$ changes in the interval $[1,\alpha]$ (or in any given interval of length $\alpha$ in case of dynamic robust PCA). So $0 \le l \le \alpha-1$.
Let $t^0:=1$; let $t^k$, with $t^k < t^{k+1}$, denote the time instants in this interval at which $\mathcal{T}_t$ changes; and let $\mathcal{T}^{[k]}$ denote the distinct sets. In other words, $\mathcal{T}_t = \mathcal{T}^{[k]}$ for $t \in [t^k, t^{k+1})$, for each $k=1,2, \dots, l$.
Assume that the following hold with a $\beta < \alpha$:
\begin{enumerate}
\item \label{moves_often} $(t^{k+1} - t^k) \le \tilde\beta$ and $|\mathcal{T}^{[k]}| \le s$;
\item \label{moves_enough} ${\rho}^2 \tilde\beta \leq \beta$ where $\rho$ is the smallest positive integer so that, for any $0 \le k \le l$,  $\mathcal{T}^{[k]}$ and $\mathcal{T}^{[k+\rho]}$ are disjoint;
\item \label{disjoint_motion} for any $k_1, k_2$ satisfying $0 \le k_1 < k_2 \le l$, the sets $(\mathcal{T}^{[k_1]} \setminus \mathcal{T}^{[k_1+1]})$ and  $(\mathcal{T}^{[k_2]} \setminus \mathcal{T}^{[k_2+1]})$ are disjoint.
\end{enumerate}
An implicit assumption for condition \ref{disjoint_motion} to hold is that $\sum_{k=0}^l |\mathcal{T}^{[k]} \setminus \mathcal{T}^{[k+1]}| \le n$.
Observe that conditions \ref{moves_enough} and \ref{disjoint_motion} enforce an upper bound on the maximum support size $s$.
\end{sigmodel}
To connect Assumption \ref{sbyrho} with the moving object example given above, condition \ref{moves_often} holds if the object's size is at most $s$ and if it moves at least once every $\tilde\beta$ frames. Condition \ref{moves_enough} holds, if, every time it moves, it moves in the same direction and by at least $\frac{s}{\rho}$ pixels.
Condition \ref{disjoint_motion} holds if, every time it moves, it moves in the same direction and by at most $d_0 \ge \frac{s}{\rho}$ pixels, with $d_0 \alpha \le n$ (or, more generally, the motion is such that, if the object were to move at each frame, and if it started at the top of the frame, it does not reach the bottom of the frame in a time interval of length $\alpha$).

The following lemma \cite{rrpcp_isit15} shows that, with Assumption \ref{sbyrho} on $\mathcal{T}_t$, $\bm{M}_{2,t} = \bm{I}_{\mathcal{T}_t}$ satisfies the assumption on $\bm{M}_{2,t}$ given in Assumption \ref{Mt_model}. Its proof generalizes the discussion below Assumption \ref{Mt_model}.

\begin{lem}\label{blockdiag1}[{\cite{rrpcp_isit15}, Lemmas 5.2 and 5.3}]
Assume that Assumption \ref{sbyrho} holds.
For any sequence of $|\mathcal{T}_t| \times |\mathcal{T}_t|$ symmetric positive-semi-definite matrices $\bm{A}_t$,
\begin{align*}
\|  \sum_{t=1}^\alpha \bm{I}_{\mathcal{T}_t} \bm{A}_t {\bm{I}_{\mathcal{T}_t}}' \| &\leq  (\rho^2 \tilde\beta) \max_{t \in [1,\alpha]} \| \bm{A}_t\| \le  \beta \max_{t \in [1,\alpha]} \| \bm{A}_t\|
\end{align*}
Thus, if $\|\bm{I}_{\mathcal{T}_t}{}' \P\| \le q < 1$, then the PCA-missing problem satisfies Assumption \ref{Mt_model}. If $\|\bm{M}_{s,t} \P\| \le q < 1$, then the PCA-SDDC problem satisfies Assumption \ref{Mt_model}.
\end{lem}

Assumption \ref{sbyrho} is one  model on $\mathcal{T}_t$ that ensures that, if $\bm{M}_{2,t} = \bm{I}_{\mathcal{T}_t}$, the assumption on $\bm{M}_{2,t}$ given in Assumption \ref{Mt_model} holds. For its many generalizations, see Supplementary Material, Sec. \ref{more_examp}, or \cite{rrpcp_isit15}.

\Section{Simple EVD} \label{simple_evd}

Simple EVD computes the top eigenvectors of the empirical covariance matrix, $\frac{1}{\alpha}\sum_{t=1}^\alpha \bm{y}_t \bm{y}_t{}'$, of the observed data.
The following can be shown.

\begin{theorem}[simple-EVD result] \label{thm1_pca}
Let $\hat{\bm{P}}$ denote the matrix containing all the eigenvectors of $\frac{1}{\alpha}\sum_{t=1}^\alpha \bm{y}_t \bm{y}_t{}'$ with eigenvalues above a threshold, $\lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^-$, as its columns.
Pick a $\zeta$ so that $r \zeta \le 0.01$.
Suppose that $\bm{y}_t$'s satisfy \eqref{yt_mod} and the following hold.
\begin{enumerate}
\item  Assumption \ref{lt_mod} on $\bm{\ell}_t$ holds.
Define
\[
\alpha_0:=C \eta^2 \frac{r^2 11
\log n }{ ( r \zeta)^2}  \max ( f, q f, q^2f )^2, \ C:= \frac{32 }{ 0.01^2}.
\]

\item  Assumption \ref{Mt_model} on $\bm{M}_t$ holds for any $\alpha \ge \alpha_0$ and for any $\beta$ satisfying
\[
\frac{\beta}{\alpha} \le  \left( \frac{1 - r\zeta}{2} \right)^2 \min \left( \frac{ (r \zeta)^2 }{4.1 (q f)^2} , \frac{ (r \zeta) }{q^2 f} \right)
\]

\item  Set algorithm parameters $\lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^- = 0.95 \lambda^-$ and $\alpha \ge \alpha_0$.

\end{enumerate}
Then, with probability at least  $1-6n^{-10}$,
$
\mathrm{SE}(\hat{\bm{P}},\P) \le  r \zeta.
$%
\end{theorem}
{\em Proof: } The proof involves a careful application of the $\sin \theta$ theorem \cite{davis_kahan} to bound the subspace error, followed by using matrix Hoeffding \cite{tail_bound} to obtain high probability bounds on each of the terms in the $\sin \theta$ bound. It is given in the Supplementary Material, Section \ref{proof_thm1_pca}.

Consider the lower bound on $\alpha$. We refer to this as the ``sample complexity".
Since $q < 1$, and $\eta$ is a small constant (e.g., for the uniform distribution, $\eta=3$), for a fixed error level, $r\zeta$, $\alpha_0$ simplifies to $c f^2 r^2 \log n$. Notice that the dependence on $n$ is logarithmic. It is possible to show that the sample complexity scales as $\log n$ because we assume that the $\bm{\ell}_t$'s are bounded r.v.s. As a result we can apply the matrix Hoeffding inequality \cite{tail_bound} to bound the perturbation between the observed data's empirical covariance matrix and that of the true data.
The bounded r.v. assumption is actually a more practical one than the usual Gaussian assumption since most sources of data have finite power.

By replacing matrix Hoeffding by Theorem 5.39 of \cite{vershynin} in places where one can apply a concentration of measure result to $\sum_t \bm{a}_t \bm{a}_t{}' / \alpha$ (which is at $r \times r$ matrix), and by matrix Bernstein \cite{tail_bound} elsewhere, it should be possible to further reduce the sample complexity to $c \max( (q f)^2 r \log n, f^2 (r + \log n))$.
It should also be possible remove the boundedness assumption and replace it by a Gaussian (or a sub-Gaussian) assumption, but, that would increase the sample complexity to $c (qf)^2 n$.

Consider the upper bound on $\beta/\alpha$. Clearly, the smaller term is the first one. This depends on $1/(q f)^2$. Thus, when $f$ is large and $q$ is not small enough, the bound required may be impractically small. As will be evident from the proof (see Remark \ref{cor_wt} in Supplementary Material), we get this bound because $\bm{w}_t$ is correlated with $\bm{\ell}_t$ and this results in $\mathbb{E}[\bm{\ell}_t \bm{w}_t{}'] \neq 0$.

If $\bm{w}_t$ and $\bm{\ell}_t$ were uncorrelated, $q f$ would get replaced by $\frac{\lambda_{\max}(\operatorname{Cov}(\bm{w}_t))}{\lambda^-}$ in the upper bound on $\beta/\alpha$ as well as in the sample complexity.

\subsubsection{Application to PCA-missing and PCA-SDDC}
By Lemma \ref{blockdiag1}, the following is immediate
\begin{corollary} \label{cor_missing}
Consider the PCA-missing model, \eqref{MC_mod}, and assume that $\max_t \| \bm{I}_{\mathcal{T}_t}{}' \P \| \le q < 1$; or consider the PCA-SDDC model, \eqref{RPCA_mod}, and assume that $\max_t \| \bm{M}_{s,t} \P\| \le q < 1$.
Assume that everything in Theorem \ref{thm1_pca} holds except that we replace Assumption \ref{Mt_model} by Assumption \ref{sbyrho}.
Then, with probability at least $1-6n^{-10}$, $\mathrm{SE}(\hat{\bm{P}},\P) \le r \zeta$.
\end{corollary}

\Section{Cluster-EVD}  \label{cluster_evd}
To try to relax the strong dependence on $f^2$ of the result above, we develop a generalization of simple-EVD that we call {\em cluster-EVD}. This requires the clustering assumption

\newcommand{\hat{r}}{\hat{r}}

\Subsection{Clustering assumption}\label{cluster_evd_assum}
To state the assumption, define the following partition of the index set $\{1,2, \dots r \}$ based on the eigenvalues of $\bm\Sigma$.
Let $\lambda_i$ denote its $i$-th largest eigenvalue.

\begin{definition}[$g$-condition-number partition of $\{1,2, \dots r \}$]
\label{def_part}
Define $\mathcal{G}_1 = \{1,2, \dots r_1\}$ where $r_1$ is the index for which $\frac{\lambda_1}{\lambda_{r_1}} \le g$  and $\frac{\lambda_1}{\lambda_{r_1+1}} > g$. In words, to define  $\mathcal{G}_{1}$, start with the index of the first (largest) eigenvalue and keep adding indices of the smaller eigenvalues to the set until the ratio of the maximum to the minimum eigenvalue first exceeds $g$.

For each $k>1$, define $\mathcal{G}_k = \{r_*+1,r_*+2, \dots, r_*+ r_k \}$ where $r_* = (\sum_{i=1}^{k-1} r_i)$,
 $r_k$ is the index for which $\frac{\lambda_{r_*+1}}{\lambda_{r_* + r_k}} \le g$  and $\frac{\lambda_{r_*+1}}{\lambda_{r_* + r_k+1}} > g$. In words, to define $\mathcal{G}_k$, start with the index of the $(r_*+1)$-th eigenvalue, and repeat the above procedure.

Stop when $\lambda_{r_* + r_k+1} = 0$, i.e., when there are no more nonzero eigenvalues. Define $\vartheta=k$ as the number of sets in the partition. Thus $\{ \mathcal{G}_{1}, \mathcal{G}_{2}, \dots , \mathcal{G}_{\vartheta} \}$ is the desired partition
\end{definition}

Define $\bm{G}_0 = [.]$,
$\bm{G}_k:= (\P)_{\mathcal{G}_k}$,
$\lambda_{k}^+:= {\max_{i\in\mathcal{G}_{k}}\lambda_i\left(\bm{\Lambda} \right)}$, $\lambda_{k}^-:= {\min_{i\in\mathcal{G}_{k}}\lambda_i\left(\bm{\Lambda} \right)}$ and
\[
\chi: = \max_{k=1,2,\dots,\vartheta} \frac{\lambda_{k+1}^+}{\lambda_{k}^-}.
\]
$\chi$  quantifies the ``distance" between consecutive sets of the above partition.  Moreover, by definition,
$\frac{\lambda_{k}^+}{\lambda_{k}^-} \le g.$
Clearly, $g \ge 1$ and $\chi \le 1$ always. We assume the following.

\begin{sigmodel
\label{clust_model}
For a $1 \le g^+ < f$ and a $0 \le \chi^+ < 1$,
assume that there exists a $g$ satisfying $1 \le g \le  g^+ $ and a $\chi$ satisfying $0 \le \chi \le \chi^+$, for which we can define a $g$-condition-number partition of $\{1,2, \dots r \}$ that satisfies $\chi \le \chi^+$.
The number of sets in the partition is $\vartheta$.
When $g^+$ and $\chi^+$ are small, we say that the eigenvalues are ``well-clustered" with ``clusters", $\mathcal{G}_k$.
\end{sigmodel}

This assumption can be understood as a generalization of the eigen-gap condition needed by the block power method, which is a fast algorithm for obtaining the $k$ top eigenvectors of a matrix \cite{golub2000eigenvalue}.
We expect it to hold for data that has variability across different scales. The large scale variations would result in the first (largest eigenvalues') cluster and the smaller scale variations would form the later clusters. This would be  true, for example, for video ``textures" such as moving waters or waving trees in a forest. We tested this assumption on some such videos.
We describe our conclusions here for three videos - ``lake" (video of moving lake waters), ``waving-tree" (video consisting of waving trees), and ``curtain" (video of window curtains moving due to the wind).
For each video, we first made it low-rank by keeping the eigenvectors corresponding to the smallest number of eigenvalues that contain at least 90\% of the total energy and projecting the video onto this subspace.
For the low-rankified lake video, $f= 74$  and Assumption \ref{clust_model} holds with  $\vartheta = 6$ clusters, $g^+  = 2.6$ and $\chi^+ = 0.7$. For the waving-tree video, $f= 180$ and Assumption \ref{clust_model} holds with $\vartheta=6$, $g^+=9.4$ and $\chi^+=0.72$. For the curtain video,  $f= 107$ and the assumption holds $\vartheta=3$, $g^+=16.1$ and $\chi^+=0.5$. We show the clusters of eigenvalues in Fig. \ref{evals_plot}.

\begin{figure}
\begin{center}
	\begin{tabular}{c}
		\includegraphics[width=78mm, height=30mm]{style1.eps}
	\end{tabular}
	\end{center}
\vspace{-0.3in}
\caption{\small{Eigenvalue clusters of the three low-rankified videos.}
}\label{evals_plot}
\vspace{-0.25in}
\end{figure}

\Subsection{Cluster-EVD algorithm} \label{algo}
The cluster-EVD approach is summarized in Algorithm \ref{cPCA_algo}. I
Its main idea is as follows.
We start by computing the empirical covariance matrix of the first set of $\alpha$ observed data points, $\cM_1:= \frac{1}{\alpha} \sum_{t=1}^\alpha \bm{y}_t \bm{y}_t{}'$. Let $\hat\lambda_i$ denote its $i$-th largest eigenvalue.
To estimate the first cluster, $\hat{\mathcal{G}}_{1}$, we start with the index of the first (largest) eigenvalue and keep adding indices of the smaller eigenvalues to it until the ratio of the maximum to the minimum eigenvalue exceeds $\hat{g}^+$ or until the minimum eigenvalue goes below a ``zero threshold", $\lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^-$.
Then, we estimate the first cluster's subspace, $\operatorname{range}(\bm{G}_{1})$ by computing the top $\hat{r}_1$ eigenvectors of $\cM_1$.
To get the second cluster and its subspace, we project the next set of $\alpha$ $\bm{y}_t$'s orthogonal to $\hat{\bm{G}}_{1}$ followed by repeating the above procedure. This is repeated for each $k>1$. The algorithm stops when $\hat\lambda_{\hat{r}_k+1} < \lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^-$.

Algorithm \ref{cPCA_algo} is related to, but significantly different from, the ones introduced in \cite{rrpcp_perf,rrpcp_aistats} for the subspace deletion step of ReProCS.
The one introduced in \cite{rrpcp_perf} assumed that the clusters were known to the algorithm (which is unrealistic). The one studied in \cite{rrpcp_aistats} has an automatic cluster estimation approach, but, one that needs a larger lower bound on $\alpha$ compared to what Algorithm \ref{cPCA_algo} needs.

\begin{algorithm}[t!]
\caption{\small{\bf Cluster-EVD}}
\label{cPCA_algo}
{\bf Parameters: } $\alpha$, $\hat{g}^+$, $\lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^-$.

Set $\hat{\bm{G}}_{0} \leftarrow [.]$. Set the flag $\mathrm{Stop} \leftarrow 0$. Set $k \leftarrow 1$.

Repeat
\begin{enumerate}
 \vspace{-0.02in} \item \vspace{-0.02in} } %\newcommand{\Item}{ \vspace{-0.05in} \Item   Let $\hat{\bm{G}}_{\det,k}:= [\hat{\bm{G}}_{0},\hat{\bm{G}}_{1}, \dots \hat{\bm{G}}_{k-1}]$ and let $\bm\Psi_k:=(\bm{I} -\hat{\bm{G}}_{\det,k} \hat{\bm{G}}_{\det,k}{}')$. Notice that $\bm\Psi_{1}=\bm{I}$. Compute
  \[
 \cM_k = \bm\Psi_k \left(\frac{1}{\alpha}\sum_{t = (k-1) \alpha+1}^{k\alpha} \bm{y}} %{\y_t \bm{y}} %{\y_t{}'\right) \bm\Psi_k
  \]

 \vspace{-0.02in} \item \vspace{-0.02in} } %\newcommand{\Item}{ \vspace{-0.05in} \Item   Find the $k$-th cluster, $\hat{\mathcal{G}}_k$: let $\hat\lambda_i = \lambda_i(\cM_k)$;
\begin{enumerate}
 \vspace{-0.02in} \item \vspace{-0.02in} } %\newcommand{\Item}{ \vspace{-0.05in} \Item   find the index $\hat{r}_k$ for which $\frac{\hat\lambda_1}{\hat\lambda_{\hat{r}_k}} \le \hat{g}^+$  and either  $\frac{\hat\lambda_1}{\hat\lambda_{\hat{r}_k+1}} > \hat{g}^+$ or $\hat\lambda_{\hat{r}_k+1} < \lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^-$;

 \vspace{-0.02in} \item \vspace{-0.02in} } %\newcommand{\Item}{ \vspace{-0.05in} \Item   set $\hat{\mathcal{G}}_k = \{ \hat{r}_*+1, \hat{r}_* +2, \dots, \hat{r}_* + \hat{r}_k  \}$ where $\hat{r}_* := \sum_{j=1}^{k-1} \hat{r}_j$;

 \vspace{-0.02in} \item \vspace{-0.02in} } %\newcommand{\Item}{ \vspace{-0.05in} \Item   if $\hat\lambda_{\hat{r}_k+1} < \lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^-$, update the flag $\mathrm{Stop} \leftarrow 1$
\end{enumerate}

 \vspace{-0.02in} \item \vspace{-0.02in} } %\newcommand{\Item}{ \vspace{-0.05in} \Item   Compute $\hat{\bm{G}}_{k} \leftarrow \mathrm{eigenvectors}(\cM_k, \hat{r}_k)$;

increment $k$
\end{enumerate}
Until $\mathrm{Stop} == 1.$

Set $\hat\vartheta \leftarrow k$. Output $\hat{\bm{P}} \leftarrow [\hat{\bm{G}}_{1} \cdots \hat{\bm{G}}_{\hat\vartheta}]$.

$\mathrm{eigenvectors}(\bm{\mathcal{M}},r)$ returns a basis matrix for the span of the top $r$ eigenvectors of $\bm{\mathcal{M}}$
\end{algorithm}

\Subsection{Main result}
We give the performance guarantee for Algorithm \ref{cPCA_algo} here. Its parameters are set as follows. We set $\hat{g}^+$ to a value that is a little larger than $g$. This is needed to allow for the fact that $\hat\lambda_i$ is not equal to the $i$-th eigenvalue of $\bm{\Lambda}$ but is within a small margin of it. For the same reason, we need to also use a nonzero ``zeroing" threshold, $\lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^-$, that is larger than zero but smaller than $\lambda^-$. We set $\alpha$ large enough to ensure that  $\mathrm{SE}(\hat{\bm{P}},\P) \le r\zeta$ holds with a high enough probability.

\begin{theorem}[cluster-EVD result] \label{thm1}
Consider Algorithm \ref{cPCA_algo}.
Pick a $\zeta$ so that
$
r^2 \zeta \le 0.0001, \ \text{and} \ r^2\zeta f \le 0.01.
$
Suppose that $\bm{y}_t$'s satisfy \eqref{yt_mod} and the following hold.
\begin{enumerate}
\item  Assumption \ref{lt_mod} and Assumption \ref{clust_model} on $\bm{\ell}_t$ hold with $\chi^+$ satisfying $\chi^+ \le \min(1 -  r\zeta - \frac{0.08}{0.25}, \frac{g^+ - 0.0001}{1.01g^++0.0001} - 0.0001)$.
Define
\begin{align*}
& \alpha_0:=C \eta^2 \frac{r^2 (11\log n + \log \vartheta) }{ (r\zeta)^2}  \max ( g^+, q g^+,  \\
& q^2 f, q (r\zeta)f, (r \zeta)^2 f, q\sqrt{f g^+},(r\zeta)\sqrt{f g^+} )^2, \ C:= \frac{32 \cdot 16}{0.01^2}.
\end{align*}

\item  Assumption \ref{Mt_model} on $\bm{M}_t$ holds with $\alpha \ge \alpha_0$ and with $\beta$ satisfying
\[
\frac{\beta}{\alpha} \le  \left( \frac{(1 - r\zeta - \chi^+)}{2}  \right)^2 \min \left( \frac{ (r_k \zeta)^2 }{4.1 (q g^+)^2} , \frac{ (r_k \zeta) }{q^2 f} \right).
\]

\item  Set algorithm parameters  $\hat{g}^+ = 1.01g^+ + 0.0001$, $\lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^- = 0.95 \lambda^-$ and $\alpha \ge \alpha_0$.

\end{enumerate}
Then, with probability at least $1-12n^{-10}$,
$
\mathrm{SE}(\hat{\bm{P}},\P) \le  r \zeta.
$
\end{theorem}
{\em Proof: } The proof is given in Section \ref{proof_thm1} in Supplementary Material.

We can also get corollaries for PCA-missing and PCA-SDDC for cluster-EVD.
We have given one specific value for $\hat{g}^+$ and $\lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^-$ in Theorem \ref{thm1} for simplicity. One can, in fact, set $\hat{g}^+$ to be anything that satisfies \eqref{g_chi_f} given in Supplementary Material and one can set $\lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^-$ to be anything satisfying $5 r\zeta \lambda^- \le \lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^- \le 0.95 \lambda^-$.
Also, it should be possible to reduce the sample complexity of cluster-EVD to $c \max(q^2 (g^+)^2 r \log n, (g^+)^2 (r + \log n))$ using the approach explained in Sec. \ref{simple_evd}.

\Section{Discussion} \label{discuss}

\subsubsection{Comparing simple-EVD and cluster-EVD}
Consider the lower bounds on $\alpha$. In the cluster-EVD (c-EVD) result, Theorem \ref{thm1}, if $q$ is small enough (e.g., if $q \le 1/ \sqrt{f}$), and if $ (r^2\zeta) f \le 0.01$, it is clear that the maximum in the $\max(.,.,.,.)$ expression is achieved by $(g^+)^2$. Thus, in this regime, c-EVD needs $\alpha \ge C \frac{r^2 (11\log n + \log \vartheta) }{ ( r \zeta)^2} g^2$ and its sample complexity is $\vartheta \alpha$.
In the EVD result (Theorem \ref{thm1_pca}), $g^+$ gets replaced by $f$ and $\vartheta$ by 1, and so, its sample complexity, $\alpha \ge C \frac{r^2 11\log n }{(r \zeta)^2} f^2$.
In situations where the condition number $f$ is very large but $g^+$ is much smaller and $\vartheta$ is small (the clustering assumption holds well), the sample complexity of c-EVD will be much smaller than that of simple-EVD.
However, notice that, the lower bound on $\alpha$ for simple-EVD holds for any $q<1$ and for any $\zeta$ with $r\zeta<0.01$ while the c-EVD lower bound given above holds only when $q$ is small enough, e.g., $q = O(1/\sqrt{f})$, and $\zeta$ is small enough, e.g., $r\zeta = O(1/f)$. This tighter bound on $\zeta$ is needed because the error of the $k$-th step of c-EVD depends on the errors of the previous steps times $f$.
Secondly, the c-EVD result also needs $\chi^+$ and $\vartheta$ to be small (clustering assumption holds well), whereas, for simple-EVD, by definition, $\chi^+=0$ and $\vartheta = 1$.
Another thing to note is that the constants in both lower bounds are very large with the c-EVD one being even larger.

To compare the upper bounds on $\beta$, assume that the same $\alpha$ is used by both, i.e., $\alpha =\max(\alpha_0(\text{EVD}),\alpha_0(\text{c-EVD}))$. As long as $r_k$ is large enough, $\chi^+$ is small enough, and $g$ is small enough, the upper bound on $\beta$ needed by the c-EVD result is significantly looser. For example, if $\chi^+ = 0.2$, $\vartheta=2$, $r_k=r/2$,  then c-EVD needs $\beta \le  (0.5 \cdot 0.79 \cdot 0.5)^2  \frac{ (r \zeta)^2 }{4.1 q^2 g^2}   \alpha$ while simple-EVD needs  $\beta \le  (0.5 \cdot 0.99)^2  \frac{ (r \zeta)^2 }{4.1 q^2 f^2}   \alpha$. If $g=3$ but $f=100$, clearly the c-EVD bound is looser.

\subsubsection{Comparison with other results for PCA-SDDC and PCA-missing}
To our knowledge, there is no other result for correlated-PCA. Hence, we provide comparisons of the corollaries given above for the PCA-missing and PCA-SDDC special cases with works that also study these or related problems.
An alternative solution for either PCA-missing or PCA-SDDC is to first recover the entire matrix $\L$ and then compute its subspace via SVD on the estimated $\L$. For the PCA-missing problem, this can be done by using any of the low-rank matrix completion techniques, e.g., nuclear norm minimization (NNM) \cite{matcomp_candes} or alternating minimization (Alt-Min-MC) \cite{lowrank_altmin}. Similarly, for PCA-SDDC, this can be done by solving any of the recent provably correct RPCA techniques such as principal components' pursuit (PCP) \cite{rpca,rpca2,rpca_zhang}  or alternating minimization (Alt-Min-RPCA) \cite{robpca_nonconvex}.

However, as explained earlier doing the above has two main disadvantages. The first is that it is much slower (see Sec. \ref{sims}).
The difference in speed is most dramatic when solving the matrix-sized convex programs such as NNM or PCP, but even the Alt-Min methods are slower. If we use the time complexity from \cite{robpca_nonconvex}, then finding the span of the top $k$ singular vectors of an $n \times m$ matrix takes $O(nm k)$ time. Thus, if $\vartheta$ is a constant, both simple-EVD and c-EVD need $O(n \alpha r)$ time, whereas, Alt-Min-RPCA needs $O(n \alpha r^2)$ time per iteration \cite{robpca_nonconvex}.
The second disadvantage is that the above methods for MC or RPCA need more assumptions to provably correctly recover $\L$. All the above methods need an incoherence assumption on both the left singular vectors, $\P$, and the right singular vectors, $\bm{V}$, of $\L$. Of course, it is possible that, if one studies these methods with the goal of only recovering the column space of $\L$ correctly,  the incoherence assumption on the right singular vectors is not needed. From simulation experiments (see Sec. \ref{sims}), the incoherence of the left singular vectors is definitely needed.
On the other hand, for the PCA-SDDC problem, simple-EVD or c-EVD do not even need the incoherence assumption on $\P$.

The disadvantage of both EVD and c-EVD, or in fact of any solution for the PCA problem,  is that they work only when $q$ is small enough (the corrupting noise is small compared to $\bm{\ell}_t$).

\begin{table*}[t!]
  \centering
  \begin{tabular}{c|cccc|cccc}
    \toprule
\multicolumn{1}{c}{ } &   \multicolumn{4}{c}{Mean Subspace Error (SE)}  & \multicolumn{4}{c}{Average Time}                   \\
    \cmidrule{1-9}
   & c-EVD & EVD & PCP & A-M-RPCA  & c-EVD & EVD & PCP & A-M-RPCA\\
    \midrule
Expt 1  & 0.0908 & 0.0911 & 1.0000 & 1.0000 &   0.0549 & 0.0255 & 0.2361 & 0.0810 \\ \hline
Expt 2   & 0.3626 & 0.3821 & 0.4970 & 0.4846 &   0.0613 & 0.0223   & 1.6784 & 5.5144  \\
    \bottomrule
  \end{tabular}
\vspace{-0.1in}
  \caption{\footnotesize{Comparison of $\mathrm{SE}(\hat{\bm{P}},\P)$ and execution time (in seconds). A-M-RPCA: Alt-Min-RPCA. Expt 1: simulated data, Expt 2: lake video with simulated foreground.}}
\label{table_real} \label{sparseP_small_n}
\vspace{-0.2in}
\end{table*}

\Section{Numerical Experiments} \label{sims}
We use the PCA-SDDC problem as our case study example. We compare EVD and cluster-EVD (c-EVD) with PCP \cite{rpca2}, solved using \cite{alm}, and with Alt-Min-RPCA \cite{robpca_nonconvex} (implemented using code from the authors' webpage). For both PCP and Alt-Min-RPCA, $\hat{\bm{P}}$ is recovered as the top $r$ eigenvectors of of the estimated $\L$.
To show the advantage of EVD or c-EVD, we let $\bm{\ell}_t = \P \bm{a}_t$ with columns of $\P$ being sparse. These were chosen as the first $r=5$ columns of the identity matrix.
We generate $\bm{a}_t$'s iid uniformly with zero mean and covariance matrix $\bm\Lambda = diag(100, 100, 100, 0.1, 0.1)$. Thus the condition number $f = 1000$. The clustering assumption holds with $\vartheta=2$, $g^+ = 1$ and $\chi^+ = 0.001$.
 The noise $\bm{w}_t$ is generated as $\bm{w}_t = \bm{I}_{\mathcal{T}_t} \bm{M}_{s,t} \bm{\ell}_t$ with $\mathcal{T}_t$ generated to satisfy Assumption \ref{sbyrho} with $s=5$, $\rho=2$, and $\tilde\beta = 1$; and the entries of $\bm{M}_{s,t}$ being iid $\mathcal{N}(0,q^2)$ with $q=0.01$.
 We used $n =500$. EVD and c-EVD (Algorithm \ref{cPCA_algo}) were implemented with $\alpha=300$, $\lambda_{\mathrm{thresh}}}  %{\mathrm{thresh}}  %{\hat{\lambda}^- = 0.095 $, $\hat{g}^+ = 3$. 10000-time Monte Carlo averaged values of $\mathrm{SE}(\hat{\bm{P}},\P)$ and execution time are shown in the first row of Table \ref{table_real}.
Since the columns of $\P$ are sparse, both PCP and Alt-Min-RPCA fail. Both have average $\mathrm{SE}$ close to one whereas the average SE of c-EVD and EVD is 0.0908 and 0.0911 respectively. Also, both EVD and c-EVD are much faster than the other two.
We also did an experiment with the settings of this experiment, but with $\P$ dense. In this case, EVD and c-EVD errors were similar, but PCP and Alt-Min-RPCA errors were less than $10^{-5}$.

For our second experiment, we used images of a low-rankified real video sequence as $\bm{\ell}_t$'s. We chose the escalator sequence from \url{http://perception.i2r.a-star.edu.sg/bk_model/bk_index.html} since the video changes are only in the region where the escalator moves (and hence can be modeled as being sparse). We made it exactly low-rank by retaining its top 5 eigenvectors and projecting onto their subspace. This resulted in a data matrix $\L$ of size $n \times r$ with $n=20800$ and $r = 5$. We overlaid a simulated moving foreground block on it. The intensity of the moving block was controlled to ensure that $q$ is small.
We estimated $\hat{\bm{P}}$ using EVD, c-EVD, PCP and Alt-Min-RPCA.
We let $\P$ be the eigenvectors of the low-rankified video with nonzero eigenvalues and computed $\mathrm{SE}(\hat{\bm{P}},\P)$. The errors and execution time are displayed in the second row of Table \ref{table_real}. Since $n$ is very large, the difference in speed is most apparent in this case.

Thus c-EVD outperforms PCP and AltMinRPCA when columns of $\P$ are sparse. It also outperforms EVD but the advantage in mean error is not as much as our theorems predict. One reason is that the constant in the required lower bounds on $\alpha$ is very large. It is hard to pick an $\alpha$ that is this large and still only $O(\log n)$  unless $n$ is very large. Secondly, both guarantees are only sufficient conditions

\Section{Conclusions and Future Work} \label{conclusions}
We studied the problem of PCA in noise that is correlated with the data (data-dependent noise). We obtained sample complexity bounds for the most commonly used PCA solution, simple EVD. We also developed and analyzed a generalization of EVD, called cluster-EVD, that has lower sample complexity under extra assumptions.
We provided a detailed comparison of our results with those for other approaches to solving its example applications - PCA with missing data and PCA with sparse data-dependent corruptions.%

We used the matrix Hoeffding inequality \cite{tail_bound} to obtain our results. As explained in Sec. \ref{simple_evd}, it should be possible to improve the sample complexity bounds if this is replaced by \cite[Theorem 5.39]{vershynin} or matrix Bernstein.
Moreover, as done in \cite{rrpcp_aistats} (for ReProCS), the mutual independence of $\bm{\ell}_t$'s can be easily replaced by a more practical assumption of $\bm{\ell}_t$'s following autoregressive model with almost no change to our assumptions.
Thirdly, by generalizing the proof techniques developed here, we can also study the problem of correlated-PCA with partial subspace knowledge. This is done in \cite{corpca_tsp}. The solution to the latter problem helps to greatly simplify the proof of correctness of ReProCS for online dynamic RPCA \cite{long_simple_proof}.
Fourthly, the boundedness assumption on $\bm{\ell}_t$'s can be replaced by a Gaussian or a  well-behaved sub-Gaussian assumption but this will increase the sample complexity to $O(n)$.
Finally, an open-ended question is how we relax  Assumption \ref{Mt_model} on $\bm{M}_t$ and still get results similar to Theorem \ref{thm1_pca} or Theorem \ref{thm1}.

\clearpage
\bibliographystyle{IEEEbib}

Machine Learning (cs.LG); Information Theory (cs.IT)