A Parameter-Uniform Finite Difference Method for Multiscale Singularly Perturbed Linear Dynamical Systems

Abstract: A system of singularly perturbed ordinary differential equations of first order with given initial conditions is considered. The leading term of each equation is multiplied by a small positive parameter. These parameters are assumed to be distinct and they determine the different scales in the solution to this problem. A Shishkin piecewise--uniform mesh is constructed, which is used, in conjunction with a classical finite difference discretization, to form a new numerical method for solving this problem. It is proved that the numerical approximations obtained from this method are essentially first order convergent uniformly in all of the parameters.

\section{Introduction}
We consider the  initial value problem for the singularly perturbed
system of linear first order differential equations
\begin{eqnarray}\label{IVP}
E\vec{u}^{\prime}(t)+A(t)\vec{u}(t)=\vec{f}(t), \;\; t \in (0,T],
\;\; \vec{u}(0)\;\; \mathrm{given}.
\end{eqnarray}
Here $\vec{u}$ is a column $n$-vector, $E$ and $A(t)$ are $n \times
n$ matrices, $E=diag(\vec \varepsilon)$, $\vec \varepsilon= (\varepsilon_1, \; \dots, \;
\varepsilon_n) $ with
 $0 < \varepsilon_i \leq 1$ for all $i=1 \dots n$.
For convenience we assume the ordering
\[\varepsilon_1 < \dots < \varepsilon_n.\]
These $n$ distinct parameters determine the $n$ distinct scales in
this multiscale problem. Cases with some of the parameters
coincident are not considered here. We write the problem in the
operator form
\begin{eqnarray*}
\vec{L}\vec{u}=\vec{f},\;\; \vec{u}(0) \;\; \mathrm{ given },
\end{eqnarray*}
where the operator $\vec L$ is defined by
\[ \vec{L}=ED+A(t)\;\; \rm{and} \;\; D=\frac{d}{dt}.  \]
We assume that, for all $t \in [0,T]$, the components $a_{ij}(t)$ of $A(t)$  satisfy the inequalities \\
\begin{eqnarray}\label{a1} a_{ii}(t) > \displaystyle{\sum_{^{j\neq
i}_{j=1}}^{n}}|a_{ij}(t)| \; \; \rm{for}\;\; 1 \le i \le n, \;\;
\rm{and} \;\;a_{ij}(t) \le 0 \;\; \rm{for} \; \; i \neq
j.\end{eqnarray} We take $\alpha$ to be any number such that
\begin{eqnarray}\label{a2} 0 <\alpha < \displaystyle{\min_{^{t \in (0,1]}_{1 \leq i \leq n}}}(\sum_{j=1}^n
a_{ij}(t)).
\end{eqnarray}
We also assume that $T \ge 2\max_{i}(\varepsilon_i)/\alpha$,
 which ensures that the solution domain contains all of the
 layers. This condition is fulfilled if, for example,  $T \ge 2/\alpha$.
We introduce the norms $\parallel \vec{V} \parallel =\max_{1 \leq k
\leq n}|V_k|$ for any n-vector $\vec{V}$, $\parallel y
\parallel =\sup_{0\leq t\leq T}|y(t)|$ for any
scalar-valued function $y$ and $\parallel \vec{y}
\parallel=\max_{1 \leq k \leq n}\parallel y_{k}
\parallel$ for any vector-valued function $\vec{y}$.
Throughout the paper $C$ denotes a generic positive constant,
which is independent of $t$ and of all singular perturbation and
discretization parameters. Furthermore, inequalities between vectors are understood in the componentwise sense.\\

The plan of the paper is as follows. In the next section both
standard and novel bounds on the smooth and singular components of
the exact solution are obtained. The sharp estimates in Lemma 2.4
are proved by mathematical induction, while an interesting
ordering of the points $t_{i,j}$ is established in Lemma 2.6.
In Section 3 the appropriate piecewise-uniform Shishkin meshes are
introduced, the discrete problem is defined and the discrete
maximum principle and discrete stability properties are
established. In Section 4 an expression for the local truncation
error is found and two distinct standard estimates are stated. In
the final section parameter-uniform estimates for the local
truncation error of the smooth and singular components are
obtained in a sequence of lemmas. The section culminates with
the statement and proof of the parameter-uniform error estimate, which is the main result of the paper.\\

The initial value problems considered here arise in many areas of applied mathematics; see for example \cite{Ath}. Parameter uniform numerical methods for simpler
problems of this kind, when all the singular perturbation parameters are equal, were considered in \cite{HV}. A special case of the present problem with $n=3$ was
considered in \cite{BAIL2008}, which also contains numerical results confirming the theory. For this reason further numerical validation is considered to be
unnecessary. A general introduction to parameter uniform numerical methods is given in \cite{FHMORS} and \cite{MORS}.

\section{Analytical results}
The operator $\vec{L}$ satisfies the following maximum principle
\begin{lemma}\label{max} Let $A(t)$ satisfy (\ref{a1}) and (\ref{a2}).
Let $\vec \psi(t)$ be any function in the domain of $\vec{L}$ such
that $\vec\psi(0)\ge 0.$ Then $\vec{L}\vec{\psi}(t) \geq 0$ for
all $t \in (0,T]$ implies that $\vec{\psi}(t) \geq 0$ for all $t
\in [0,T]$.
\end{lemma}
\begin{proof}Let $i^*, t^*$ be such that $\psi_{i^*}(t^{*})=\min_{i,t}\psi_i(t)$
and assume that the lemma is false. Then $\psi_{i^*}(t^{*})<0$ .
From the hypotheses we have $t^*\neq 0$ and
$\psi^{\prime}_{i^*}(t^*)\leq 0$. Thus
\begin{eqnarray*}(\vec{L}\vec \psi(t^*))_{i^*}=\varepsilon_{i^*}\psi^{\prime}_{i^*}(t^*)+a_{i^*,i^*}(t^*)\psi_{i^*}(t^*)+\sum_{j=1,\;j
\neq i^* }^n a_{i^*,j}(t^*)\psi_j(t^*) \\ <
\psi_i^*(t^*)\sum_{j=1\;j \neq i^* }^n a_{i^*,j}<0.
\end{eqnarray*} which contradicts the assumption and proves the
result for $\vec{L}$. \end{proof}

Let $\tilde{A}(t)$ be any principal sub-matrix of $A(t)$ and
$\vec{\tilde{L}}$ the corresponding operator. To see that any
$\vec{\tilde{L}}$ satisfies the same maximum principle as $\vec{L}$,
it suffices to observe that the elements of $\tilde{A}(t)$ satisfy
\emph{a fortiori} the same inequalities as those of $A(t)$.

We remark that the maximum principle is not necessary for the
results that follow, but it is a convenient tool in their proof.
\begin{lemma}\label{stab} Let $A(t)$ satisfy (\ref{a1}) and (\ref{a2}).
If $\vec \psi(t)$ is any function in the domain of $\vec{L}$
then\[\parallel\vec \psi(t)\parallel\le \max\left\{\parallel\vec
\psi(0)\parallel,\frac{1}{\alpha}\parallel \vec{L}\vec
\psi\parallel\right\},\qquad t\in [0,T]\]
\end{lemma}
\begin{proof}Define the two functions
\[\vec{\theta}^{\pm}(t)=\max\{||\vec{\psi}(0)||,\frac{1}{\alpha}||\vec{L}\vec{\psi}||\}\vec{e}\pm
\vec{\psi}(t), \]where $\vec{e}=(1,\;\dots \;,1)^{\prime}$ is the
unit column vector. Using the properties of $A$ it is not hard to
verify that $\vec{\theta}^{\pm}(0)\geq 0$ and
$\vec{L}\vec{\theta}^{\pm}(t)\geq 0$. It follows from Lemma
\ref{max} that $\vec{\theta}^{\pm}(t)\geq 0$ for all $t \in [0,T]$.
\end{proof}

The Shishkin decomposition of the solution $\vec u\;$ of (1) is
given by $\vec u = \vec v + \vec w\;$ where $\vec v \;$ is the
solution of $\vec{L} \vec v = \vec f\;$ on $(0,T]\;$ with $\vec v(0)
= A^{-1}(0)\vec f(0)\;$ and $\vec w \;$ is the solution of
$\vec{L}\vec w = \vec 0\;$ on $(0,T]\;$ with $\vec w(0) = \vec u(0)
- \vec v(0).\;$ Here $\vec v\; , \vec w $ are, respectively, the
smooth
and singular components of $\vec u\;$.\\
The smooth component $\vec{v}$ of $\vec u$ and its derivatives are
estimated the following lemma, which gives bounds showing the
explicit dependence on the inhomogeneous term and the initial
condition.
\begin{lemma}\label{smooth} Let $A(t)$ satisfy (\ref{a1}) and (\ref{a2}). Then there exists a constant $C$, independent of $\varepsilon, \vec{u}(0)$ and
$\vec{f}$, such that
\[\parallel \vec{v}\parallel \leq C\parallel\vec{f}\parallel, \;
\parallel \vec{v}^{\prime}\parallel \leq
C(\parallel\vec{f}\parallel+\parallel\vec{f}^{\prime}\parallel)\]
 and, for all $1 \leq i \leq n$,  \[\parallel \varepsilon_i v_i^{\prime\prime}\parallel \leq
C(\parallel\vec{f}\parallel+\parallel\vec{f}^{\prime}\parallel)\]
\end{lemma}
\begin{proof}We introduce the two functions
$\vec{\psi}^{\pm}(t)=C||\vec{f}||\vec{e}\pm \vec{v}(t)$ where
$\vec{e}$ is the unit column vector. Noting that
$\vec{v}(0)=A^{-1}(0)\vec{f}(0)$, it is not hard to see that
$\vec{\psi}^{\pm}(0)\geq 0$ and $\vec{L}\vec{\psi}^{\pm}(t)\geq
0$. It follows from Lemma \ref{max} that $\vec{\psi}^{\pm}(t)\geq
0$ for all $t \in [0,T]$ and so $\parallel \vec{v}\parallel \leq
C\parallel\vec{f}\parallel$. To estimate the derivative we now
define the two functions
$\vec{\phi}^{\pm}(t)=C(||\vec{f}||+||\vec{f}^{\prime}||)\vec{e}\pm
\vec{v^{\prime}}(t)$. Since $\vec{v}^{\prime}(0)= 0$ and
$\vec{L}\vec{v}^{\prime}= \vec{f}^{\prime}-A^{\prime}\vec{v}$, it
may be verified that $\vec{\phi}^{\pm}(0)\ge 0$ and
$\vec{L}\vec{\phi}^{\pm}(t)\geq 0$. Again by Lemma \ref{max} we
have $\vec{\phi}^{\pm}(t)\geq 0$, which proves the result.
Finally, differentiating the equation $\varepsilon_i
v_i^{\prime}+(A\vec{v})_i=f_i$ and using the estimates of
$\vec{v}$ and $\vec{v}^{\prime}$, we obtain the required bound on
$\varepsilon_i v_i^{\prime \prime}$ \end{proof}

We define the layer functions $B_{i}, 1 \leq i \leq n$, associated
with the solution $\vec u\;$ by
\[B_{i}(t) = e^{-\alpha t / \varepsilon_i}, \; t \in [0,\infty).\]
The following elementary properties of these layer functions, for
all $1 \leq i < j \leq n$, should be
noted:\\
(i) $B_i(t) < B_j(t)$, for all  $t>0$.\\
(ii) $B_i(s) > B_i(t)$,  for all  $0 \leq s<t < \infty$.\\
(iii) $B_i(0)=1$  and  $0<B_i(t) < 1$  for all   $t>0$.\\
Bounds on the singular component $\vec{w}$ of $\vec{u}$ and its
derivatives are contained in
\begin{lemma}\label{singular} Let $A(t)$ satisfy (\ref{a1}) and
(\ref{a2}).Then there exists a constant $C,$ such that, for each
$t \in [0,T]$ and $i=1,\; \dots , \; n$,
\[\left|w_i(t)\right| \;\le\; C B_{n}(t),\;\;
\left|w_i^\prime(t)\right| \;\le\; C\sum_{q=i}^n
\frac{B_{q}(t)}{\varepsilon_q},\;\; \left|\varepsilon_i
w_i^{\prime\prime}(t)\right| \;\le\; C\sum_{q=1}^n
\frac{B_{q}(t)}{\varepsilon_q}.\]
\end{lemma}

\begin{proof}First we obtain the bound on $\vec{w}$. We define the two
functions $\vec{\psi}^{\pm}=CB_n\vec{e} \pm \vec{w}$. Then clearly
$\vec{\psi}^{\pm}(0) \geq 0$ and $L\vec{\psi}^{\pm}=CL(B_n\vec{e})$.
Then, for $i=1,\dots, n$, $(L\vec{\psi}^{\pm})_i
=C(\sum_{j=1}^{n}a_{i,j}-\alpha\frac{\varepsilon_i}{\varepsilon_n})B_n >0$. By
Lemma \ref{max} $\vec{\psi}^{\pm}\geq 0$, which leads to the
required bound on $\vec{w}$.

To establish the bound on $\vec{w}^{\prime}$ we begin with the
$n^{th}$ equation in $\vec{L}\vec{w}=0$, namely
\[ \varepsilon_nw_n^\prime + a_{n,1}w_1 +\dots +a_{n,n}w_n=0, \] from
which the bound for $i=n$ follows. We now bound $w_i^{\prime}$ for
$1 \le i \le n-1$. We define $\vec{p}=(w_1, \dots,w_{n-1})$ and,
taking the first $n-1$ equations satisfied by $\vec{w}$, we get
\[\tilde{A}\vec{p}=\vec{g},\] where $\tilde{A}$ is the matrix
obtained from $A$ by deleting the last row and column and the
components of $\vec{g}$ are $g_k=-a_{k,n}w_n$ for $1 \le k \le n-1$.
Using the bounds already obtained for $\vec{w}$ we see that
$\vec{g}$ is bounded by $CB_n (t)$ and its derivative by $C\frac{B_n
(t)}{\varepsilon_n}$. The initial condition for $\vec{p}$ is
$\vec{p}(0)=\vec{u}(0)-\vec{u}^0(0)$, where $\vec{u}^0$ is the
solution of the reduced problem $\vec{u}^0=A^{-1}\vec{f}$, and is
therefore bounded by
$C(\parallel\vec{u}(0)\parallel+\parallel\vec{f}(0)\parallel)$.
Decomposing $\vec{p}$ into smooth and singular components we get
\[\vec{p}=\vec{q}+\vec{r}, \;\;\ \vec{p}^{\prime}=\vec{q}^{\prime}+\vec{r}^{\prime}.\]
Applying Lemma \ref{smooth}  to $\vec{q}$, from the bounds on the
inhomogeneous term $\vec{g}$ and its derivative $\vec{g}^{\prime}$,
we conclude that $\parallel\vec{q}^{\prime}(t)\parallel \leq
C\frac{B_n(t)}{\varepsilon_n}$. We now use mathematical induction. We
assume that Lemma \ref{singular} is valid for all systems with $n-1$
equations. Then Lemma \ref{singular} applies to $\vec{r}$ and so,
for $i=1, \dots, n-1$,
\[
|r^{\prime}_{i}(t) | \leq
C(\frac{B_{i}(t)}{\varepsilon_i}+\dots+\frac{B_{n-1}(t)}{\varepsilon_{n-1}}).\]
Combining the bounds for $q_i$ and $r_i$ we obtain
\[|p^{\prime}_i(t)|\leq
C(\frac{B_{i}(t)}{\varepsilon_i}+\dots+\frac{B_{n}(t)}{\varepsilon_n}).\]
Recalling the definition of $\vec{p}$ this is the same as
\[|w^{\prime}_i(t)|\leq
C(\frac{B_{i}(t)}{\varepsilon_i}+\dots+\frac{B_{n}(t)}{\varepsilon_n}).\] We have
thus proved that Lemma \ref{singular} holds for our system with $n$
equations. Since Lemma \ref{singular} is true for a system with one
equation,  we conclude by mathematical induction that it is true for
any system of $n>1$ equations.

Finally, to estimate the second derivative, we differentiate the
$i^{th}$ equation of the system $\vec{L}\vec{w}=0$ to get
\[\varepsilon_{k}w_{i}^{\prime\prime}=-(A\vec{w}_i^{\prime} + A^{\prime}\vec{w})_i\]
and we see that the bound on $w_{i}^{\prime\prime}$ follows easily
from the bounds on $\vec{w}$ and $\vec{w}^{\prime}$. \end{proof}
\begin{definition}
For each $1 \leq i \neq j \leq n$ we define the point $t_{i,j}$ by
\begin{equation}\label{t0}\frac{B_i(t_{i,j})}{\varepsilon_i}=\frac{B_j(t_{i,j})}{\varepsilon_j}. \end{equation}
\end{definition}
In the next lemma it is shown that these points exist, are
uniquely defined and have an interesting ordering. Sufficient
conditions for them to lie in the domain $[0,T]$ are also
provided.

 \begin{lemma}\label{ts} For all $i,j$ with $1 \leq i < j \leq n$ the points
$t_{i,j}$ exist, are uniquely defined and satisfy the following
inequalities
\begin{equation}\label{t1}
\varepsilon_i^{-1}B_{i}(t) > \varepsilon_j^{-1}B_{j}(t)\qquad t \in [0,t_{ij})
\end{equation}
and
\begin{equation}\label{t2}
\varepsilon_i^{-1}B_{i}(t) < \varepsilon_j^{-1}B_{j}(t)\qquad t \in (t_{ij},\infty).
\end{equation}
In addition the following ordering holds
\begin{equation}\label{t3}t_{i,j}< t_{i+1,j}, \; \mathrm{if} \;\; i+1<j \;\; \mathrm{and} \;\; t_{i,j}<
t_{i,j+1}, \;\; \mathrm{if} \;\; i<j \end{equation} \\
and
\begin{equation}\label{t4} \varepsilon_i \leq \varepsilon_{j}/2 \;\; \mathrm{implies \;\; that}
\;\; t_{ij} \in (0,T] \;\; \mathrm{for \;\; all} \;\; i<j.
\end{equation}
\end{lemma}
\begin{proof} Existence, uniqueness, (\ref{t1}) and (\ref{t2}) all follow
from the observation that for $i<j$ we have $\varepsilon_i<\varepsilon_j$ and
the ratio of the two sides of (\ref{t0}), namely
\[\frac{B_{i}(t)}{\varepsilon_i}\frac{\varepsilon_j}{B_{j}(t)}=
\frac{\varepsilon_j}{\varepsilon_i}\exp{(-\alpha
t(\frac{1}{\varepsilon_i}-\frac{1}{\varepsilon_j}))},\] is monotonically
decreasing from $\frac{\varepsilon_j}{\varepsilon_i} >1$ to $0$ as
$t$ increases
from $0$ to $\infty$.\\

Rearranging (\ref{t0}) gives
\[t_{i,j}=\frac{\ln(\frac{1}{\varepsilon_i})-\ln(\frac{1}{\varepsilon_j})}{\alpha(\frac{1}{\varepsilon_i}-\frac{1}{\varepsilon_j})}
.\] Writing $\varepsilon_k = \exp(-p_k)$ for some $p_k > 0$ and all $k$
gives
\[t_{i,j}=\frac{p_i -p_j}{\alpha(\exp{p_i} -\exp{p_j})}.\] The
inequality $t_{i,j}< t_{i+1,j}$ is equivalent to
\[\frac{p_i -p_j}{\exp{p_i} -\exp{p_j}}<\frac{p_{i+1} -p_j}{\exp{p_{i+1}} -\exp{p_j}}, \]
which can be written in the form
\[(p_{i+1}-p_j)\exp(p_i-p_j)+(p_{i}-p_{i+1})-(p_{i}-p_j)\exp(p_{i+1}-p_j)>0. \]
With $a=p_i-p_j$ and $b=p_{i+1}-p_j$ it is not hard to see that
$a>b>0$ and $a-b=p_i-p_{i+1}$. Moreover, the previous inequality is
then equivalent to \[\frac{\exp{a}-1}{a}>\frac{\exp{b}-1}{b}, \]
which is true because $a>b$ and proves the first part of
(\ref{t3}). The second part is proved by a similar argument.\\

Finally, to prove (\ref{t4}) it suffices to rearrange (\ref{t0})
in the form
\[t_{i,j}=\frac{\ln(\frac{\varepsilon_j}{\varepsilon_i})}{\alpha(\frac{1}{\varepsilon_i}-\frac{1}{\varepsilon_j})}.
\] Since $T > \frac{2}{\alpha}$ and $\varepsilon_i \leq \frac{\varepsilon_j}{2}$ it follows that
$\ln(\frac{\varepsilon_j}{\varepsilon_i}) \leq \frac{\varepsilon_j}{\varepsilon_i}$ and $t_{i,j}
\in (0,T]$. \end{proof}

\section{The discrete problem}
We construct a piecewise uniform mesh with $N$ mesh-intervals and
mesh-points $\{t_i\}_{i=0}^N$ by dividing the interval $[0,T]$ into
$n+1$ sub-intervals as follows
\[
[0,T]=[0,\sigma_1]\cup(\sigma_1,\sigma_2]\cup\dots(\sigma_{n-1},\sigma_n]\cup(\sigma_n,T]\]
Then, on the sub-interval $[0,\sigma_1]$, a uniform mesh with
$\frac{N}{2^n}$ mesh-intervals is placed, and similarly on
$(\sigma_i, \sigma_{i+1}], 1 \leq i \leq n-1$,  a uniform mesh with
$ \frac{N}{2^{n-i+1}}$ mesh-intervals  and  on  $(\sigma_n,T]$ a
uniform mesh with $\frac{N}{2}$
 mesh-intervals. In practice it is convenient to take $N=2^{n}k$ where $k$
 is some positive power of 2. The $n$ transition
points between the uniform meshes are defined by
\[\sigma_{i}=\min\{\frac{\sigma_{i+1}}{2},\frac{\varepsilon_i}{\alpha}\ln
N\}\]for $i=1,\dots,n-1$ and
\[\sigma_{n}=\min\{\frac{T}{2},\frac{\varepsilon_n}{\alpha}\ln N\}. \]
Clearly \[ 0<\sigma_1 < \dots < \sigma_n \leq \frac{T}{2}. \]This
construction leads to a class of $2^n$ piecewise uniform Shishkin
 meshes $M_{\vec{b}}$, where $\vec b$ denotes an $n$--vector with
$b_i=0$ if $\sigma_i=\frac{\sigma_{i+1}}{2}$ and $b_i=1$ otherwise.
Writing $\delta_j=t_j-t_{j-1}$ we remark that, on any $M_{\vec b}$,
we have
\begin{equation}\label{geom1}\delta_j \leq
CN^{-1}, \;\;\; 1 \leq j \leq N \end{equation}
 and
\begin{equation}\label{geom2}\sigma_i \leq C \varepsilon_i \ln N, \; \;\; 1 \leq i \leq n. \end{equation}
On any $M_{\vec b}$ we now consider the discrete solutions defined
by the backward Euler finite difference scheme
\[ED^{-}\vec{U} +A(t)\vec{U}=\vec{f},  \qquad \vec{U}(0)=\vec{u}(0), \]
or  in operator form
\[\vec{L}^N \vec{U} =\vec{f},  \qquad  \vec{U}(0)=\vec{u}(0), \] where
\[\vec{L}^N=ED^{-}+A(t)\]
and $D^{-}$ is the backward difference operator
\[
D^-\vec{U}(t_j)= \frac {\vec{U}(t_j)-\vec{U}(t_{j-1})}{\delta_j}.
\]
We have the following discrete maximum principle analogous to the
continuous case.
\begin{lemma}\label{dmax} Let $A(t)$ satisfy (\ref{a1}) and (\ref{a2}).
Then, for any mesh function $\vec \Psi$, the inequalities $\vec
{\Psi}(0)\;\ge\;\vec 0 \;\rm{and}\;\vec{L}^N
\vec{\Psi}(t_j)\;\ge\;\vec 0\;$ for $1\;\le\;j\;\le\;N,\;$ imply
that $\;\vec \Psi(t_j)\ge \vec 0\;$ for $0\;\le\;j\;\le\;N.\;$
\end{lemma}

\begin{proof} Let $i^*, j^*$ be such that
$V_{i^*}(t_{j^{*}})=\min_{i,j}V_i(t_j)$ and assume that the lemma
is false. Then $V_{i^*}(t_{j^{*}})<0$ . From the hypotheses we
have $j^*\neq 0$ and $V_{i^*} (t_{j^*})-V_{i^*}(t_{j^*-1})\leq 0$.
Thus
\begin{eqnarray*}(\vec{L^N}\vec{V}(t_{j^*}))_{i^*}=
\varepsilon_{i^*}\frac{V_{i^*}(t_{j^*})-V_{i^*}(t_{j^*-1})}{\delta_{j^*}}+a_{i^*,i^*}(t_{j^*})V_{i^*}(t_{j^*})+\sum_{k=1\;k
\neq i^* }^n a_{i^*,k}(t_{j^*})V_{k}(t_{j^*}) \\ <
V_{i^*}(t_{j^*})\sum_{k=1\;k \neq i^* }^n a_{i^*,k}<0,
\end{eqnarray*} which contradicts the assumption, as required. \end{proof}

An immediate consequence of this is the following discrete stability
result.
\begin{lemma}\label{dstab} Let $A(t)$ satisfy (\ref{a1}) and (\ref{a2}).
Then, for any mesh function $\vec \Psi $,
\[\parallel\vec \Psi(t_j)\parallel\;\le\;\max\left\{\parallel\vec \Psi(0)\parallel,\frac{1}{\alpha}\parallel
\vec{L}^N\vec \Psi\parallel\right\}, 0\leq j \leq N \]
\end{lemma}
\begin{proof} Define the two functions
\[\vec{\Theta}^{\pm}(t)=\max\{||\vec{\Psi}(0)||,\frac{1}{\alpha}||\vec{L^N}\vec{\Psi}||\}\vec{e}\pm
\vec{\Psi}(t)\]where $\vec{e}=(1,\;\dots \;,1)$ is the unit vector.
Using the properties of $A$ it is not hard to verify that
$\vec{\Theta}^{\pm}(0)\geq 0$ and
$\vec{L^N}\vec{\Theta}^{\pm}(t_j)\geq 0$. It follows from Lemma
\ref{dmax} that $\vec{\Theta}^{\pm}(t_j)\geq 0$ for all $0\leq j
\leq N$.
\end{proof}
\section{The local truncation error}
From Lemma \ref{dstab}, we see that in order to bound the error
$\parallel\vec{U}-\vec{u}\parallel$ it suffices to bound
$\vec{L}^N(\vec{U}-\vec{u})$. But this expression satisfies
\[
\vec{L}^N(\vec{U}-\vec{u})=\vec{L}^N(\vec{U})-\vec{L}^N(\vec{u})=
\vec{f}-\vec{L}^N(\vec{u})=\vec{L}(\vec{u})-\vec{L}^N(\vec{u})\]
\[=(\vec{L}-\vec{L}^N)\vec{u} =-E(D^- -D)\vec{u},\] which is the local
truncation of the first derivative. We have
\[E(D^- -D)\vec{u}
=E(D^- -D)\vec{v}+E(D^- -D)\vec{w}
\] and so, by the triangle inequality, \begin{equation}\label{triangleinequality}\parallel \vec{L}^N(\vec{U}-\vec{u})\parallel \leq \parallel
E(D^- -D)\vec{v}\parallel+\parallel E(D^- -D)\vec{w}\parallel.
\end{equation} Thus, we can treat the smooth and singular components
of the local truncation error separately. In view of this we note
that, for any smooth function $\psi$, we have the following two
distinct estimates of the local truncation error of its first
derivative
\begin{equation}\label{lte1}
|(D^- -D)\psi(t_j)|\le2\max_{s\in
I_j}|\psi^{\prime}(s)|\qquad\qquad\;\;
\end{equation}
and
\begin{equation}\label{lte2}
|(D^- -D)\psi(t_j)|\le \frac{\delta_j}{2}\max_{s\in
I_j}|\psi^{\prime\prime}(s)|,
\end{equation}
where $I_j=[t_{j-1},t_j]$.

\section{Error estimate}
We now establish the error estimate by generalizing the approach
based on Shishkin decompositons used in \cite{BAIL2008}. For a
reaction-diffusion boundary value problem in the special case $n=2$
a parameter uniform numerical method was analyzed in \cite{MS} by a
similar technique and in the general case in
\cite{LM} using discrete Green's functions.\\

We estimate the smooth component of the local truncation error in
the following lemma.
\begin{lemma} \label{smootherror} Let $A(t)$ satisfy (\ref{a1}) and (\ref{a2}). Then, for
each $i=1,\; \dots ,\; n$ and $j=1, \;\dots,\; N$, we have
\[ |\varepsilon_{i}(D^- -D)v_i(t_j)|\leq C N^{-1}.\]
\end{lemma}

\begin{proof} Using (\ref{lte2}), Lemma \ref{smooth} and (\ref{geom1}) we
obtain
\[|\varepsilon_{i}(D^{-}-D)v_i(t_j)| \leq C\delta_j
\max_{s \epsilon I_j}|\varepsilon_i v_i^{\prime\prime}(s)| \\
 \leq  C\delta_j \\
  \leq  C N^{-1}
\] as required.
\end{proof}

For the singular component we obtain a similar estimate, but in the
proof we must distinguish between the different types of mesh.  We
need the following preliminary lemmas.
\begin{lemma}\label{est1}  Let $A(t)$ satisfy (\ref{a1}) and (\ref{a2}). Then, for each $i=1,\;
\dots ,\; n$ and $j=1, \;\dots,\; N$, on each mesh $M_{\vec{b}}$, we
have the estimate  \[ |\varepsilon_i(D^- -D)w_i(t_j)| \leq
C\frac{\delta_j}{\varepsilon_1}. \]
\end{lemma}
\begin{proof} From (\ref{lte2}) and Lemma \ref{singular},  we have
\[
\begin{array}{lll} |\varepsilon_{i}(D^{-}-D)w_i(t_j)| & \leq &
C\delta_j
\max_{s \epsilon I_j}|\varepsilon_i w_i^{\prime\prime}(s)| \\
 & \leq & C\delta_j\sum_{q=1}^n \frac{B_{q}(t_{j-1})}{\varepsilon_q} \\
 & \leq &  C\frac{\delta_j}{\varepsilon_1}
\end{array}\] as required. \end{proof}

In what follows we make use of second degree polynomials of the
form
\[p_{i;\theta}=\sum_{k=0}^2
\frac{(t-t_{\theta})^k}{k!}w_{i}^{(k)}(t_{\theta}), \] where
$\theta$ denotes a pair of integers separated by a comma.

\begin{lemma}\label{general} Let $A(t)$ satisfy (\ref{a1}) and (\ref{a2}). Then, for each
$i=1,\; \dots ,\; n$, $j=1, \;\dots,\; N$ and $k=1,\;\;\dots,\;\;
n-1$, on each mesh $M_{\vec{b}}$ with $b_k =1$, there exists a
decomposition \[ w_i=\sum_{m=1}^{k+1}w_{i,m}, \] for which we have
the following estimates for each $m$,  $1 \le m \le k$,
\[|\varepsilon_i
w_{i,m}^{\prime}(t)| \leq CB_m(t), \;\;|\varepsilon_i
w_{i,m}^{\prime\prime}(t)| \leq C\frac{B_{m}(t)}{\varepsilon_m}\] and
\[|\varepsilon_i
w_{i,k+1}^{\prime\prime}(t)| \leq
C\sum_{q=k+1}^{n}\frac{B_{q}(t)}{\varepsilon_q}.\]

Furthermore \[ |\varepsilon_i(D^{-}-D)w_i(t_j)| \leq
C(B_{k}(t_{j-1})+\frac{\delta_j}{\varepsilon_{k+1}}).\]
\end{lemma}
\begin{proof} Since $b_k =1$ we have $\varepsilon_{k}\leq \varepsilon_{k+1}/2$,  so
$t_{k,k+1} \in (0,T]$ and we can define the decomposition
\[w_i=\sum_{m=1}^{k+1}w_{i,m},\] where the components of the
decomposition are defined by
\[w_{i,k+1}=\left\{ \begin{array}{ll} p_{i;k,k+1} & {\rm on}\;\;[0,t_{k,k+1})\\
 w_i & {\rm otherwise} \end{array}\right. \]
and for each $m$,  $k \ge m \ge 2$,
\[w_{i,m}=\left\{ \begin{array}{ll} p_{i;m-1,m} & \rm{on} \;\; [0,t_{m-1,m})\\
w_i-\sum_{q=m+1}^{k+1} w_{i,q} & {\rm otherwise}
\end{array}\right. \]
and
\[w_{i,1}=w_i-\sum_{q=2}^{k+1} w_{i,q}\;\; \rm{on} \;\; [0,T]. \]
From the above definitions we note that for each $m$, $1 \leq m
\leq k$,
$w_{i,m}=0 \;\; \rm{on} \;\; [t_{m,m+1},T]$.\\
To establish the bounds on the second derivatives we observe that:

in $[t_{k,k+1},T]$, using Lemma \ref{singular} and $t \geq
t_{k,k+1}$, we obtain
\[|\varepsilon_i w_{i,k+1}^{\prime \prime}(t)| =|\varepsilon_i w_{i}^{\prime \prime}(t)| \leq
C\sum_{q=1}^n \frac{B_q(t)}{\varepsilon_q} \leq C\sum_{q=k+1}^n
\frac{B_q(t)}{\varepsilon_q};\]

in $[0, t_{k,k+1}]$, using Lemma \ref{singular} and $t \leq
t_{k,k+1}$ , we obtain
\[|\varepsilon_i w_{i,k+1}^{\prime \prime}(t)| =|\varepsilon_i w_{i}^{\prime
\prime}(t_{k,k+1})| \leq \sum_{q=1}^{n}
\frac{B_q(t_{k,k+1})}{\varepsilon_q} \leq \sum_{q=k+1}^{n}
\frac{B_q(t_{k,k+1})}{\varepsilon_q} \leq \sum_{q=k+1}^{n}
\frac{B_q(t)}{\varepsilon_q};\]

and for each $m=k, \;\; \dots \;\;,2$, we see that\\

in $[t_{m,m+1},T]$, $w_{i,m}^{\prime \prime}=0;$

in $[t_{m-1,m},t_{m,m+1}]$, using Lemma \ref{singular}, we obtain
\[|\varepsilon_i w_{i,m}^{\prime \prime}(t)| \leq |\varepsilon_i w_{i}^{\prime
\prime}(t)|+\sum_{q=m+1}^{k+1}|\varepsilon_i w_{i,q}^{\prime \prime}(t)|
\leq C\sum_{q=1}^n \frac{B_q(t)}{\varepsilon_q} \leq
C\frac{B_m(t)}{\varepsilon_m};\]

in $[0, t_{m-1,m}]$, using Lemma \ref{singular} and $t \leq
t_{m-1,m}$, we obtain
\[|\varepsilon_i w_{i,m}^{\prime \prime}(t)| =|\varepsilon_i w_{i}^{\prime
\prime}(t_{m-1,m})| \leq C\sum_{q=1}^n \frac{B_q(t_{m-1,m})}{\varepsilon_q}
\leq C\frac{B_m(t_{m-1,m})}{\varepsilon_m} \leq C\frac{B_m(t)}{\varepsilon_m};
\]

in $[t_{1,2},T],\;\; w_{i,1}^{\prime \prime}=0;$

in $[0, t_{1,2}]$, using Lemma \ref{singular}, \[|\varepsilon_i
w_{i,1}^{\prime \prime}(t)| \leq |\varepsilon_i w_{i}^{\prime
\prime}(t)|+\sum_{q=2}^{k+1}|\varepsilon_i w_{i,q}^{\prime \prime}(t)|\leq
C\sum_{q=1}^n \frac{B_q(t)}{\varepsilon_q} \leq C\frac{B_1(t)}{\varepsilon_1}.\]

For the bounds on the first derivatives we observe that for each
$m$, $1 \leq m \leq k $ :

in $[t_{m,m+1},T],\;\; w_{i,m}^{\prime}=0;$

in $[0, t_{m,m+1}]\;\; \int_t^{t_{m,m+1}}\varepsilon_i w_{i,m}^{\prime
\prime}(s)ds= \varepsilon_i w_{i,m}^{\prime}(t_{m,m+1})- \varepsilon_i
w_{i,m}^{\prime}(t)= -\varepsilon_i w_{i,m}^{\prime}(t)$ \\
and so
\[|\varepsilon_i w_{i,m}^{\prime}(t)| \leq \int_t^{t_{m,m+1}}|\varepsilon_i
w_{i,m}^{\prime \prime}(s)|ds \leq
\frac{C}{\varepsilon_m}\int_{t}^{t_{m,m+1}} B_m(s)ds \leq CB_m(t).\]
Finally, since \[|\varepsilon_i(D^{-}-D)w_i(t_j)| \leq
\sum_{m=1}^{k}|\varepsilon_i(D^{-}-D)w_{i,m}(t_j)|+
|\varepsilon_i(D^{-}-D)w_{i,k+1}(t_j)|,\] using  (\ref{lte2}) on the last
term and (\ref{lte1}) on all other terms on the right hand side, we
obtain
\[|\varepsilon_i(D^{-}-D)w_i(t_j)| \leq C(\sum_{m=1}^{k}\max_{s \in
I_j}|\varepsilon_iw_{i,m}^\prime(s)| +\delta_j\max_{s \in
I_j}|\varepsilon_iw_{i,k+1}^{\prime \prime}(s)|).\]  The desired result
follows by applying the bounds on the derivatives in the first part
of this lemma. \end{proof}

\begin{lemma}\label{est3} Let $A(t)$ satisfy (\ref{a1}) and (\ref{a2}). Then, for each $i=1,\;
\dots ,\; n$ and $j=1, \;\dots,\; N$, on each mesh $M_{\vec{b}}$,
 we have the estimate \[
|\varepsilon_i(D^{-}-D)w_i(t_j)| \leq CB_n(t_{j-1}).\]
\end{lemma}
\begin{proof} From (\ref{lte1}) and Lemma \ref{singular}, for each $i=1,\;
\dots ,\; n$ and $j=1, \;\dots,\; N$, we have
\[
\begin{array}{lll} |\varepsilon_{i}(D^{-}-D)w_i(t_j)| & \leq &
C\max_{s \epsilon I_j}|\varepsilon_i w_i^{\prime}(s)| \\
 & \leq & C\varepsilon_{i}\sum_{q=i}^n \frac{B_{q}(t_{j-1})}{\varepsilon_q}\\
 & \leq &  CB_n(t_{j-1})
\end{array}\] as required.
\end{proof}

Using the above preliminary lemmas on appropriate subintervals we
obtain the desired estimate of the singular component of the local
truncation error in the following lemma.
\begin{lemma}\label{singularerror} Let $A(t)$ satisfy (\ref{a1}) and (\ref{a2}). Then, for each $i=1,\;
\dots ,\; n$ and $j=1, \;\dots,\; N$, we have the estimate
\[ |\varepsilon_{i}(D^- -D)w_i(t_j)| \leq CN^{-1}\ln N. \]
\end{lemma}
\begin{proof}  We consider each subinterval separately.\\
\noindent In the subinterval $(0,\sigma_1]$ we have $\delta_j \leq
CN^{-1}\sigma_1$. On any mesh $M_{\vec{b}}$, using Lemma
\ref{est1}, we get $|\varepsilon_i(D^{-}-D)w_i(t_j)| \leq
CN^{-1}\frac{\sigma_1}{\varepsilon_1}\leq CN^{-1}\ln N $.
\noindent In
the subinterval $(\sigma_1,\sigma_2]$ we have $\delta_j \leq
CN^{-1}\sigma_2.$

On any mesh $M_{\vec{b}}$ with $b_1=0$, we have
$\sigma_2=2\sigma_1$. Using Lemma \ref{est1} we get $|\varepsilon_{i}(D^-
-D)w_i(t_j)|\leq CN^{-1}\frac{\sigma_2}{\varepsilon_1}\leq
CN^{-1}\frac{\sigma_1}{\varepsilon_1}\leq CN^{-1}\ln N$.

On any mesh $M_{\vec{b}}$ with $b_1=1$, we have
$\sigma_1=\frac{\varepsilon_1}{\alpha}\ln N$. Using Lemma \ref{general}
with $k=1$ we get $|\varepsilon_{i}(D^- -D)w_i(t_j)|\leq
C(B_1(\sigma_1)+N^{-1}\frac{\sigma_2}{\varepsilon_2})\leq CN^{-1}\ln N$.

\noindent In a general subinterval $(\sigma_m,\sigma_{m+1}]\; 2 \leq
m \leq n-1,$ we have $\delta_j \leq CN^{-1}\sigma_{m+1}$.

On any mesh $M_{\vec{b}}$ with $b_q=0, \; q=1, \;\dots \;,m$, we
have $\sigma_{m+1}=C\sigma_1$. Using Lemma \ref{est1} we get
$|\varepsilon_{i}(D^- -D)w_i(t_j)|\leq
CN^{-1}\frac{\sigma_{m+1}}{\varepsilon_1}\leq
CN^{-1}\frac{\sigma_1}{\varepsilon_1}\leq CN^{-1}\ln N$.

On any mesh $M_{\vec{b}}$ with $ b_1=1, b_q=0, \; q=2, \;\dots
\;,m$, we have $\sigma_1=\frac{\varepsilon_1}{\alpha}\ln
N,\;\sigma_{m+1}=C\sigma_2$. Using Lemma \ref{general} with $k=1$ we
get $|\varepsilon_{i}(D^- -D)w_i(t_j)|\leq
C(B_1(\sigma_m)+N^{-1}\frac{\sigma_{m+1}}{\varepsilon_2})\leq
C(B_1(\sigma_1)+N^{-1}\frac{\sigma_2}{\varepsilon_2})\leq CN^{-1}\ln N$.

On any mesh $M_{\vec{b}}$ with $ b_k=1,\; b_q=0, \; q=k+1, \;\dots
\;,m$, we have $\sigma_k=\frac{\varepsilon_k}{\alpha}\ln
N,\;\sigma_{m+1}=C\sigma_{k+1}$. Using Lemma \ref{general} with
general $k$ we get $|\varepsilon_{i}(D^- -D)w_i(t_j)|\leq
C(B_k(\sigma_m)+N^{-1}\frac{\sigma_{m+1}}{\varepsilon_{k+1}})\leq
C(B_k(\sigma_k)+N^{-1}\frac{\sigma_{k+1}}{\varepsilon_{k+1}})\leq
CN^{-1}\ln N$.

On any mesh $M_{\vec{b}}$ with $ b_m=1$, we have
$\sigma_m=\frac{\varepsilon_m}{\alpha}\ln N$. Using Lemma \ref{est3} we get
$|\varepsilon_{i}(D^- -D)w_i(t_j)|\leq CN^{-1}B_m(\sigma_m)\leq CN^{-1}\ln
N$.

\noindent In the subinterval $(\sigma_n,T]$ we have $\delta_j \leq
CN^{-1}$.

On any mesh $M_{\vec{b}}$ with $b_q=0, \; q=1, \;\dots \;,n$, we
have $1/\varepsilon_1 \leq C\ln N$. Using Lemma \ref{est1} we get
$|\varepsilon_{i}(D^- -D)w_i(t_j)|\leq CN^{-1}/\varepsilon_1 \leq
 CN^{-1}\ln N$.

On any mesh $M_{\vec{b}}$ with $ b_1=1,\; b_q=0, \; q=2, \;\dots
\;,n$, we have $\sigma_1=\frac{\varepsilon_1}{\alpha}\ln N,\;1/ \varepsilon_2 \leq
C\ln N$. Using Lemma \ref{general} with $k=1$ we get $|\varepsilon_{i}(D^-
-D)w_i(t_j)|\leq C(B_1(\sigma_n)+N^{-1}\ /\varepsilon_2)\leq
C(B_1(\sigma_1)+N^{-1}\ /\varepsilon_2)\leq CN^{-1}\ln N$.

On any mesh $M_{\vec{b}}$ with $ b_k=1,\; b_q=0, \; q=k+1, \;\dots
\;,n,\; 2 \leq k \leq n-1$, we have
$\sigma_k=\frac{\varepsilon_k}{\alpha}\ln N,\;1/\varepsilon_{k+1} \leq C\ln N$.
Using Lemma \ref{general} with general $k$ we get $|\varepsilon_{i}(D^-
-D)w_i(t_j)|\leq C(B_k(\sigma_n)+N^{-1}/ \varepsilon_{k+1})\leq
C(B_k(\sigma_k)+N^{-1}/ \varepsilon_{k+1})\leq CN^{-1}\ln N$.

On any mesh $M_{\vec{b}}$ with $ b_n=1$, we have
$\sigma_n=\frac{\varepsilon_n}{\alpha}\ln N$. Using Lemma \ref{est3} we
get $|\varepsilon_{i}(D^- -D)w_i(t_j)|\leq CN^{-1}B_n(\sigma_n)\leq
CN^{-1}\ln N$.

It is not hard to verify that on each of the $n+1$ subintervals we
have obtained the required estimate for all of the $2^n$ possible
meshes. \end{proof}

 Let $\vec u$ denote the exact solution of
(\ref{IVP}) and $\vec U$ the discrete solution. Then, the main
result of this paper is the following $\varepsilon$-uniform error
estimate
\begin{theorem} Let $A(t)$ satisfy (\ref{a1}) and (\ref{a2}). Then
 there exists a constant $C$ such that \[\parallel\vec{U}-\vec{u}\parallel \leq CN^{-1}\ln
 N,
\] for all $N > 1$
\end{theorem}
\begin{proof} This follows immediately by applying Lemmas
\ref{smootherror} and \ref{singularerror} to
(\ref{triangleinequality}) and using Lemma \ref{dstab}.  \end{proof}
\section*{Acknowledgments}
The first author acknowledges the support of the UGC, New Delhi,
India  under the Minor Research Project-X Plan period.

Numerical Analysis (math.NA); Dynamical Systems (math.DS)


Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced

Abstract: We study the implicit regularization imposed by gradient descent for learning multi-layer homogeneous functions including feed-forward fully connected and convolutional deep neural networks with linear, ReLU or Leaky ReLU activation. We rigorously prove that gradient flow (i.e. gradient descent with infinitesimal step size) effectively enforces the differences between squared norms across different layers to remain invariant without any explicit regularization. This result implies that if the weights are initially small, gradient flow automatically balances the magnitudes of all layers. Using a discretization argument, we analyze gradient descent with positive step size for the non-convex low-rank asymmetric matrix factorization problem without any regularization. Inspired by our findings for gradient flow, we prove that gradient descent with step sizes $\eta_t = O\left(t^{-\left( \frac12+\delta\right)} \right)$ ($0<\delta\le\frac12$) automatically balances two low-rank factors and converges to a bounded global optimum. Furthermore, for rank-$1$ asymmetric matrix factorization we give a finer analysis showing gradient descent with constant step size converges to the global minimum at a globally linear rate. We believe that the idea of examining the invariance imposed by first order algorithms in learning homogeneous models could serve as a fundamental building block for studying optimization for learning deep models.

\subsection{Fully Connected Neural Networks} \label{subsec:conserved-fully-connected}

We first formally define a fully connected feed-forward neural network with $N$ ($N\ge2$) layers.
Let $\mathbf{W}^{(h)} \in \mathbb{R}^{n_h \times n_{h-1}}$ be the weight matrix in the $h$-th layer, and define $\vect{w} = ( \mathbf{W}^{(h)} )_{h=1}^N $ as a shorthand of the collection of all the weights.
Then the function $f_{\vect{w}}: \mathbb{R}^d \to \mathbb{R}^p$ ($d = n_0, p = n_N$) computed by this network can be defined recursively: $f_{\vect{w}}^{(1)}(\vect{x}) = \mathbf{W}^{(1)}\vect{x}$, $f_{\vect{w}}^{(h)}(\vect{x}) = \mathbf{W}^{(h)} \phi_{h-1}(f_{\vect{w}}^{(h-1)}(\vect{x}))$ ($h = 2, \ldots, N$), and $f_{\vect{w}}(\vect{x}) = f_{\vect{w}}^{(N)}(\vect{x})$, where each $\phi_h$ is an activation function that acts coordinate-wise on vectors.\footnote{We omit the trainable bias weights in the network for simplicity, but our results can be directly generalized to allow bias weights.}
We assume that each $\phi_h$ ($h\in[N-1]$) is \emph{homogeneous}, namely, $\phi_h(x) = \phi_h'(x)\cdot x$ for all $x$ and all elements of the sub-differential $\phi_h'(\cdot) $ when $\phi_h$ is non-differentiable at $x$.
This property is satisfied by functions like ReLU $\phi(x) = \max\{x, 0\}$, Leaky ReLU $\phi(x) = \max\{x, \alpha x\}$ ($0<\alpha<1$), and linear function $\phi(x) = x$.

Let $\ell: \mathbb{R}^p \times \mathbb{R}^p \to \mathbb{R}_{\ge0}$ be a differentiable loss function. Given a training dataset $\left\{ (\vect{x}_i, \vect{y}_i) \right\}_{i=1}^m \subset \mathbb{R}^d\times\mathbb{R}^p$, the training loss as a function of the network parameters $\vect{w}$ is defined as \begin{align}
L(\vect{w}) = \frac1m \sum_{i=1}^m \ell\left( f_{\vect{w}}(\vect{x}_i), \vect{y}_i \right). \label{eqn:nn_loss}
\end{align}

We consider gradient descent with infinitesimal step size (also known as gradient flow) applied on $L(\vect{w})$, which is captured by the differential inclusion:
\begin{equation} \label{eqn:gf-nn}
\frac{\d\mathbf{W}^{(h)}}{\mathrm{d}t} \in - \frac{\partial L(\vect{w})}{\partial \mathbf{W}^{(h)}}, \qquad h = 1, \ldots, N,
\end{equation}
where $t$ is a continuous time index, and $\frac{\partial L(\vect{w})}{\partial \mathbf{W}^{(h)}}$ is the Clarke sub-differential \citep{clarke2008nonsmooth}. If curves  ${\mathbf{W}}^{(h)} = {\mathbf{W}}^{(h)} (t)$ ($h\in[N]$) evolve with time according to \eqref{eqn:gf-nn} they are said to be a solution of the gradient flow differential inclusion.

Our main result in this section is the following invariance imposed by gradient flow.
\begin{thm}[Balanced incoming and outgoing weights at every neuron] \label{thm:conserved-neuron}
	For any $h\in[N-1]$ and $i\in[n_h]$, we have
	\begin{equation} \label{eqn:conserved-neuron}
	\frac{\d}{\mathrm{d}t} \left( \|\mathbf{W}^{(h)}[i, :]\|^2 - \|\mathbf{W}^{(h+1)}[:,i]\|^2 \right) = 0.
	\end{equation}
\end{thm}
Note that $\mathbf{W}^{(h)}[i, :]$ is a vector consisting of network weights coming into the $i$-th neuron in the $h$-th hidden layer, and $\mathbf{W}^{(h+1)}[:,i]$ is the vector of weights going out from the same neuron.
 Therefore, Theorem~\ref{thm:conserved-neuron} shows that gradient flow exactly preserves the difference between the squared $\ell_2$-norms of incoming weights and outgoing weights at any neuron.

Taking sum of \eqref{eqn:conserved-neuron} over $i\in[n_h]$, we obtain the following corollary which says gradient flow preserves the difference between the squares of Frobenius norms of weight matrices.
\begin{cor}[Balanced weights across layers] \label{cor:conserved-F-norm}
	For any $h\in[N-1]$, we have
	\begin{equation*}
	\frac{\d}{\mathrm{d}t} \left( \|\mathbf{W}^{(h)}\|_F^2 - \|\mathbf{W}^{(h+1)}\|_F^2 \right) = 0.
	\end{equation*}
\end{cor}

Corollary~\ref{cor:conserved-F-norm} explains why in practice, trained multi-layer models usually have similar magnitudes on all the layers:
if we use a small initialization,  $\|\mathbf{W}^{(h)}\|_F^2 - \|\mathbf{W}^{(h+1)}\|_F^2$ is very small at the beginning, and Corollary~\ref{cor:conserved-F-norm} implies this difference remains small at all time.
This finding also partially explains why gradient descent converges.
Although the objective function like \eqref{eqn:nn_loss} may not be smooth over the entire parameter space, given that $\|\mathbf{W}^{(h)}\|_F^2 - \|\mathbf{W}^{(h+1)}\|_F^2$ is small for all $h$, the objective function may have smoothness.
Under this condition, standard theory shows that gradient descent converges.
We believe this finding serves as a key building block for understanding first order methods for training deep neural networks.

For linear activation, we have the following stronger invariance than Theorem~\ref{thm:conserved-neuron}:
\begin{thm}[Stronger balancedness property for linear activation] \label{thm:conserved-linear}
	If for some $h\in[N-1]$ we have $\phi_h(x) = x$, then
	\begin{equation*}
	\frac{\d}{\mathrm{d}t} \left( \mathbf{W}^{(h)} (\mathbf{W}^{(h)})^\top  - (\mathbf{W}^{(h+1)})^\top \mathbf{W}^{(h+1)} \right) = \mathbf{0}.
	\end{equation*}
\end{thm}
This result was known for linear networks \citep{arora2018optimization}, but the proof there relies on the entire network being linear while Theorem~\ref{thm:conserved-linear}  only needs two consecutive layers to have no nonlinear activations in between.

While Theorem~\ref{thm:conserved-neuron} shows the invariance in a node-wise manner, Theorem~\ref{thm:conserved-linear} shows for linear activation, we can derive a layer-wise invariance.
Inspired by this strong invariance, in Section~\ref{sec:mf} we prove gradient descent with positive step sizes preserves this invariance approximately for matrix factorization.

\subsection{Convolutional Neural Networks}
\label{subsec:cnn}
Now we show that the conservation property in Corollary~\ref{cor:conserved-F-norm} can be generalized to convolutional neural networks.
In fact, we can allow \emph{arbitrary sparsity pattern and weight sharing structure} within a layer; convolutional layers are a special case.

\paragraph{Neural networks with sparse connections and shared weights.}
We use the same notation as in Section~\ref{subsec:conserved-fully-connected}, with the difference that some weights in a layer can be \emph{missing} or \emph{shared}.
Formally, the weight matrix $\mathbf W^{(h)} \in \mathbb{R}^{n_h\times n_{h-1}}$ in layer $h$ ($h\in [N]$) can be described by a vector $\vect{v}^{(h)} \in \mathbb{R}^{d_h}$  and a function $g_h: [n_h]\times[n_{h-1}] \to [d_h]\cup\{0\}$.
Here $\vect{v}^{(h)}$ consists of the actual \emph{free parameters} in this layer and $d_h$ is the number of free parameters (e.g. if there are $k$ convolutional filters in layer $h$ each with size $r$, we have $d_h = r\cdot k$).
The map $g_h$ represents the sparsity and weight sharing pattern:
\begin{align*}
\mathbf W^{(h)}[i, j] = \begin{cases}
0, & g_h(i, j) = 0, \\
\vect{v}^{(h)}[k], & g_h(i, j) =k > 0.
\end{cases}
\end{align*}
Denote by $\vect{v} = \left( \vect{v}^{(h)} \right)_{h=1}^N$ the collection of all the parameters in this network, and we consider gradient flow to learn the parameters:
\begin{equation*}
\frac{\d\vect{v}^{(h)}}{\mathrm{d}t} \in  - \frac{\partial L(\vect{v})}{\partial \vect{v}^{(h)}}, \qquad h = 1, \ldots, N.
\end{equation*}

The following theorem generalizes Corollary~\ref{cor:conserved-F-norm} to neural networks with sparse connections and shared weights:
\begin{thm}\label{thm:cnn}
	For any $h\in[N-1]$, we have
	\begin{equation*}
	\frac{\d}{\mathrm{d}t} \left( \|\vect{v}^{(h)}\|^2 - \|\vect{v}^{(h+1)}\|^2 \right) = 0.
	\end{equation*}
\end{thm}
Therefore, for a neural network with arbitrary sparsity pattern and weight sharing structure, gradient flow still balances the magnitudes of all layers.

\subsection{Proof of Theorem~\ref{thm:conserved-neuron}}
\label{subsec:proof_main}

The proofs of all theorems in this section are similar. They are based on the use of the chain rule (i.e. back-propagation) and the property of homogeneous activations.
Below we provide the proof of Theorem~\ref{thm:conserved-neuron} and defer the proofs of other theorems to Appendix~\ref{sec:proof-conserved}.

\begin{proof}[Proof of Theorem~\ref{thm:conserved-neuron}]
	First we note that we can without loss of generality assume $L$ is the loss associated with one data sample $(\vect{x}, \vect{y}) \in \mathbb{R}^d\times\mathbb{R}^p$, i.e., $L(\vect{w}) = \ell(f_{\vect{w}}(\vect{x}), \vect{y})$.
	In fact, for $L(\vect{w}) = \frac1m \sum_{k=1}^m L_k(\vect{w})$ where $L_k(\vect{w}) = \ell\left( f_{\vect{w}}(\vect{x}_k), \vect{y}_k \right)$, for any single weight $\mathbf W^{(h)}[i, j]$ in the network we can compute $\frac{\d}{\mathrm{d}t} (\mathbf{W}^{(h)}[i,j])^2 = 2 \mathbf{W}^{(h)}[i,j] \cdot \frac{\d \mathbf{W}^{(h)}[i,j]}{\mathrm{d}t} = -2 \mathbf{W}^{(h)}[i,j] \cdot \frac{\partial L(\vect{w})}{\partial \mathbf{W}^{(h)}[i,j]} = -2 \mathbf{W}^{(h)}[i,j] \cdot \frac1m \sum_{k=1}^m \frac{\partial L_k(\vect{w})}{\partial \mathbf{W}^{(h)}[i,j]}$, using the sharp chain rule of differential inclusions for tame functions \citep{drusvyatskiy2015curves,davis2018stochastic}.
	Thus, if we can prove the theorem for every individual loss $L_k$, we can prove the theorem for $L$ by taking average over $k\in[m]$.

	Therefore in the rest of proof we assume $L(\vect{w}) = \ell(f_{\vect{w}}(\vect{x}), \vect{y})$.
	For convenience, we denote $\vect{x}^{(h)} = f_{\vect{w}}^{(h)}(\vect{x})$ ($h\in[N]$), which is the input to the $h$-th hidden layer of neurons for $h\in[N-1]$ and is the output of the network for $h=N$.
	We also denote $\vect{x}^{(0)} = \vect{x}$ and $\phi_0(x)=x$ ($\forall x$).

	Now we prove \eqref{eqn:conserved-neuron}.
	Since $\mathbf{W}^{(h+1)}[k,i]$ ($k\in[n_{h+1}]$) can only affect $L(\vect{w})$ through $\vect{x}^{(h+1)}[k]$  , we have for $k\in[n_{h+1}]$,
	\begin{equation*}
	\frac{\partial L(\vect{w})}{ \partial \mathbf{W}^{(h+1)}[k,i] }
	=  \frac{\partial L(\vect{w})}{ \partial \vect{x}^{(h+1)}[k] } \cdot \frac{\partial \vect{x}^{(h+1)}[k]}{\partial \mathbf{W}^{(h+1)}[k,i]}
	= \frac{\partial L(\vect{w})}{ \partial \vect{x}^{(h+1)}[k] } \cdot \phi_{h}(\vect{x}^{(h)}[i]),
	\end{equation*}
	which can be rewritten as
	\begin{equation*}
	\frac{\partial L(\vect{w})}{ \partial \mathbf{W}^{(h+1)}[:,i] } = \phi_{h}(\vect{x}^{(h)}[i]) \cdot \frac{\partial L(\vect{w})}{ \partial \vect{x}^{(h+1)} }.
	\end{equation*}
	It follows that
	\begin{equation} \label{eqn:proof-conserved-neuron-1}
	\begin{aligned}
	\frac{\d}{\mathrm{d}t} \|\mathbf{W}^{(h+1)}[:,i]\|^2
	&= 2 \left\langle \mathbf{W}^{(h+1)}[:,i], \frac{\d}{\mathrm{d}t} \mathbf{W}^{(h+1)}[:,i]  \right\rangle = -2 \left\langle \mathbf{W}^{(h+1)}[:,i], \frac{\partial L(\vect{w})}{ \partial \mathbf{W}^{(h+1)}[:,i] }   \right\rangle \\

	= -2 \phi_{h}(\vect{x}^{(h)}[i]) \cdot \left\langle \mathbf{W}^{(h+1)}[:,i],  \frac{\partial L(\vect{w})}{ \partial \vect{x}^{(h+1)} }  \right\rangle.
	\end{aligned}
	\end{equation}
	On the other hand, $\mathbf{W}^{(h)}[i, :]$ only affects $L(\vect{w})$ through $\vect x^{(h)}[i]$. Using the chain rule, we get
	\begin{align*}
	\frac{ \partial L(\vect{w}) }{ \partial \mathbf{W}^{(h)}[i, :] }
	&= \frac{ \partial L(\vect{w}) }{ \partial \vect x^{(h)}[i] } \cdot \phi_{h-1} (\vect x^{(h-1)})= \left\langle \frac{ \partial L(\vect{w}) }{ \partial \vect x^{(h+1)} } , \mathbf{W}^{(h+1)}[:, i] \right\rangle \cdot \phi_h'(\vect x^{(h)}[i]) \cdot \phi_{h-1} (\vect x^{(h-1)}),
	\end{align*}
	where $\phi'$ is interpreted as a set-valued mapping whenever it is applied at a non-differentiable point.\footnote{More precisely, the equalities  should be an inclusion whenever there is a sub-differential, but as we see in the next display the ambiguity in the choice of sub-differential does not affect later calculations.}

	It follows that\footnote{This holds for any choice of element of the sub-differential, since $\phi'(x) x = \phi(x)$ holds at $x=0$ for any choice of sub-differential.}
	\begin{align*}
&	\frac{\d}{\mathrm{d}t} 	\|\mathbf{W}^{(h)}[i, :]\|^2
= 2 \left\langle \mathbf{W}^{(h)}[i, :], \frac{\d}{\mathrm{d}t} \mathbf{W}^{(h)}[i, :]  \right\rangle
	= -2 \left\langle \mathbf{W}^{(h)}[i, :], \frac{\partial L(\vect{w})}{ \partial \mathbf{W}^{(h)}[i, :] }   \right\rangle \\
	 =\,& -2 \left\langle \frac{ \partial L(\vect{w}) }{ \partial \vect x^{(h+1)} } , \mathbf{W}^{(h+1)}[:, i] \right\rangle \cdot \phi_h'(\vect x^{(h)}[i]) \cdot \left\langle \mathbf{W}^{(h)}[i, :], \phi_{h-1} (\vect x^{(h-1)}) \right\rangle \\
	=\,& -2 \left\langle \frac{ \partial L(\vect{w}) }{ \partial \vect x^{(h+1)} } , \mathbf{W}^{(h+1)}[:, i] \right\rangle \cdot \phi_h'(\vect x^{(h)}[i]) \cdot \vect x^{(h)}[i] = -2 \left\langle \frac{ \partial L(\vect{w}) }{ \partial \vect x^{(h+1)} } , \mathbf{W}^{(h+1)}[:, i] \right\rangle \cdot \phi_{h}(\vect{x}^{(h)}[i]).
	\end{align*}
	Comparing the above expression to \eqref{eqn:proof-conserved-neuron-1}, we finish the proof.
\end{proof}

\section{Introduction}
\label{sec:intro}
\input{intro.tex}

\section{The Auto-Balancing Properties in Deep Neural Networks}
\label{sec:conserved}
\input{conserved.tex}

\section{Gradient Descent Converges to Global Minimum for Asymmetric Matrix Factorization}
\label{sec:mf}
\input{mf.tex}

\section{Empirical Verification}
\label{sec:exp}
\input{exp.tex}

\section{Conclusion and Future Work}
\label{sec:con}
\input{conclusion.tex}

\section*{Acknowledgements}
\label{sec:ack}
\input{ack.tex}

\subsection{Related Work}
\label{sec:rel} \input{rel.tex}

\subsection{Paper Organization}
The rest of the paper is organized as follows.
In Section~\ref{sec:conserved}, we present our main theoretical result on the implicit regularization property of gradient flow for optimizing neural networks.
In Section~\ref{sec:mf}, we analyze the dynamics of randomly initialized gradient descent for asymmetric matrix factorization problem with unregularized objective function~\eqref{eqn:intro_mf_obj}.
In Section~\ref{sec:exp}, we empirically verify the theoretical result in Section~\ref{sec:conserved}.
We conclude and list future directions in Section~\ref{sec:con}.
Some technical proofs are deferred to the appendix.

\subsection{Notation}
We use bold-faced letters for vectors and matrices.
For a vector $\vect x$, denote by $\vect x[i]$ its $i$-th coordinate.
For a matrix $\vect A$, we use $\mathbf A[i, j]$ to denote its $(i, j)$-th entry, and use $\mathbf A[i, :]$ and $\mathbf A[:, j]$ to denote its $i$-th row and $j$-th column, respectively (both as column vectors).
We use $\norm{\cdot}_2$ or $\norm{\cdot}$ to denote the Euclidean norm of a vector, and use $\norm{\cdot}_F$ to denote the Frobenius norm of a matrix.
We use $\langle \cdot, \cdot \rangle$ to denote the standard Euclidean inner product between two vectors or two matrices.
Let $[n] = \{1, 2, \ldots, n\}$.
\subsection{The General Rank-$r$ Case}

First we consider the general case of $r\ge1$.
Our main theorem below says that if we use a random small initialization $(\mathbf U_0, \mathbf V_0)$, and set step sizes $\eta_t$ to be appropriately small, then gradient descent \eqref{eqn:mf-gd-dynamics} will converge to a solution close to the global minimum of \eqref{eqn:mf}.
To our knowledge, this is the first result showing that gradient descent with random initialization directly solves the un-regularized asymmetric matrix factorization problem~\eqref{eqn:mf}.

\begin{thm}\label{thm:mf-main}
	Let $0<\epsilon < \norm{\mathbf M^*}_F$.
	Suppose we initialize the entries in $\mathbf U_0$ and $\mathbf V_0$ i.i.d. from $\mathcal{N}(0, \frac{\epsilon}{\mathrm{poly}(d)})$ ($d = \max\{d_1, d_2\}$), and run  \eqref{eqn:mf-gd-dynamics} with step sizes $\eta_t = \frac{\sqrt{\epsilon/r}}{100(t+1) \norm{\mathbf M^*}_F^{3/2}}$ ($t=0,1,\ldots$).\footnote{The dependency of $\eta_t$ on $t$ can be $\eta_t = \Theta\left( t^{-(1/2+\delta)} \right)$ for any constant $\delta \in (0, 1/2]$.}
	Then with high probability over the initialization, $\lim_{t\to\infty}(\mathbf U_t, \mathbf V_t) = (\bar{\mathbf U}, \bar{\mathbf V})$ exists and satisfies $\norm{\bar{\mathbf U} \bar{\mathbf V}^\top - \mathbf M^*}_F \le \epsilon$.
\end{thm}

\paragraph{Proof sketch of Theorem~\ref{thm:mf-main}.}
	First let's imagine that we are using infinitesimal step size in GD. Then according to Theorem~\ref{thm:conserved-linear} (viewing problem~\eqref{eqn:mf} as learning a two-layer linear network where the inputs are all the standard unit vectors in $\mathbb{R}^{d_2}$), we know that $\mathbf U^\top \mathbf U - \mathbf V^\top \mathbf V$ will stay invariant throughout the algorithm.
	Hence when $\mathbf U$ and $\mathbf V$ are initialized to be small, $\mathbf U^\top \mathbf U - \mathbf V^\top \mathbf V$ will stay small forever.
	Combined with the fact that the objective $f(\mathbf U, \mathbf V)$ is decreasing over time (which means $\mathbf U \mathbf V^\top$ cannot be too far from $\mathbf M^*$), we can show that $\mathbf U$ and $\mathbf V$ will always stay bounded.

	Now we are using positive step sizes $\eta_t$, so we no longer have the invariance of $\mathbf U^\top \mathbf U - \mathbf V^\top \mathbf V$.
	Nevertheless, by a careful analysis of the updates, we can still prove that $\mathbf U_t^\top \mathbf U_t - \mathbf V_t^\top \mathbf V_t$ is small, the objective $f(\mathbf U_t, \mathbf V_t)$ decreases, and $\mathbf U_t$ and $\mathbf V_t$ stay bounded.
	Formally, we have the following lemma:
	\begin{lem} \label{lem:mf-balance}
		With high probability over the initialization $(\mathbf U_0, \mathbf V_0)$, for all $t$ we have:
		\begin{enumerate}[(i)]
			\item Balancedness: $\norm{\mathbf U_t^\top \mathbf U_t - \mathbf V_t^\top \mathbf V_t}_F \le \epsilon$;
			\item Decreasing objective: $f(\mathbf U_{t}, \mathbf V_{t}) \le f(\mathbf U_{t-1}, \mathbf V_{t-1}) \le \cdots \le f(\mathbf U_{0}, \mathbf V_{0}) \le 2\norm{\mathbf M^*}_F^2$;
			\item Boundedness: $\norm{\mathbf U_{t}}_F^2 \le 5\sqrt{r} \norm{\mathbf M^*}_F, \norm{\mathbf V_t}_F^2 \le  5\sqrt{r} \norm{\mathbf M^*}_F$.
		\end{enumerate}
	\end{lem}

	Now that we know the GD algorithm automatically constrains $(\mathbf U_t, \mathbf V_t)$ in a bounded region, we can use the smoothness of $f$ in this region and a standard analysis of GD to show that $(\mathbf U_t, \mathbf V_t)$ converges to a stationary point $(\bar{\mathbf U}, \bar{\mathbf V})$ of $f$ (Lemma~\ref{lem:mf-convergence}).
	Furthermore, using the results of \citep{lee2016gradient, panageas2016gradient} we know that $(\bar{\mathbf U}, \bar{\mathbf V})$ is almost surely not a strict saddle point.
	Then the following lemma implies that $(\bar{\mathbf U}, \bar{\mathbf V})$ has to be close to a global optimum since we know $\norm{\bar{\mathbf U}^\top \bar{\mathbf U} - \bar{\mathbf V}^\top \bar{\mathbf V}}_F \le \epsilon$ from Lemma~\ref{lem:mf-balance} (i). This would complete the proof of Theorem~\ref{thm:mf-main}.

\begin{lem} \label{lem:mf-strict-saddle}
	Suppose $(\mathbf{U}, \mathbf{V})$ is a stationary point of $f$ such that $\norm{\mathbf U^\top \mathbf U - \mathbf V^\top \mathbf V}_F \le \epsilon$.
	Then either $\norm{\mathbf U \mathbf V^\top - \mathbf M^*}_F \le \epsilon$, or $(\mathbf{U}, \mathbf{V})$ is a strict saddle point of $f$.
\end{lem}

The full proof of Theorem~\ref{thm:mf-main} and the proofs of Lemmas~\ref{lem:mf-balance} and \ref{lem:mf-strict-saddle} are given in Appendix~\ref{sec:proof-mf}.

\subsection{The Rank-$1$ Case}

We have shown in Theorem~\ref{thm:mf-main} that GD with small and diminishing step sizes converges to a global minimum for matrix factorization.
Empirically, it is observed that a constant step size $\eta_t \equiv \eta$ is enough for GD to converge quickly to global minimum.
Therefore,
some natural questions are how to prove convergence of GD with a constant step size, how fast it converges, and how the discretization affects the invariance we derived in Section~\ref{sec:conserved}.

While these questions remain challenging for the general rank-$r$ matrix factorization, we resolve them for the case of $r=1$.
Our main finding is that with constant step size, the norms of two layers are always within a constant factor of each other (although we may no longer have the stronger balancedness property as in Lemma~\ref{lem:mf-balance}), and
 we utilize this property to prove the \emph{linear convergence} of GD to a global minimum.

When $r=1$, the asymmetric matrix factorization problem and its GD dynamics become
$$
\min_{\vect{u} \in \mathbb{R}^{d_1}, \vect{v} \in \mathbb{R}^{d_2}} \frac12 \norm{\vect{u}\vect{v}^\top - \mathbf{M}^*}_F^2
$$
and
\begin{align*}
\vect{u}_{t+1} = \vect{u}_t - \eta (\vect{u}_t\vect{v}_t^\top - \mathbf{M}^*)\vect{v}_t, \qquad
\vect{v}_{t+1} = \vect{v}_t - \eta\left(\vect{v}_t\vect{u}_t^\top - \mathbf{M}^{*\top} \right)\vect{u}_t.
\end{align*}
Here we assume $\mathbf{M}^*$ has rank $1$, i.e., it can be factorized as $\mathbf{M}^* = \sigma_1 \vect{u}^*\vect{v}^{*\top}$ where $\vect{u}^*$ and $\vect{v}^*$ are unit vectors and $\sigma_1>0$.

Our main theoretical result is the following.
\begin{thm}[Approximate balancedness and linear convergence of GD for rank-$1$ matrix factorization]
	\label{thm:rank_1}
	Suppose $\vect{u}_0 \sim \mathcal{N}(\vect 0,\delta\mathbf{I})$, $\vect{v}_0 \sim \mathcal{N}(\vect 0,\delta \mathbf I)$ with $\delta = c_{init} \sqrt{\frac{\sigma_1}{d}}  $ ($d = \max\{d_1, d_2\}$)
	 for some sufficiently small constant $c_{init} >0$,

	 and $\eta = \frac{c_{step}}{\sigma_1}$ for some sufficiently small constant $ c_{step} >0$.
	Then with constant probability over the initialization, for all $t$ we have $c_0\le \frac{\abs{\vect{u}_t^\top \vect u^*}}{\abs{\vect{v}_t^\top \vect{v}^*}}\le C_0$ for some universal constants $ c_0,C_0>0$.
	Furthermore, for any $0<\epsilon< 1$, after $t = O\left( \log\frac{d}{\epsilon} \right)$ iterations, we have $\norm{\vect{u}_t\vect{v}_t^\top - \mathbf{M}^*}_F \le \epsilon \sigma_1$.
\end{thm}
Theorem~\ref{thm:rank_1} shows for $\vect{u}_t$ and $\vect{v}_t$, their strengths in the signal space, $\abs{\vect{u}_t^\top \vect{u}^*}$ and $\abs{\vect{v}_t^\top \vect{v}^*}$, are of the same order.
This approximate balancedness helps us prove the linear convergence of GD.
We refer readers to Appendix~\ref{sec:proof-rank-1} for the proof of Theorem~\ref{thm:rank_1}.

\subsection{Proof of Lemma~\ref{lem:mf-balance}}

Recall the following three properties we want to prove in Lemma~\ref{lem:mf-balance}, which we call $\mathcal{A}(t)$, $\mathcal{B}(t)$ and $\mathcal{C}(t)$, respectively:
\begin{align*}
	\mathcal{A}(t):\qquad & \norm{\mathbf U_t^\top \mathbf U_t - \mathbf V_t^\top \mathbf V_t}_F \le \epsilon, \\
	\mathcal{B}(t):\qquad & f(\mathbf U_{t}, \mathbf V_{t}) \le f(\mathbf U_{t-1}, \mathbf V_{t-1}) \le \cdots \le f(\mathbf U_{0}, \mathbf V_{0}) \le 2 \norm{\mathbf M^*}_F^2, \\
	\mathcal{C}(t):\qquad & \norm{\mathbf U_{t}}_F^2 \le  5\sqrt{r} \norm{\mathbf M^*}_F, \norm{\mathbf V_t}_F^2 \le  5\sqrt{r} \norm{\mathbf M^*}_F.
\end{align*}

We use induction to prove these statements.
For $t=0$, we can make the Gaussian variance in the initialization sufficiently small such that with high probability we have $$\norm{\mathbf U_{0}}_F^2 \le  \epsilon,\qquad \norm{\mathbf V_0}_F^2 \le  \epsilon, \qquad \norm{\mathbf U_0^\top \mathbf U_0 - \mathbf V_0^\top \mathbf V_0}_F \le \frac\epsilon2.$$
From now on we assume they are all satisfied.
Then $\mathcal{A}(0)$ is already satisfied, $\mathcal{C}(0)$ is satisfied because $\epsilon < \norm{\mathbf M^*}_F$,
and $\mathcal{B}(0)$ can be verified by $f(\mathbf U_{0}, \mathbf V_{0}) = \frac12 \norm{\mathbf U_0 \mathbf V_0^\top - \mathbf M^*}_F^2 \le \norm{\mathbf U_0 \mathbf V_0^\top}_F^2 + \norm{\mathbf M^*}_F^2 \le \norm{\mathbf U_0}_F^2 \norm{\mathbf V_0^\top}_F^2 + \norm{\mathbf M^*}_F^2 \le \epsilon^2+\norm{\mathbf M^*}_F^2\le 2\norm{\mathbf M^*}_F^2$.

To prove $\mathcal{A}(t)$, $\mathcal{B}(t)$ and $\mathcal{C}(t)$ for all $t$, we prove the following three claims. Since we have $\mathcal{A}(0)$, $\mathcal{B}(0)$ and $\mathcal{C}(0)$, if the following claims are all true, the proof will be completed by induction.
\begin{enumerate}[(i)]
	\item $\mathcal{B}(0), \ldots, \mathcal{B}(t), \mathcal{C}(0),  \ldots, \mathcal{C}(t) \implies \mathcal{A}(t+1)$;
	\item $\mathcal{B}(0), \ldots, \mathcal{B}(t), \mathcal{C}(t) \implies \mathcal{B}(t+1)$;
	\item $\mathcal{A}(t), \mathcal{B}(t) \implies \mathcal{C}(t)$.
\end{enumerate}

\begin{claim}
	$\mathcal{B}(0), \ldots, \mathcal{B}(t), \mathcal{C}(0),  \ldots, \mathcal{C}(t) \implies \mathcal{A}(t+1)$.
\end{claim}
\begin{proof}
	Using the update rule \eqref{eqn:mf-gd-dynamics} we can calculate
	\begin{align*}
	& \mathbf U_{t+1}^\top \mathbf U_{t+1} - \mathbf V_{t+1}^\top \mathbf V_{t+1} \\
	=\, & \left(\mathbf U_t - \eta_t (\mathbf U_t \mathbf V_t^\top - \mathbf M^*) \mathbf V_t\right)^\top \left(\mathbf U_t - \eta_t (\mathbf U_t \mathbf V_t^\top - \mathbf M^*) \mathbf V_t\right) \\ & - \left(\mathbf V_t - \eta_t (\mathbf U_t \mathbf V_t^\top - \mathbf M^*)^\top \mathbf U_t\right)^\top \left(\mathbf V_t - \eta_t (\mathbf U_t \mathbf V_t^\top - \mathbf M^*)^\top \mathbf U_t\right) \\
	=\, & \mathbf U_t^\top \mathbf U_t - \mathbf V_t^\top \mathbf V_t + \eta_t^2 \left( \mathbf V_t^\top \mathbf R_t^\top \mathbf R_t \mathbf V_t - \mathbf U_t^\top \mathbf R_t^\top \mathbf R_t \mathbf U_t \right),
	\end{align*}
	where $\mathbf R_t = \mathbf U_t \mathbf V_t^\top - \mathbf M^*$.
	Then we have
	\begin{equation} \label{eqn:mf-inproof-1}
	\begin{aligned}
	&\norm{\mathbf U_{t+1}^\top \mathbf U_{t+1} - \mathbf V_{t+1}^\top \mathbf V_{t+1}}_F\\
	\le\,& \norm{\mathbf U_t^\top \mathbf U_t - \mathbf V_t^\top \mathbf V_t}_F + \eta_t^2 \left( \norm{\mathbf V_t^\top \mathbf R_t^\top \mathbf R_t \mathbf V_t }_F + \norm{ \mathbf U_t^\top \mathbf R_t^\top \mathbf R_t \mathbf U_t}_F \right) \\
	\le\,& \norm{\mathbf U_t^\top \mathbf U_t - \mathbf V_t^\top \mathbf V_t}_F + \eta_t^2 \left( \norm{\mathbf V_t}_F^2 \norm{\mathbf R_t}_F^2  + \norm{\mathbf U_t}_F^2 \norm{\mathbf R_t}_F^2  \right) \\
	=\,& \norm{\mathbf U_t^\top \mathbf U_t - \mathbf V_t^\top \mathbf V_t}_F + 2\eta_t^2 \left( \norm{\mathbf V_t}_F^2   + \norm{\mathbf U_t}_F^2   \right) f(\mathbf U_t, \mathbf V_t) \\
	\le\,& \norm{\mathbf U_t^\top \mathbf U_t - \mathbf V_t^\top \mathbf V_t}_F + 2\eta_t^2 \cdot 10\sqrt{r} \norm{\mathbf M^*}_F \cdot 2 \norm{\mathbf M^*}_F^2,
	\end{aligned}
	\end{equation}
	where the last line is due to $\mathcal{B}(t)$ and $\mathcal{C}(t)$.

	Since we have $\mathcal{B}(t')$ and $\mathcal{C}(t')$ for all $t' \le t$, \eqref{eqn:mf-inproof-1} is still true when substituting $t$ with any $t'\le t$. Summing all of them and noting $\norm{\mathbf U_0^\top \mathbf U_0 - \mathbf V_0^\top \mathbf V_0}_F \le \frac{\epsilon}{2}$, we get
	\begin{align*}
	&\norm{\mathbf U_{t+1}^\top \mathbf U_{t+1} - \mathbf V_{t+1}^\top \mathbf V_{t+1}}_F \\
	\le\,& \norm{\mathbf U_0^\top \mathbf U_0 - \mathbf V_0^\top \mathbf V_0}_F + 40\sqrt{r} \norm{\mathbf M^*}_F^3 \sum_{i=0}^t \eta_i^2 \\
	\le\,& \frac\epsilon2 + 40\sqrt{r} \norm{\mathbf M^*}_F^3 \sum_{i=0}^t \frac{1}{(i+1)^2} \cdot \frac{\epsilon/r}{100^2\norm{\mathbf M^*}_F^3}\\
	\le\,&\epsilon.
	\end{align*}
	Therefore we have proved $\mathcal{A}(t+1)$.
\end{proof}

\begin{claim}
	$\mathcal{B}(0), \ldots, \mathcal{B}(t), \mathcal{C}(t) \implies \mathcal{B}(t+1)$.
\end{claim}
\begin{proof}
	Note that we only need to show $f(\mathbf U_{t+1}, \mathbf V_{t+1}) \le f(\mathbf U_{t}, \mathbf V_{t})$.
	We prove this using the standard analysis of gradient descent, for which we need the smoothness of the objective function $f$ (Lemma~\ref{lem:mf-local-smooth}).
	We first need to bound $\norm{\mathbf U_t}_F$, $\norm{\mathbf V_t}_F$, $\norm{\mathbf U_{t+1}}_F$ and $\norm{\mathbf V_{t+1}}_F$. We know from $\mathcal{C}(t)$ that $\norm{\mathbf U_{t}}_F^2 \le  5\sqrt{r} \norm{\mathbf M^*}_F$ and $\norm{\mathbf V_t}_F^2 \le  5\sqrt{r} \norm{\mathbf M^*}_F$.
	We can also bound $\norm{\mathbf U_{t+1}}_F^2$ and $\norm{\mathbf V_{t+1}}_F^2$ easily from the GD update rule:
	\begin{align*}
		&\norm{\mathbf U_{t+1}}_F^2 \\
		=\,& \norm{\mathbf U_t - \eta_t (\mathbf U_t \mathbf V_t^\top - \mathbf M^*) \mathbf V_t}_F^2 \\
		\le\,& 2 \norm{\mathbf U_t}_F^2 + 2\eta_t^2 \norm{\mathbf U_t \mathbf V_t^\top - \mathbf M^*}_F^2 \norm{\mathbf V_t}_F^2 \\
		\le\,& 2\cdot 5\sqrt{r} \norm{\mathbf M^*}_F + 2 \eta_t^2 \cdot 2f(\mathbf U_t, \mathbf V_t) \cdot 5\sqrt{r} \norm{\mathbf M^*}_F\\
		\le\,& 10\sqrt{r} \norm{\mathbf M^*}_F + 2 \cdot \frac{\epsilon/r}{100^2 (t+1)^2 \norm{\mathbf M^*}_F^3} \cdot 4\norm{\mathbf M^*}_F^2 \cdot 5\sqrt{r} \norm{\mathbf M^*}_F & \text{(using $\mathcal{B}(t)$)} \\
		\le \,& 10\sqrt{r} \norm{\mathbf M^*}_F + \frac{\epsilon}{100} \\
		\le\, & 11\sqrt{r} \norm{\mathbf M^*}_F. & \text{(using $\epsilon < \norm{\mathbf M^*}_F$)}
	\end{align*}
	Let $\beta =  (66\sqrt{r}+2) \norm{\mathbf M^*}_F $.
	 From Lemma~\ref{lem:mf-local-smooth}, $f$ is $\beta$-smooth over $\mathcal{S} = \{(\mathbf U, \mathbf V): \norm{\mathbf U}_F^2 \le 11\sqrt{r} \norm{\mathbf M^*}_F, \norm{\mathbf V}_F^2 \le 11\sqrt{r} \norm{\mathbf M^*}_F \}$.
	 Also note that $\eta_t < \frac{1}{\beta}$ by our choice. Then using smoothness we have
	 \begin{equation} \label{eqn:mf-obj-decrease}
	 \begin{aligned}
	 &f(\mathbf U_{t+1}, \mathbf V_{t+1}) \\
	 \le\,& f(\mathbf U_{t}, \mathbf V_{t}) + \left\langle  \nabla f(\mathbf U_t, \mathbf V_t), \begin{pmatrix}
		\mathbf U_{t+1}\\ \mathbf V_{t+1}
	 	\end{pmatrix} - \begin{pmatrix}
	 	\mathbf U_{t}\\ \mathbf V_{t}
	 	\end{pmatrix} \right\rangle + \frac{\beta}{2} \norm{ \begin{pmatrix}
	 		\mathbf U_{t+1}\\ \mathbf V_{t+1}
	 		\end{pmatrix} - \begin{pmatrix}
	 		\mathbf U_{t}\\ \mathbf V_{t}
	 		\end{pmatrix}  }_F^2 \\
 	=\,& f(\mathbf U_{t}, \mathbf V_{t}) - \eta_t \norm{ \nabla f(\mathbf U_t, \mathbf V_t) }_F^2 + \frac{\beta}{2}\eta_t^2 \norm{ \nabla f(\mathbf U_t, \mathbf V_t) }_F^2 \\
 	\le\,& f(\mathbf U_{t}, \mathbf V_{t}) - \frac{\eta_t}{2} \norm{ \nabla f(\mathbf U_t, \mathbf V_t) }_F^2.
	 \end{aligned}
	 \end{equation}
	 Therefore we have shown $\mathcal{B}(t+1)$.
\end{proof}

\begin{claim}
	$\mathcal{A}(t), \mathcal{B}(t) \implies \mathcal{C}(t)$.
\end{claim}
\begin{proof}
	From $\mathcal{B}(t)$ we know $\frac12 \norm{\mathbf U_t \mathbf V_t^\top - \mathbf M^*}_F^2 \le 2 \norm{\mathbf M^*}_F^2$ which implies $\norm{\mathbf U_t \mathbf V_t^\top}_F \le 3\norm{\mathbf M^*}_F$.
	Therefore it suffices to prove
	\begin{equation} \label{eqn:mf-inproof-toshow-1}
		 \norm{\mathbf U \mathbf V^\top}_F \le 3\norm{\mathbf M^*}_F,
		 \norm{\mathbf U^\top \mathbf U - \mathbf V^\top \mathbf V}_F \le \epsilon
		 \implies \norm{\mathbf U}_F^2 \le 5\sqrt{r} \norm{\mathbf M^*}_F, \norm{\mathbf V}_F^2 \le 5\sqrt{r} \norm{\mathbf M^*}_F.
	\end{equation}

	Now we prove \eqref{eqn:mf-inproof-toshow-1}.
	Consider the SVD $\mathbf U = \mathbf\Phi \mathbf\Sigma \mathbf{\Psi}^\top$, where $\mathbf \Phi \in \mathbb{R}^{d_1\times d_1}$ and $\mathbf{\Psi} \in \mathbb{R}^{r\times r}$ are orthogonal matrices, and $\mathbf\Sigma \in \mathbb{R}^{d_1\times r}$ is a diagonal matrix.
	Let $\sigma_i = \mathbf \Sigma[i, i]$ ($i\in[r]$) which are all the singular values of $\mathbf U$.
	Define $\widetilde{\mathbf V} = \mathbf V \mathbf\Psi$. Then we have
	\begin{align*}
	3\norm{\mathbf M^*}_F
	\ge \norm{\mathbf U \mathbf V^\top}_F
	= \norm{\mathbf\Phi \mathbf\Sigma \mathbf{\Psi}^\top  \mathbf{\Psi} \widetilde{\mathbf V}^\top }_F
	= 	\norm{\mathbf \Sigma \widetilde{\mathbf V}^\top }_F
	= \sqrt{\sum_{i=1}^r \sigma_i^2 \norm{\widetilde{\mathbf V}[:,i]}^2}
	\end{align*}
	and
	\begin{align*}
	\epsilon &\ge  \norm{\mathbf U^\top \mathbf U - \mathbf V^\top \mathbf V}_F
	= \norm{ \mathbf\Psi \mathbf\Sigma^\top \mathbf{\Phi}^\top \mathbf\Phi \mathbf\Sigma \mathbf{\Psi}^\top - \mathbf{\Psi} \widetilde{\mathbf V}^\top \widetilde{\mathbf V} \mathbf{\Psi}^\top }_F
	= \norm{ \mathbf\Sigma^\top \mathbf\Sigma  -  \widetilde{\mathbf V}^\top \widetilde{\mathbf V} }_F \\
	&\ge \sqrt{\sum_{i=1}^r \left( \sigma_i^2  - \norm{\widetilde{\mathbf V}[:,i]}^2 \right)^2}.
	\end{align*}
	Using the above two inequalities we get
	\begin{align*}
	\sum_{i=1}^r \sigma_i^4
	&\le \sum_{i=1}^r \left( \sigma_i^4 + \norm{\widetilde{\mathbf V}[:,i]}^4 \right)
	= \sum_{i=1}^r \left( \sigma_i^2  - \norm{\widetilde{\mathbf V}[:,i]}^2 \right)^2 + 2 \sum_{i=1}^r \sigma_i^2 \norm{\widetilde{\mathbf V}[:,i]}^2 \\
	&\le \epsilon^2 + 2 \left(3\norm{\mathbf M^*}_F\right)^2
	\le 19 \norm{\mathbf M^*}_F^2.
	\end{align*}
	Then by the Cauchy-Schwarz inequality we have
	\begin{align*}
	\norm{\mathbf U}_F^2 = \sum_{i=1}^r \sigma_i^2
	\le \sqrt{r \sum_{i=1}^r \sigma_i^4}
	\le \sqrt{r\cdot 19 \norm{\mathbf M^*}_F^2}
	\le 5\sqrt{r} \norm{\mathbf M^*}_F.
	\end{align*}
	Similarly, we also have $\norm{\mathbf V}_F^2 \le 5\sqrt{r} \norm{\mathbf M^*}_F$. Therefore we have proved \eqref{eqn:mf-inproof-toshow-1}.
\end{proof}

\subsection{Convergence to a Stationary Point}

With the balancedness and boundedness properties in Lemma~\ref{lem:mf-balance}, it is then standard to show that $(\mathbf U_t, \mathbf V_t)$ converges to a stationary point of $f$.

\begin{lem} \label{lem:mf-convergence}
	Under the setting of Theorem~\ref{thm:mf-main}, with high probability $\lim_{t\to\infty}(\mathbf U_t, \mathbf V_t) = (\bar{\mathbf U}, \bar{\mathbf V})$ exists, and $(\bar{\mathbf U}, \bar{\mathbf V})$ is a stationary point of $f$.
	Furthermore, $(\bar{\mathbf U}, \bar{\mathbf V})$ satisfies $\norm{\bar{\mathbf U}^\top \bar{\mathbf U} - \bar{\mathbf V}^\top \bar{\mathbf V}}\le \epsilon$.
\end{lem}
\begin{proof}
	We assume the three properties in Lemma~\ref{lem:mf-balance} hold, which happens with high probability.
	Then from \eqref{eqn:mf-obj-decrease} we have
	\begin{equation}\label{eqn:mf-obj-decrease-2}
	\begin{aligned}
	f(\mathbf U_{t+1}, \mathbf V_{t+1})
	&\le f(\mathbf U_{t}, \mathbf V_{t}) - \frac{\eta_t}{2} \norm{ \nabla f(\mathbf U_t, \mathbf V_t) }_F^2 \\
	&= f(\mathbf U_{t}, \mathbf V_{t}) - \frac{1}{2} \norm{ \nabla f(\mathbf U_t, \mathbf V_t) }_F \norm{\begin{pmatrix}
		\mathbf U_{t+1} \\ \mathbf V_{t+1}
		\end{pmatrix} - \begin{pmatrix}
		\mathbf U_{t} \\ \mathbf V_{t}
		\end{pmatrix}}_F .
	\end{aligned}
	\end{equation}
	Under the above descent condition, the result of \cite{absil2005convergence} says that the iterates either diverge to infinity or converge to a fixed point.
	According to Lemma~\ref{lem:mf-balance}, \{$(\mathbf U_t, \mathbf V_t)\}_{t=1}^\infty$ are all bounded, so they have to converge to a fixed point $(\bar{\mathbf U}, \bar{\mathbf V})$ as $t\to\infty$.

	Next, from \eqref{eqn:mf-obj-decrease-2} we know that $\sum_{t=1}^\infty \frac{\eta_t}{2} \norm{ \nabla f(\mathbf U_t, \mathbf V_t) }_F^2 \le f(\mathbf U_0, \mathbf V_0)$ is bounded.
	Notice that $\eta_t$ scales like $1/t$. So we must have $\liminf_{t\to\infty} \norm{ \nabla f(\mathbf U_t, \mathbf V_t) }_F = 0$.
	Then according to the smoothness of $f$ in a bounded region (Lemma~\ref{lem:mf-local-smooth}) we conclude $ \nabla f(\bar{\mathbf U}, \bar{\mathbf V})  = \mat0$, i.e., $(\bar{\mathbf U}, \bar{\mathbf V})$ is a stationary point.

	The second part of the lemma is evident according to Lemma~\ref{lem:mf-balance} (i).
\end{proof}

\subsection{Proof of Lemma~\ref{lem:mf-strict-saddle}}

The main idea in the proof is similar to \cite{ge2017no}. We want to find a direction $\mathbf \Delta$ such that either $[\nabla^2 f(\mathbf U, \mathbf V)] (\mathbf \Delta, \mathbf \Delta)$ is negative or $(\mathbf U, \mathbf V)$ is close to a global minimum. We show that this is possible when $\norm{\mathbf U^\top \mathbf U - \mathbf V^\top \mathbf V}_F \le \epsilon$.

First we define some notation.
Take the SVD $\mathbf M^* = \mathbf \Phi^* \mathbf \Sigma^* \mathbf \Psi^{*\top}$, where $\mathbf \Phi^* \in \mathbb{R}^{d_1\times r}$ and $\mathbf \Psi^* \in \mathbb{R}^{d_2\times r}$ have orthonormal columns and $\mathbf \Sigma^* \in \mathbb{R}^{r\times r}$ is diagonal.
Denote $\mathbf U^* = \mathbf \Phi^*  (\mathbf \Sigma^*)^{1/2}$ and $\mathbf V^* = \mathbf \Psi^*  (\mathbf \Sigma^*)^{1/2}$.
Then we have $\mathbf U^* \mathbf V^{*\top} = \mathbf M^*$ (i.e., $(\mathbf U^*, \mathbf V^*)$ is a global minimum) and $\mathbf U^{*\top} \mathbf U^* = \mathbf V^{*\top} \mathbf V^*$.

Let $\mathbf M = \mathbf U \mathbf V^\top$, $\mathbf W = \begin{pmatrix}
\mathbf U\\ \mathbf V
\end{pmatrix}$ and $\mathbf W^* = \begin{pmatrix}
\mathbf U^*\\ \mathbf V^*
\end{pmatrix}$.
Define $$\mathbf{R} = \mathrm{argmin}_{\mathbf R' \in \mathbb{R}^{r\times r} \text{, orthogonal}} \norm{\mathbf W - \mathbf W^* \mathbf R'}_F$$
and
\[
\mathbf\Delta = \mathbf W - \mathbf W^* \mathbf R.
\]

We will show that $\mathbf\Delta$ is the desired direction.
Recall \eqref{eqn:mf-hessian}:
\begin{equation} \label{eqn:mf-hessian-to-bound}
\begin{aligned}  \
[\nabla^2 f(\mathbf U, \mathbf V)] (\mathbf \Delta, \mathbf \Delta)
= 2 \left\langle \mathbf M - \mathbf M^*, \mathbf\Delta_{\mathbf U} \mathbf{\Delta}_{\mathbf V}^\top \right\rangle + \norm{\mathbf U \mathbf\Delta_{\mathbf V}^\top  + \mathbf\Delta_{\mathbf U} \mathbf V^\top }_F^2,
\end{aligned}
\end{equation}
where $ \mathbf \Delta = \begin{pmatrix}
\mathbf \Delta_{\mathbf U} \\ \mathbf{\Delta}_{\mathbf V}
\end{pmatrix},
\mathbf \Delta_{\mathbf U} \in \mathbb{R}^{d_1\times r}, \mathbf \Delta_{\mathbf V} \in \mathbb{R}^{d_2\times r}$.
We consider the two terms in \eqref{eqn:mf-hessian-to-bound} separately.

For the first term in \eqref{eqn:mf-hessian-to-bound}, we have:
\begin{claim} \label{claim:mf-hessian-to-bound-1}
	$\left\langle \mathbf M - \mathbf M^*, \mathbf\Delta_{\mathbf U} \mathbf{\Delta}_{\mathbf V}^\top \right\rangle  = -\norm{\mathbf M - \mathbf M^*}_F^2$.
\end{claim}
\begin{proof}
	Since $(\mathbf U, \mathbf V)$ is a stationary point of $f$, we have the first-order optimality condition:
	\begin{equation} \label{eqn:mf-inproof-first-order-opt-cond}
	\frac{\partial f(\mathbf U, \mathbf V)}{\partial \mathbf U} = (\mathbf M - \mathbf M^*) \mathbf V = \mat0, \qquad
	\frac{\partial f(\mathbf U, \mathbf V)}{\partial \mathbf V} = (\mathbf M - \mathbf M^*)^\top \mathbf U = \mat0.
	\end{equation}

Note that $\mathbf \Delta_{\mathbf U} = \mathbf U - \mathbf U^* \mathbf R$ and $\mathbf \Delta_{\mathbf V} = \mathbf V - \mathbf V^* \mathbf R$.
We have
\begin{align*}
&\left\langle \mathbf M - \mathbf M^*, \mathbf\Delta_{\mathbf U} \mathbf{\Delta}_{\mathbf V}^\top \right\rangle \\
=\,& \left\langle \mathbf M - \mathbf M^*, (\mathbf U - \mathbf U^* \mathbf R) (\mathbf V - \mathbf V^* \mathbf R)^\top \right\rangle \\
=\,& \left\langle \mathbf M - \mathbf M^*, \mathbf M - \mathbf U^* \mathbf R \mathbf V^\top - \mathbf U \mathbf R^\top \mathbf V^{*\top} + \mathbf M^*  \right\rangle \\
=\,& \left\langle \mathbf M - \mathbf M^*, \mathbf M^*\right\rangle \\
=\,& \left\langle \mathbf M - \mathbf M^*, \mathbf M^* - \mathbf M \right\rangle \\
=\,& -\norm{\mathbf M - \mathbf M^*}_F^2,
\end{align*}
where we have used the following consequences of \eqref{eqn:mf-inproof-first-order-opt-cond}:
\begin{align*}
&\left\langle \mathbf M - \mathbf M^*, \mathbf M \right\rangle = \left\langle \mathbf M - \mathbf M^*, \mathbf U \mathbf V^\top \right\rangle = 0, \\
&\left\langle \mathbf M - \mathbf M^*, \mathbf U^* \mathbf R \mathbf V^\top \right\rangle  = 0, \\
&\left\langle \mathbf M - \mathbf M^*, \mathbf U \mathbf R^\top \mathbf V^{*\top}  \right\rangle  = 0.
\end{align*}
\end{proof}

The second term in \eqref{eqn:mf-hessian-to-bound} has the following upper bound:
\begin{claim} \label{claim:mf-hessian-to-bound-2}
$\norm{\mathbf U \mathbf\Delta_{\mathbf V} + \mathbf\Delta_{\mathbf U} \mathbf V}_F^2 \le \norm{\mathbf M - \mathbf M^*}_F^2 + \frac12 \epsilon^2 $.
\end{claim}
\begin{proof}
	We make use of the following identities, all of which can be directly verified by plugging in definitions:
	\begin{equation} \label{eqn:mf-inproof-identity-1}
	\mathbf U \mathbf\Delta_{\mathbf V}^\top + \mathbf\Delta_{\mathbf U} \mathbf V^\top = \mathbf\Delta_{\mathbf U} \mathbf\Delta_{\mathbf V}^\top + \mathbf M - \mathbf M^*,
	\end{equation}
	\begin{equation} \label{eqn:mf-inproof-identity-2}
	\norm{\mathbf{\Delta} \mathbf{\Delta}^\top}_F^2 = 4 \norm{\mathbf\Delta_{\mathbf U} \mathbf\Delta_{\mathbf V}^\top}_F^2 + \norm{\mathbf\Delta_{\mathbf U}^\top\mathbf\Delta_{\mathbf U} - \mathbf\Delta_{\mathbf V}^\top \mathbf\Delta_{\mathbf V}}_F^2,
	\end{equation}
	\begin{equation} \label{eqn:mf-inproof-identity-3}
	\begin{aligned}
	\norm{\mathbf W \mathbf W^\top - \mathbf W^* \mathbf W^{*\top}}_F^2 =\ & 4\norm{\mathbf M - \mathbf M^*}_F^2 - 2 \norm{\mathbf U^\top \mathbf U^* - \mathbf V^\top \mathbf V^*}_F^2 \\&+ \norm{\mathbf U^\top \mathbf U - \mathbf V^\top \mathbf V}_F^2 + \norm{\mathbf U^{*\top} \mathbf U^* - \mathbf V^{*\top} \mathbf V^*}_F^2.
	\end{aligned}
	\end{equation}
	We also need the following inequality, which is \citep[Lemma 6]{ge2017no}:
	\begin{equation} \label{eqn:mf-inproof-rong-ineq}
	\norm{\mathbf{\Delta} \mathbf{\Delta}^\top}_F^2 \le 2 \norm{\mathbf W \mathbf W^\top - \mathbf W^* \mathbf W^{*\top}}_F^2.
	\end{equation}

	Now we can prove the desired bound as follows:
	\begin{align*}
	& \norm{\mathbf U \mathbf\Delta_{\mathbf V} + \mathbf\Delta_{\mathbf U} \mathbf V}_F^2 \\
	=\, & \norm{\mathbf\Delta_{\mathbf U} \mathbf\Delta_{\mathbf V}^\top + \mathbf M - \mathbf M^*}_F^2 & (\eqref{eqn:mf-inproof-identity-1}) \\
	=\,& \norm{\mathbf\Delta_{\mathbf U} \mathbf\Delta_{\mathbf V}^\top }_F^2 + 2 \left\langle \mathbf M - \mathbf M^*, \mathbf\Delta_{\mathbf U} \mathbf{\Delta}_{\mathbf V}^\top \right\rangle  + \norm{\mathbf M - \mathbf M^*}_F^2 \\
	=\,& \norm{\mathbf\Delta_{\mathbf U} \mathbf\Delta_{\mathbf V}^\top }_F^2 - \norm{\mathbf M - \mathbf M^*}_F^2 & (\text{Claim~\ref{claim:mf-hessian-to-bound-1}})\\
	\le\, & \frac14 \norm{\mathbf{\Delta} \mathbf{\Delta}^\top}_F^2 - \norm{\mathbf M - \mathbf M^*}_F^2  & (\eqref{eqn:mf-inproof-identity-2}) \\
	\le\, & \frac12 \norm{\mathbf W \mathbf W^\top - \mathbf W^* \mathbf W^{*\top}}_F^2  - \norm{\mathbf M - \mathbf M^*}_F^2  & (\eqref{eqn:mf-inproof-rong-ineq}) \\
	=\, & 2\norm{\mathbf M - \mathbf M^*}_F^2 -  \norm{\mathbf U^\top \mathbf U^* - \mathbf V^\top \mathbf V^*}_F^2  + \frac12\norm{\mathbf U^\top \mathbf U - \mathbf V^\top \mathbf V}_F^2 \\& + \frac12\norm{\mathbf U^{*\top} \mathbf U^* - \mathbf V^{*\top} \mathbf V^*}_F^2 - \norm{\mathbf M - \mathbf M^*}_F^2 & (\eqref{eqn:mf-inproof-identity-3}) \\
	\le\,& \norm{\mathbf M - \mathbf M^*}_F^2 + \frac12 \epsilon^2,
	\end{align*}
	where in the last line we have used $\mathbf U^{*\top} \mathbf U^* = \mathbf V^{*\top} \mathbf V^*$ and $\norm{\mathbf U^{\top} \mathbf U - \mathbf V^{\top} \mathbf V} \le \epsilon$.
\end{proof}

Using Claims~\ref{claim:mf-hessian-to-bound-1} and \ref{claim:mf-hessian-to-bound-2}, we obtain an upper bound on \eqref{eqn:mf-hessian-to-bound}:
\begin{align*}
[\nabla^2 f(\mathbf U, \mathbf V)] (\mathbf \Delta, \mathbf \Delta) \le -\norm{\mathbf M - \mathbf M^*}_F^2 + \frac12 \epsilon^2.
\end{align*}
Therefore, we have either $\norm{\mathbf U \mathbf V^\top - \mathbf M^*}_F = \norm{\mathbf M - \mathbf M^*}_F \le \epsilon$ or $[\nabla^2 f(\mathbf U, \mathbf V)] (\mathbf \Delta, \mathbf \Delta) \le -\frac12 \epsilon^2 <0$.
In the latter case, $(\mathbf U, \mathbf V)$ is a strict saddle point of $f$.
This completes the proof of Lemma~\ref{lem:mf-strict-saddle}.

\subsection{Finishing the Proof of Theorem~\ref{thm:mf-main}}

Theorem~\ref{thm:mf-main} is a direct corollary of Lemma~\ref{lem:mf-convergence}, Lemma~\ref{lem:mf-strict-saddle}, and the fact that gradient descent does not converge to a strict saddle point almost surely \citep{lee2016gradient, panageas2016gradient}.

\subsection{Proof Sketch of Theorem~\ref{thm:rank_1}}
To analyze the convergence, we define four key quantities.
\begin{align*}
\alpha_t = \vect{u}^\top\vect{x}_t,\quad
\alpha_{t,\perp} = \norm{\vect{u}_\perp^\top\vect{x}_t}_2, \quad
\beta_{t} = \vect{v}^\top \vect{y}_t, \quad
\beta_{t,\perp} = \norm{\vect{v}_\perp^\top \vect{y}_t}_2
\end{align*}
where $\vect{u}_\perp^\top$ and $\vect{v}^\top_{\perp}$ represent the projection matrix onto the complement space of $\vect{u}$ and $\vect{v}$, respectively.
Notice that $\norm{\vect{x}_t}_2^2 = \alpha_t^2 + \alpha_{t,\perp}^2$ and $\norm{\vect{y}_t}_2^2 = \beta_t^2 + \beta_{t,\perp}^2$.
Note if eventually we find a global minimum, we have $\beta_{t,\perp} = 0, \alpha_{t,\perp}=0$ and $\alpha_t \beta_t = \sigma_1$.
It turns out we can write out an explicit formula for the dynamics of the these quantities.
\begin{align*}
\alpha_{t+1} =
\left(1-\eta\left(\beta_t^2 +\beta_{t,\perp}^2\right)\right)\alpha_t + \eta\sigma_1\beta_t, \quad
&\beta_{t+1} = \left(1-\eta\left(\alpha_t^2+\alpha_{t,\perp}^2\right)\right)\beta_t + \eta_1\sigma_1 \alpha_t\\
\alpha_{t+1,\perp} = \left(1-\eta\left(\beta_t^2+\beta_{t,\perp}^2\right)\right)\alpha_{t,\perp}, \quad
&\beta_{t+1,\perp} = \left(1-\eta\left(\alpha_t^2+\alpha_{t,\perp}^2\right)\right)\beta_{t,\perp}
\end{align*}

Since we use Gaussian initialization, by standard random matrix theory, with overwhelming probability, we know the following facts about these quantities \[
\alpha_0 \asymp \frac{c_{init}\sigma_1}{\sqrt{d}}, \quad \alpha_{0,\perp} \asymp c_{init} \sqrt{\sigma_1}, \quad \beta_0 \asymp\frac{c_{init}\sigma_1}{\sqrt{d}}, \quad\beta_{0,\perp} \asymp c_{init} \sqrt{\sigma_1}.
\]
We also assume signal at the beginning is positive: $\alpha_0 > 0, \beta_0 > 0.$, which holds with probability $0.5$ using Gaussian initialization.

To prove $c_0\le \frac{\abs{\vect{u^\top\vect{x}_t}}}{\abs{\vect{v}^\top \vect{y}_t}}\le C_0$ for all iterations, we cannot only analyze quantity $\frac{\abs{\vect{u^\top\vect{x}_t}}}{\abs{\vect{v}^\top \vect{y}_t}}$ along, the convergence rate to the global minimum also plays an important role in affecting $\frac{\abs{\vect{u^\top\vect{x}_t}}}{\abs{\vect{v}^\top \vect{y}_t}}$.
We divide the dynamics into two stages.
\begin{thm}[Stage 1: Escaping from $(\vect{
0},\vect{0})$ Saddle Point]\label{thm:rank_1_stage_1}
Let $T := \mathrm{argmin}_{t} \left\{\alpha_t^2 + \beta_t^2 \le c_1\sigma_1\right\}$ for $c_1 = 1/2$.
If $\eta \le \frac{c_{step}}{\sigma_1}$, then for $t=1,\ldots,T-1$, the followings hold\begin{itemize}
\item Magnitude in complement space remain small: $\xi_t \le \xi_0$;
\item Growth of magnitude in signal space: $\abs{\alpha_{t+1}+\beta_{t+1}} \ge \left(1+\frac{\eta\sigma_1}{3}\right) \abs{\alpha_t+\beta_t}$;
\item Bounded ratio between two layers: $\abs{\alpha_t - \beta_t} \le c_{diff}\abs{\alpha_t+\beta_t}$ where $c_{diff} \triangleq \frac{\abs{\alpha_0-\beta_0}}{\abs{\alpha_0+\beta_0}}$.
\end{itemize}
Furthermore, after at most $T_1 = O\left(\frac{1}{\eta\sigma_1}\log\left(\frac{\sigma_1}{\alpha_0+\beta_0}\right)\right)$ iterations, $T_1 \le T$, we have $\alpha_{T_1} \beta_{T_1} \ge c \sigma_1$ for some small constant $0 < c < 1$.
\end{thm}
In this stage, the strength in the noise space remains small ($\xi_t \le \xi_0$) and the strength in the signal space is growing ($\abs{\alpha_{t+1}+\beta_{t+1}} \ge \left(1+\frac{\eta\sigma_1}{3}\right) \abs{\alpha_t+\beta_t}$).
Further, because  $\abs{\alpha_t - \beta_t} \le c_{diff}\abs{\alpha_t+\beta_t}$, we know $\frac{1-c_{diff}}{1+c_{diff}}\le \frac{\alpha_t}{\beta_t} \le \frac{1+c_{diff}}{1-c_{diff}}$, which is our desired result.
Here we consider $\abs{\alpha_t - \beta_t} / \abs{\alpha_t - \beta_t}$ instead of directly bounding $\frac{\alpha_t}{\beta_t}$ because the dynamics of $\abs{\alpha_t - \beta_t} / \abs{\alpha_t - \beta_t}$ is much simpler to analyze.
This will be apparent in the proof.

Now we enter stage 2, which is essentially a local convergence phase.
The following theorem characterizes the behaviors of the strength in the signal and in the noise space in this stage.
\begin{thm}[Stage 2: Local Convergence]\label{thm:rank_1_stage_2}

Suppose at $T_1$-th iteration, the followings hold \begin{align*}
\alpha_{T_1}\beta_{T_1} \ge c\sigma_1, \quad
\alpha_{T_1,\perp}^2+\beta_{T_1,\perp}^2 \le &10c_{init} \sigma_1,\quad
\abs{\alpha_{T_1}-\beta_{T_1}} \le c_{diff}\abs{\alpha_{T_1}+\beta_{T_1}}
\end{align*} where $c > 1/5$, $c_{init} < 1/100$, $c_{diff}< 1$ and $\eta \le \frac{c_{step}}{\sigma_1}$ for $c_{step} \le 1/100$.
Let $\tau_t = \min\{\alpha_t,\beta_t\}$ and $\tau_{min}  \triangleq \frac{\tau_{T_1}}{2}$.
Then for all $t = T+1,T+2,\ldots$, the followings hold
\begin{align*}
\tau_t \ge \prod_{i=T+1}^{t}\left(1-\eta\xi_0\left(1-\eta\tau_{min}^2\right)^{i-T}\right)\tau_T, \quad
\xi_t \le \left(1-\eta\tau_{min}^2\right)^{t-T} \xi_0, \quad \alpha_t\beta_t \le \sigma_1.
\end{align*}
\end{thm}
In Theorem~\ref{thm:rank_1_stage_2}, $\tau_t$ characterizes the smaller signal strength in $\vect{x}_t$ and $\vect{y}_t$ and $\xi_t$ characterizes the strength in the noise space.
Note even though $\tau_t$ may decrease, with some algebra, Theorem~\ref{thm:rank_1_stage_2} shows $\tau_t$ is uniformly bounded by $\Omega\left(\sqrt{\sigma_1}\right)$ for all $t \ge T_1$.
Combining with the fact that $\alpha_t\beta_t \le \sigma_1$, we know $c_0\le \frac{\alpha_t}{\beta_t} \le C_0$ for all $t \ge T_1$.
Similar to Theorem~\ref{thm:rank_1_stage_1}, here we consider $\tau_t$ and $\alpha_t\beta_t$ instead of directly analyzing $\frac{\alpha_t}{\beta_t}$ because dynamics of $\tau_t$ and $\alpha_t\beta_t$ are easier to analyze.

A directly corollary is the following local convergence rate result, which is proved by examining the dynamics of $\alpha_t\beta_t$.
\begin{cor}\label{cor:rank_1local_convergence_rate}
Under the same assumptions as in Theorem~\ref{thm:rank_1_stage_2}, we have after $T_2 = O\left(\frac{1}{\eta \sigma_1}\log\left(\frac{1}{\epsilon}\right)\right)$ iterations, $
\norm{\vect{x}\vect{y}^\top - \mathbf{M}}_F^2 \le \epsilon^2 \sigma_1^2.$
\end{cor}
Theorem~\ref{thm:rank_1_stage_1}, Theorem~\ref{thm:rank_1_stage_2} and Corollary~\ref{cor:rank_1local_convergence_rate} together imply Theorem~\ref{thm:rank_1}.

Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)