A matrix description for $K_1$ of graded rings

Abstract: The current paper is dedicated to the study of the classical $K_1$ groups of graded rings. Let $A$ be a $\Gamma$ graded ring with identity $1$, where the grading $\Gamma$ is an abelian group. We associate a category with suspension to the $\Gamma$ graded ring $A$. This allows us to construct the group valued functor $K_1$ of graded rings. It will be denoted by $K_1^{gr}$. It is not only an abelian group but also a $\mathbb Z[\Gamma]$-module. From the construction, it follows that there exists "locally" a matrix description of $K_1^{gr}$ of graded rings. The matrix description makes it possible to compute $K_1^{gr}$ of various types of graded rings. The $K_1^{gr}$ satisfies the well known $K$-theory exact sequence $$ K_{1}^{gr}(A,I)\to K_1^{gr}(A)\to K_1^{gr}(A/I) $$ for any graded ideal $I$ of $A$. The above is used to compute $K^{gr}_1$ of cross products.

\section*{Introduction}
There has been a recent upturn in interest in graded $K$-theory, motivated by the classification problem for Leavitt path algebras \cite{RH1}, the representation theory of Hecke algebra \cite{AK}  and by  the graded $K$-theory itself \cite{RHTH}.

The classical group valued functor $K_1$ of rings  was  introduced  by Hyman Bass about half century ago. The matrix description of $K_1$ is  one of the most important tools available for computing $K_1$. However in the setting of graded rings, there is no obvious way of defining $K_1$ by matrices. In the current paper, we fill in this gap and give an explicit matrix description of $K_1$ of graded rings.

The strategy of our construction of $K_1$ of graded rings to build up   $K_1$ locally  from ``pieces'' and then ``glue'' all the  pieces by a direct limit.    Each piece, which we think of as a ``local'' $K_1$, has its own $K_1$ group  possessing a matrix description. Graded $K_1$ of graded rings is a direct limit of  the ``local'' $K_1$ groups, each of which has a matrices description.

More precisely, for any   $\Gamma$ graded ring $A$, the category $\mM A$ of graded projective $A$-modules is  a category with suspension (cf.
Definition~\ref{def:suspension}). Let $S$ be a finite subfamily of elements of  the grading $\Gamma$. We construct a full subcategory $\mM A(S)$ of $\mM A$, which is a category with exact sequences (cf. Definition~\ref{def:K1_C}). The $K_1$ group of the subcategory $\mM A(S)$, denoted by $K_1(A)(S)$, is known by Bass' categorical definition of $K_1$ \cite[Definition~VII.1.4]{Bass}. These groups are our ``local'' $K_1$ groups. The $K_1(A)(S)$ groups together with the natural homomorphisms between them form a direct system of abelian groups, whose limit coincides with the $K_1$ of $\mM A$. On the other hand,  general arguments in linear algebra show that each $K_1(A)(S)$ possesses a matrix description. Thus graded $K_1$ of graded rings can be described ``locally'' by matrices. In general, the author does not know if there is a ``global'' matrix description of graded $K_1$ of graded rings.

The local matrix description of graded $K_1$ introduces a concrete tool for calculating graded $K_1$ groups. Graded Dedekind rings, graded local rings and  graded rings with finite graded stable rank (see \cite{RH2}) are promising cases for computing their graded $K_1$ groups. We are working towards this goal.

The rest of the paper is organized as follows. In Section~\ref{sec:cat},  we recall the notion of a category with suspension and  $K_1$ groups of  categories with suspension. In Section~\ref{s:1}, we briefly review some well-known facts of graded rings and graded
modules, as well as  fix notation. We refer readers to \cite{Oy1,RH2} for more details. Then in Section~\ref{s:4}, we construct $K_1^{gr}$ from ``finite pieces''  and show that the new construction coincides with the categorical definition in Section~\ref{sec:cat}. Furthermore, the new definition allows us to obtain  an group action of the grading $\Gamma$  on $K_1^{gr}$.
In Section~\ref{matrix},  we  introduce  the notions of elementary and congruence  subgroups in the setting of graded rings, and study  the finite pieces of $K_1^{gr}$  using matrices.   In Section~\ref{s:5}, the well-known $K$-theory exact sequence is established  for $K_1^{gr}$. Finally, as an application, we show in Section~\ref{s:final} that if $A$ is a  cross product then $K_1^{gr}(A)\cong K_1(A_0)$ with trivial $\Gamma$ action.

\section{Category with suspension and its $K_1$}\label{sec:cat}
In this section,  we recall the categorical definition of $K_1$ groups due to Hyman Bass   in the setting of  categories with  suspension.

\begin{Def}\label{def:K1_C}
A category $\mathcal C$ is called a {\em category with exact sequences} if $\mathcal C$ is an abelian category with a full additive subcategory $\mathcal P$ satisfying the following additional conditions:
\begin{item}
\item[C1.] $\mathcal P$ is closed under extensions, i.e., if
$$0\to  P_1\to P\to P_2\to 0$$
is an exact sequence in $\mathcal C$ and $P_1, P_2\in \mathcal P$ then $P\in \mathcal P$.
\item[C2.] $\mathcal P$ has a small skeleton (cf. \cite[Definition~3.1.1]{Rosen}).
\end{item}
\end{Def}
\begin{Def}\label{def:suspension}
Let $\Gamma$ be an abelian group and  $\mathcal C$  a category with exact sequences. Suppose that
$$\mathcal S_\Gamma=\{\mathcal S_\lambda\text{ is an  exact auto-equivalence on }  \mathcal C \mid \lambda\in\Gamma\},$$
and suppose that $\mathcal S_\Gamma$ satisfies the following relations
\begin{itemize}
\item[1.] $\mathcal S_0$ is the identity functor on $\mathcal C$;
\item[2.] $\mathcal S_\alpha\circ\mathcal S_\beta=\mathcal S_{\alpha+\beta}$ for every $\alpha,\beta\in \Gamma$.
\end{itemize}
The set $\mathcal S_\Gamma$ forms an abelian group where multiplication is defined by composition of functors.
The  set $\mathcal S_\Gamma$ is called a {\em suspension set} of $\mathcal C$ and the functor  $\mathcal S_\lambda\in \mathcal S_\Gamma$  the {\em $\lambda$-suspension} in $\mathcal S_\Gamma$.  By a {\em category with suspension}  we mean  a category with exact sequences  equipped with a suspension set.
\end{Def}
\begin{Def}
Suppose that $\mathcal C$ is a category with suspension, $\mathcal S_\Gamma$ a suspension set on $\mathcal C$, and  $\mathcal P_0$ is a small skeleton of $\mathcal C$. If  the restriction of $\mathcal S_\lambda$ induces an auto-equivalence on $\mathcal P_0$ for each $\lambda\in \Gamma$,  then $\mathcal P_0$ is called a skeleton of $\mathcal C$ with respect to the suspension set $\mathcal S_\Gamma$.
\end{Def}
\begin{Lem}
Let $\mathcal C$ be a category with a suspension $\mathcal S_\Gamma$. Then there is a small skeleton $\mathcal P_0$ of $\mathcal C$ with respected to the suspension $\mathcal S_\Gamma$.
\end{Lem}
\begin{proof}
Suppose that $\mathcal P_1$ is a small skeleton of $\mathcal C$. Then
$$
\mathcal P_0=\bigcup_{\lambda\in \Gamma} \mathcal S_{\lambda}(\mathcal P_1)
$$
which is a small skeleton of $\mathcal C$.  Clearly the restriction of each $\mathcal S_{\lambda}$ on $\mathcal P_0$ is an equivalence. This proves the lemma.
\end{proof}
In the sequel, a small skeleton of $\mathcal C$ will always mean that it respects the corresponding suspension of $\mathcal C$.

\begin{Def}\label{def:K1}
Suppose that $\mathcal C$ is a category with exact sequences which has a small skeleton $\mathcal P_0$. Then $K_1(\mathcal C)$ is defined to be the free abelian group on all pairs
$(P,\sigma)$ such that $P\in \mathcal P_0$ and $\sigma\in \operatorname{Aut}( P)$,  subject to  the following relations:
\begin{item}
\item[(K1)] $[(P,\sigma)]+[(P,\tau)]=[(P,\sigma\circ\tau)]$;
\item[(K2)] $[(P_1,\sigma_1)]+[(P_2,\sigma_2)]=[(P,\sigma)]$, if there is a commutative diagram
$$\xymatrix{0\ar[r]&P_1\ar[r]^i\ar[d]_{\sigma_1}&P\ar[r]^{\pi} \ar[d]_{\sigma}&P_2\ar[r]\ar[d]_{\sigma_2}&0\\
0\ar[r]&P_1\ar[r]^i&P\ar[r]^{\pi} &P_2\ar[r]&0
}
$$
in $\mathcal C$ with exact rows.
\end{item}
\end{Def}
We recall the following well-known properties of $K_1(\mathcal C)$.
\begin{Prop}\label{prop:K1}
Suppose that $\mathcal C$ is a category with exact sequences with a small skeleton $\mathcal P_0$. Then the group law, which we shall write additively, on $K_1(\mathcal C)$ satisfies the following:
\begin{itemize}
\item[1.] $[(P_1, \operatorname{id})]$ is the zero element in $K_1(\mathcal C)$, for each $P_1\in \mathcal P_0$;
\item[2.] the inverse of $[(P_1,\sigma)]$ in $K_1(\mathcal C)$ is $[(P_1,\sigma^{-1})]$;
\item[3.] $[(P_1\oplus P_1, \sigma\oplus\sigma^{-1})]=[(P_1\oplus P_1,\operatorname{id})]$ is the zero in $K_1(\mathcal C)$.
\end{itemize}
\end{Prop}

Let $\mathcal C$ be a category with  suspension $\mathcal S_\Gamma$. By definition,  $\mathcal S_{\Gamma}$ consists of exact auto-equivalences. Therefore each element $\mathcal S_{\lambda}$ in $\mathcal S_{\Gamma}$ preserves the identities (K1) and (K2) in Definition~\ref{def:K1}. This implies that each $\mathcal S_{\lambda}$ induces an automorphism of $K_1(\mathcal C)$. In  other words,  $\mathcal S_{\Gamma}$ induces  a $\Gamma$-action  on $K_1(\mathcal C)$.
\begin{Lem}\label{lem:sus}
Let $\mathcal C$ be a category with suspension $\mathcal S_\Gamma$. Then there is a group action of $\Gamma$ on $K_1(\mathcal C)$
$$
\lambda \cdot [(P,\sigma)]=[(\mathcal S_{\lambda}(P),\mathcal S_{\lambda}(\sigma))],
$$
which is well-defined.
\end{Lem}
It follows immediately that $K_1(\mathcal C)$ of a category $\mathcal C$ with the suspension $\mathcal S_\Gamma$ is not only an abelian group but also a $\mathbb Z[\Gamma]$-module.

\section{Graded rings and graded modules}\label{s:1}
In this section, we briefly review the notions of graded ring and graded module.
 For details see \cite{Oy1} or \cite{RH2}.

Unless otherwise stated, all rings in the current paper are assumed to be associative with identity $1$, all ring homomorphisms are assumed to be identity preserving, and all the modules in sequel are left modules.

\begin{defn}
Let $A$ be a ring and  $\Gamma$ an abelian group. $A$ is called a  {\em $\Gamma$ graded ring}, if
\begin{itemize}
\item $A=\oplus_{\lambda\in \Gamma}A_\lambda$, where  each $A_\lambda$ is an additive subgroup of $A$;
\item $A_\lambda A_\delta \subseteq A_{\lambda+\delta}$ for all $\lambda,\delta\in \Gamma$.
\end{itemize}
The none-zero elements in $A_\lambda$ are called {\em the homogeneous elements of level $\lambda$}.  We say  that a ring $A$ has  {\em trivial grading}, if $A$ is graded by the trivial group.
\end{defn}

In the case that $A_\lambda A_\delta =A_{\lambda+\delta}$ for all $\lambda, \delta \in \Gamma$, we call $A$  a {\em  strongly $\Gamma$  graded ring}. If there is an invertible element in  $A_\lambda$ for each $\lambda\in \Gamma$, then we say that $A$ is a {\em cross product}.  A cross product is always a strongly graded ring, but in general the converse is not true.

Let $A$ be a $\Gamma$ graded ring. A two-sided ideal  $I$ of $A$ is called a {\em graded ideal } if
$$
I=\bigoplus_{\lambda\in\Gamma }I\cap A_{\lambda}.
$$
We shall denote $I\cap A_{\lambda}$ by $I_\lambda$. It is called the {\em homogeneous component of level} $\lambda$ of $I$.
Then the quotient ring
$$
A/I= \bigoplus_{\lambda\in \Gamma} A_\lambda /I_{\lambda}.
$$
is again a $\Gamma$ graded ring.

Let $A$ and $B$ be two $\Gamma$ graded rings.  Then a {\em $\Gamma$ graded ring homomorphism} $\varphi: A\to B$ is a ring homomorphism  which respects to the $\Gamma$ grading, i.e.,
$$
\varphi (A_\lambda)\subseteq B_{\lambda}
$$
for all $\lambda \in \Gamma$. It is easy to check  the following facts:
\begin{itemize}
 \item[1.] The kernel of  a  graded homomorphism is a graded ideal;
 \item[2.] There is a canonical graded homomorphism  $A \to A/I$ for every graded ideal $I$ of $A$.
 \end{itemize}
 A graded homomorphism is called surjective, injective or bijective if it is so in the usual sense.  A bijective graded homomorphism is called a graded isomorphism and has a graded inverse.

 Let $A$ be a $\Gamma$ graded ring. An $A$-module is called a $\Gamma$ {\em graded $A$-module} (or simply a graded $A$-module when the grading is clear from  context), if $M$ is a directly sum   $M=\oplus_{\lambda\in \Gamma} M_\lambda$ of subgroups  and $A_\lambda M_{\delta}\subseteq M_{\lambda+\delta}$ for all $\lambda,\delta\in \Gamma$. We may define  graded submodule, graded quotient module in an obvious way.

Additionally, we have one further operation called {\em suspension} (or {\em shifting}) of a graded module. An {\em $\alpha$-suspended module} $M(\alpha)$ of $M$ is a $\Gamma$ graded module such that
$$
M(\alpha)= \bigoplus_{\lambda\in \Gamma} M(\alpha)_\lambda,
$$
 where $M(\alpha)_\lambda =M_{\alpha+\lambda}$ and $\alpha\in \Gamma$.

Let $M$ and $N$ be graded $A$-modules and let $\delta\in \Gamma$.  A {\em graded module homomorphism of degree $\delta$} is an $A$-module homomorphism such that
$\varphi(M_\lambda)\subseteq N_{\lambda+\delta}$ for each $\lambda\in \Gamma$. We denote the set of graded $A$-module homomorphisms of degree $\delta$ from $M$ to $N$   by $\operatorname{Hom}_{gr}(M,N)_\delta$ and the set of graded $A$-module endomorphism of degree $\delta$ on $M$ by  $\operatorname{End}_{gr}(M)_{\delta}$.

By a {\em graded module homomorphism}, we mean a graded module homomorphism of degree $0$. The set of graded $A$-module automorphism on $M$ is denoted by $\operatorname{Aut}_{gr}(M)$.

A  graded $A$-module $M$ is said to be free, if it has a free basis as an $A$-module consisting of homogeneous elements, equivalently  $M\cong_{gr}\oplus_{i\in I} A(\alpha_i)$ where $\{\alpha_i\mid i\in I\}$ is a family\footnote{By a family we mean a collection of elements whose elements can be repeated.} of elements of the grading $\Gamma$. When the basis happens to be finite, we denote  the graded free module $\oplus_{i=1}^{n} A(\alpha_i)$ by $ A(\alpha_1,\cdots,\alpha_n)$.

$\operatorname{End}_{gr}(A(\alpha_1,\cdots,\alpha_n))_{\delta}$ possesses a  representation by  $n\times n$ matrices  (see \cite[\S 2.10]{Oy1}) :
$$
\operatorname{End}_{gr}(A(\alpha_1,\cdots,\alpha_n))_{\delta}
\cong \left(
\begin{array}{ccccc}
A_{\delta}& A_{\delta+\alpha_1-\alpha_2}& A_{\delta+\alpha_1-\alpha_3}&\cdots& A_{\delta+\alpha_1-\alpha_n}\\
A_{\delta+\alpha_2-\alpha_1}& A_{\delta}&A_{\delta+\alpha_2-\alpha_3}&\cdots& A_{\delta+\alpha_2-\alpha_n}\\
A_{\delta+\alpha_3-\alpha_1}& A_{\delta+\alpha_3-\alpha_2}&A_{\delta}&\cdots& A_{\delta+\alpha_3-\alpha_n}\\
\vdots&\vdots&\vdots&&\vdots\\
A_{\delta+\alpha_n-\alpha_1}& A_{\delta+\alpha_n-\alpha_2}&A_{\delta+\alpha_n-\alpha_3}&\cdots& A_{\delta}\\
\end{array}
\right)
$$
which is denoted by $M_n(A)(\alpha_1,\alpha_2,\cdots,\alpha_n)_{\delta}$.

In particular, when the degree $\delta=0$, we have
$$
\operatorname{End}_{gr}(A(\alpha_1,\cdots,\alpha_n))_{0}
\cong  \left(
\begin{array}{ccccc}
A_{0}& A_{\alpha_1-\alpha_2}& A_{\alpha_1-\alpha_3}&\cdots& A_{\alpha_1-\alpha_n}\\
A_{\alpha_2-\alpha_1}& A_{0}&A_{\alpha_2-\alpha_3}&\cdots& A_{\alpha_2-\alpha_n}\\
A_{\alpha_3-\alpha_1}& A_{\alpha_3-\alpha_2}&A_{0}&\cdots& A_{\alpha_3-\alpha_n}\\
\vdots&\vdots&\vdots&&\vdots\\
A_{\alpha_n-\alpha_1}& A_{\alpha_n-\alpha_2}&A_{\alpha_n-\alpha_3}&\cdots& A_{0}\\
\end{array}
\right).
$$

Finitely generated projective modules play a crucial in algebraic $K$-theory. In the graded setting, a  graded  $A$-module $P$ is said to be projective, if
for any diagram of grade homomorphisms with exact row
$$
\xymatrix{&P\ar[d]^i &\\
M\ar[r]^g&N\ar[r]&0,
}
$$
there is a grade $A$-module homomorphism $h: P\to M$ making the resulting diagram commutes.

\begin{Prop}\label{gr-projective}
Let $A$ be a $\Gamma$ graded ring and let $P$ be a graded projective $A$-module. Then the following conditions are equivalent:
\begin{itemize}
\item[1.] $P$ is a direct summand of a graded free $A$-module.
\item[2.] Every short exact sequence of graded $A$-module homomorphisms
$$
0\to M\to N\to P\to 0
$$
is split by  a graded $A$-module homomorphism.
\item[3. ] $Hom_A^{gr}(P, \underline{\phantom{A}} )$ is an exact functor.
\end{itemize}
\end{Prop}
\begin{proof}
See \cite{RH2}.
\end{proof}

\section{Category of finitely generated graded  projective modules and its $K_1$.}\label{K_1}\label{s:4}
In this section, we show that the category $\mM A$ of  finitely generated graded projective $A$-modules is a category with suspension.  By analogy with the usual procedures of defining $K_1$ (see \cite{Bass, Rosen, Magurn}),  we construct  stable graded $K_1^{gr}$ for graded rings in two different ways. One is the categorical definition of Section~\ref{sec:cat} and the other is an explicit definition using matrices. We shall show that these two definitions agree with each other.

Let $A$ be a $\Gamma$ graded ring. The category of finitely generated graded projective $A$-modules $\mM A$  is defined as follows:
\begin{itemize}
\item the objects of $\mM {A} $ are finitely generated graded  projective $A$-modules.
\item the morphisms (arrows) are the graded homomorphisms between graded $A$-modules.
\end{itemize}
It is routine to show that the category $\mM A$ is a category with exact sequences.

For each $\lambda\in \Gamma$, we define a map $\mathcal S_{\lambda} : \mM A\to \mM A$  in the following way. For each object $M\in \mM A$, set $
\mathcal S_{\lambda}(M)=M(\lambda),
$
and  for each morphism $\varphi: M\to N$ in $\mM A$, set $\mathcal S_{\lambda}(\varphi): M(\lambda)\to N(\lambda)$ such that
$$\mathcal S_{\lambda}(\varphi)\big( M(\lambda)_\alpha\big) = \varphi (M_{\lambda+\alpha})$$
for every $\alpha\in \Gamma$.
It is easy to see that $\mathcal S_{\lambda} : \mM A \to \mM A$ is an exact auto-equivalence.

Let $\mathcal S_{\Gamma}=\{\mathcal S_{\lambda}\mid\lambda\in \Gamma\}$. Then the category $\mM A$ is a category with the suspension $\mathcal S_{\Gamma}$. This observation allow us to apply the functor $K_1$ to  the category $\mM A$ (see Section~\ref{sec:cat}).  We shall denote $K_1(\mM A )$ by $K_1^{gr}(A)$.

\begin{Lem}\label{functorial}
$K_1^{gr}(A)$ defines a  functor from the category of $\Gamma$ graded rings to the category of $\mathbb Z[\Gamma]$-modules.
\end{Lem}

Next, we will construct a family of full additive subcategories of $\mM A$, and calculate the operation of  the suspension functor on them.

Let $\{\alpha_1,\cdots, \alpha_n\}$ be a finite family of elements of  $\Gamma$.  The graded free module
$$A\big(\underbrace{\alpha_1,\cdots,\alpha_n,\alpha_1,\cdots,\alpha_n,\cdots,\alpha_1,\cdots,\alpha_n}_{m \text{ copies of } \alpha_1,\cdots,\alpha_n}\big)$$
is denoted by $A(\alpha_1,\cdots,\alpha_n)^m$.
Let $\mM A(\alpha_1,\cdots,\alpha_n)$ denote the full subcategory  of $\mM A$, whose objects are all  objects in $\mM A$ which are graded isomorphic to a direct summand of a graded module in $\{A(\alpha_1,\cdots,\alpha_n)^m\mid m\in\mathbb N\}$. Since $\mM A(\alpha_1,\cdots,\alpha_n)$ is a full additive subcategory of $\mM A$,  the group $K_1(\mM A(\alpha_1,\cdots,\alpha_n))$ can be defined by Bass' categorical approach \cite[Definition~VII.1.4]{Bass} and will be denoted by $K_1(A)(\alpha_1,\cdots,\alpha_n)$.
\begin{Lem}
If $S\subseteq T$ are two finite families of $\Gamma$, then $\mM A(S)$ is a full subcategory of $\mM A(T)$. Furthermore, there is a natural group homomorphism $\varphi: K_1(A)(S)\to K_1(A)(T)$ induced by the inclusion functor from $\mM A(S)$ to $\mM A(T)$.
\end{Lem}
\begin{proof}
Since the inclusion functor is exact, it induces a map
\begin{eqnarray*}
\varphi: &K_1(A)(S)\to K_1(A)(T)&\\
&\varphi([(P,\sigma )])=[(P,\sigma)],
\end{eqnarray*}
which is clearly a group homomorphism.
\end{proof}

The set of all finite families $S$ of elements of $\Gamma$ forms a diagram of sets under natural inclusion of sets. This diagram is clearly directed. Thus the diagram of all $K_1(A)(S)$, where $S$ ranges over all finite families of elements of $\Gamma$ forms a directed digram of groups and thus has a direct limit $\varinjlim_{S} K_1(A)(S)$.

\begin{The}\label{the:1}
Let $A$ be a $\Gamma$ graded ring. Then there is a $\mathbb Z[\Gamma]$-module isomorphism
$$K_1^{gr}(A)\cong \varinjlim_{S} K_1(A)(S),$$
where the $\mathbb Z[\Gamma]$-action on $\varinjlim_{S} K_1(A)(S)$ is induced from the action on $K_1^{gr}(A)$.
\end{The}
\begin{proof}
For each finite family $S$ of elements of $\Gamma$, there is a group homomorphism $K_1(A)(S)\to K_1^{gr}(A)$ defined by sending every class $[(P,\sigma)]\in K_1(A)(S)$ to its corresponding class $[(P,\sigma)]$ in $K_1^{gr}(A)$. From the formalities of direct limits, we get a group homomorphism
$$\varphi: \varinjlim_{S}K_1(A)(S)\to K_1^{gr}(A).$$
Each elements of $K_1^{gr}(A)$ is represented by an isomorphism classes $[(P,\sigma)]$, where the graded module $P$ is  finitely generated and projective. Hence the homomorphism  is clearly surjective.

On the other hand, $K_1^{gr}(A)$ is the free group generated by isomorphism classes $[(P,\sigma)]$. Let $\displaystyle [(Q,\tau)]\in \varinjlim_{S}K_1(A)(S)$ such that $[(Q,\tau)]$ represents the identity  in $K_1^{gr}(A)$.   Then $ [(Q,\tau)]$ is generated by  finite terms of free generators in $K_1^{gr}(A)$.
Hence there is a finite family $S$ of elements of $\Gamma$, such that all the given free generators of  $ [(Q,\tau)]$ sit in $K_1(A)(S)$.  Thus  $[(Q,\tau)]$ is already the identity element in $K_1(A)(S)$. Therefore it is the identity element in $\displaystyle\varinjlim_{S} K_1(A)(S)$. This shows that $\varphi$ is injective. Hence the whole theorem.
\end{proof}

\begin{Lem}
Let $\{\alpha_1,\cdots, \alpha_n\}$ be a finite family of  elements of $\Gamma$. For each $\lambda\in \Gamma$, the $\lambda$-suspension $\mathcal S_\lambda$ induces a group isomorphism
$$\varphi_\lambda: K_1(A)(\alpha_1,\cdots, \alpha_n) \to K_1(A)(\alpha_1+\lambda,\cdots, \alpha_n+\lambda).$$
\end{Lem}
\begin{proof}
Each $\lambda$-suspension $\mathcal S_\lambda$  is an exact auto-equivalence on $\mM A$, and the image of $\mM A(\alpha_1,\cdots, \alpha_n)$ under functor $\mathcal S_\lambda$ is $\mM A(\alpha_1+\lambda,\cdots, \alpha_n+\lambda)$. Therefore $\mathcal S_\lambda$ induces a surjective group homomorphism
$$\varphi_\lambda: K_1(A)(\alpha_1,\cdots, \alpha_n) \to K_1(A)(\alpha_1+\lambda,\cdots, \alpha_n+\lambda).$$
Conversely, $\mathcal S_{-\lambda}$ induces a surjective group homomorphism
$$
\varphi_{-\lambda}:  K_1(A)(\alpha_1+\lambda,\cdots, \alpha_n+\lambda) \to K_1(A)(\alpha_1,\cdots, \alpha_n).
$$
By Definition~\ref{def:suspension}, the compositions  $\varphi_{\lambda}\circ\varphi_{-\lambda}$ and $\varphi_{-\lambda}\circ\varphi_{\lambda}$ are identity homomorphisms on  $K_1(A)(\alpha_1+\lambda,\cdots, \alpha_n+\lambda)$ and  $K_1(A)(\alpha_1,\cdots, \alpha_n)$, respectively. Hence $\varphi_{\lambda}$ is an isomorphism and this finishes the proof.
\end{proof}

\section{The matrix description of  $K_1^{gr}$.}\label{matrix}
We introduce the notions of elementary and congruence subgroups of stable general linear groups and   construct an explicit matrix description of graded  $K_1$ of graded rings.

Let $A$ be a $\Gamma$-graded ring with identity  as usual.

\begin{Def} Let $A$ be a $\Gamma$ graded ring and let $\{\alpha_1,\alpha_2,\cdots,\alpha_n\}$ be a family of elements of $\Gamma$.  The set of all  invertible matrices in $M_n(A)(\alpha_1,\alpha_2,\cdots,\alpha_n)_0$ is called the {\em nonstable graded general linear group with parameter $(\alpha_1,\alpha_2,\cdots,\alpha_n)$} and will be  denoted by
$$
\operatorname{GL}_n(A)(\alpha_1,\alpha_2,\cdots,\alpha_n).
$$
\end{Def}

\begin{Def}
Let $A$ be a $\Gamma$ graded ring and let $\{\alpha_1,\alpha_2,\cdots,\alpha_n\}$ be a family of elements of $\Gamma$. The {\em  nonstable graded elementary subgroup of rank $n$} of $\operatorname{GL}_n(A)(\alpha_1,\cdots, \alpha_n)$ is defined as follows:
$$\operatorname{E}_n(A)(\alpha_1,\cdots, \alpha_n)=\langle e_{i,j}(r)\mid r\in A_{\alpha_i-\alpha_j}, i\ne j \rangle,$$
where $e_{i,j}(r)$ is an elementary matrix in the usual sense. Let $I$ be a graded two-sided ideal of $A$. Let $\varphi_I$ be the canonical map $A\to A/I$. It induces a group homomorphism, which is again denoted by $\varphi_I$,
$$
\varphi_I: \operatorname{GL}_n(A)(\alpha_1,\cdots, \alpha_n)\to \operatorname{GL}_n(A/I)(\alpha_1,\cdots, \alpha_n).
$$
The kernel of $\varphi_I$ is called the {\em congruence subgroup $\operatorname{GL}_n(A,I)(\alpha_1,\cdots, \alpha_n)$ of level $I$} of
$\operatorname{GL}_n(A)(\alpha_1,\cdots, \alpha_n)$.
The {\em graded elementary subgroup} $\operatorname{E}_n(I)(\alpha_1,\cdots, \alpha_n)$ of level $I$  is the  subgroup of  $\operatorname{E}_n(A)(\alpha_1,\cdots, \alpha_n)$ which is  generated by all $e_{i,j}(r)$ with $r\in I_{\alpha_j-\alpha_i}$. The minimal normal subgroup of $\operatorname{E}_n(A)(\alpha_1,\cdots, \alpha_n)$ containing $\operatorname{E}_n(I)(\alpha_1,\cdots, \alpha_n)$ is called the {\em graded relative elementary subgroup of level $I$} and is denoted by $\operatorname{E}_n(A,I)(\alpha_1,\cdots, \alpha_n)$.
\end{Def}
For every $m\in \mathbb N$, we embed  $\operatorname{GL}_{nm}(A)(\alpha_1,\cdots,\alpha_n)^m$ into $\operatorname{GL}_{n(m+1)}(A)(\alpha_1,\cdots,\alpha_n)^{m+1}$ by the  homomorphism $\varphi_{m}$ defined by
$$\begin{array}{lrcl} \varphi_{m}: &\operatorname{GL}_{nm}(A)(\alpha_1,\cdots,\alpha_n)^m&\to& \operatorname{GL}_{n(m+1)}(A)(\alpha_1,\cdots,\alpha_n)^{m+1}\\
&\varphi_{m}(g)&=&\left(
\begin{array}{cc}
g&0\\0&I_{n}
\end{array}
\right).
\end{array}
$$
The infinite union of the $\operatorname{GL}_{nm}(A)(\alpha_1,\cdots,\alpha_n)^m$ is denoted by $\operatorname{GL}(A)(\alpha_1,\cdots,\alpha_n)$. In the same way,  the infinite unions of $\operatorname{GL}_{nm}(A,I)(\alpha_1,\cdots, \alpha_n)^m$, $\operatorname{E}_{nm}(A)(\alpha_1,\cdots,\alpha_n)^m$ and $\operatorname{E}_{nm}(A,I)(\alpha_1,\cdots, \alpha_n)^{m}$ are denoted by $\operatorname{GL}(A,I)(\alpha_1,\cdots, \alpha_n)$, $\operatorname{E}(A)(\alpha_1,\cdots,\alpha_n)$ and \linebreak $\operatorname{E}(A,I)(\alpha_1,\cdots, \alpha_n)$ respectively.

In general, $\operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n)$  is not necessarily to be perfect, even when $n\ge 3$.

\begin{Exp}
Let $R$ be a commutative ring with a non-trivial ideal $I$. Then we construct a ring
$A=(R,I)$ where addition is defined to be component-wise and the multiplication is defined as follows
$$
(r_1,a_1)(r_2,a_2)=(r_1r_2,r_1a_2+r_2a_1).
$$
The ring $A$ has a $\mathbb Z_3$ grading, namely
$$
A=(R,I)=(R,0)\oplus (0,I)\oplus (0,0),
$$
where $(R,0)$ is the homogeneous component of degree $0$, $(0,I)$ is the homogeneous component of degree $1$, and  the homogeneous component of degree $2$ is the trivial group.

Consider $\operatorname{E}_{3}(A)(0,1,2)$. A straight forward calculation shows that
$$
[\operatorname{E}_{3}(A)(0,1,2),\operatorname{E}_{3}(A)(0,1,2)]\neq\operatorname{E}_{3}(A)(0,1,2).
$$

\end{Exp}

 However $\operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n)$ is indeed perfect, when $A$ happens to be a strongly graded ring with $n\ge 3$.
\begin{Lem}\label{perfect-1}
If $A$ is a strongly  $\Gamma$ graded ring, then $\operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n)$ is a perfect subgroup for any finite family $\{\alpha_1,\cdots,\alpha_n\}$ of elements of  $\Gamma$ and $n\ge 3$.
\end{Lem}
\begin{proof}
Suppose that $e_{i,j}(r)$ is a typical generator of $ \operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n)$ with $r\in A_{\alpha_i-\alpha_j}$. Since $A$ is strongly graded,
$$
A_{\alpha_i -\alpha_k}A_{\alpha_k-\alpha_j}=A_{\alpha_i-\alpha_j}
$$
for any $0\le k\le n$. By the assumption $n\ge 3$,  we may choose $k\ne i , j$. There is a decomposition
$$
r= s_1t_1+s_2t_2+\cdots+s_mt_m
$$
with $s_l\in A_{\alpha_i-\alpha_k}$ and $t_l\in A_{\alpha_k-\alpha_j}$ for $l=1,\cdots, m$. We obtain
$$
e_{i,j}(r)=\prod_{l=1}^m e_{i,j}(s_lt_l).
$$
But for every $l$, we have
$$
e_{i,j}(s_lt_l)=[e_{i,k}(s_l), e_{k,j}(t_l)],
$$
where $e_{i,k}(s_l)$ and  $e_{k,j}(t_l)$ are in $\operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n)$.
It follows immediately that
$$e_{i,j}(r)\in [\operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n), \operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n)].$$
Hence
$$
\operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n) =[\operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n), \operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n)].
$$
This finishes the proof.
\end{proof}
On the other hand, a stable elementary subgroup is always  perfect and normal in its corresponding stable general linear group.
\begin{Lem}\label{perfect-2}
$\operatorname{E}(A)(\alpha_1,\cdots,\alpha_n)$ is a perfect group.
\end{Lem}
\begin{proof}
We prove the lemma by showing that the image of $\operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n)$ under the stablizing homomorphism is a subgroup of  the commutator subgroup $$[\operatorname{E}_{2n}(A)(\alpha_1,\cdots,\alpha_n)^2,\operatorname{E}_{2n}(A)(\alpha_1,\cdots,\alpha_n)^2].$$

Let  $e_{i,j}(r)$ be a generator of $\operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n)$.   We have the identity
$$
e_{i,j}(r)=[e_{i,n+j}(r), e_{n+j,j}(1)]
$$
where $e_{i,n+j}(r)$ and $e_{n+j,j}(1)$  belong to $\operatorname{E}_{2n}(A)(\alpha_1,\cdots,\alpha_n)^2$. Hence
$$e\in[\operatorname{E}_{2n}(A)(\alpha_1,\cdots,\alpha_n)^2,\operatorname{E}_{2n}(A)(\alpha_1,\cdots,\alpha_n)^2].$$
Therefore,
$$
\operatorname{E}(A)(\alpha_1,\cdots,\alpha_n)=[\operatorname{E}(A)(\alpha_1,\cdots,\alpha_n),\operatorname{E}(A)(\alpha_1,\cdots,\alpha_n)],
$$
which proves the lemma.
\end{proof}

\begin{Lem}\label{[gg]=e} For any $h\in \operatorname{GL}_{n}(A)(\alpha_1,\cdots,\alpha_n)$, we have
$$
\left(\begin{array}{cc}h&0\\0&h^{-1}\end{array}\right)
\in \operatorname{E}_{2n}(A)(\alpha_1,\cdots,\alpha_n)^2.$$
\end{Lem}
\begin{proof}
We have the matrix equation
$$
\left(\begin{array}{cc}h&0\\0&h^{-1}\end{array}\right)=\left(\begin{array}{cc}I_n&h\\0&I_n\end{array}\right)
\left(\begin{array}{cc}I_n&0\\-h^{-1}&I_n\end{array}\right)\left(\begin{array}{cc}I_n&h\\0&I_n\end{array}\right)\left(\begin{array}{cc}0&-I_n\\I_n&0\end{array}\right).
$$
By a standard argument, for example that in \cite[Corollary 2.1.3]{Rosen}, all  four factors in the above equation lie in $\operatorname{E}_{2n}(A)(\alpha_1,\cdots,\alpha_n)^2$.
\end{proof}
\begin{The}[Graded Whitehead Lemma]\label{White}
Let $A$ be a $\Gamma$ graded ring. Then
$$[\operatorname{GL}(A)(\alpha_1,\cdots,\alpha_n),\operatorname{GL}(A)(\alpha_1,\cdots,\alpha_n)]= \operatorname{E}(A)(\alpha_1,\cdots,\alpha_n).$$
Therefore $\operatorname{E}(A)(\alpha_1,\cdots,\alpha_n)$ is a normal subgroup of $\operatorname{GL}(A)(\alpha_1,\cdots,\alpha_n)$.
\end{The}
\begin{proof}
Suppose that   $g$ and $h$ are two elements in  $\operatorname{GL}(A)(\alpha_1,\ldots,\alpha_n)$. Then there exists an $m\in \mathbb Z$ such that
$g,h\in \operatorname{GL}_{nm}(A)(\alpha_1,\cdots,\alpha_n)^m$.
Consider the images   of $g$ and $h$ in $\operatorname{GL}_{2nm}(A)(\alpha_1,\cdots,\alpha_n)^{2m}$ , which will be denoted by $g'$ and $h'$, respectively.
The matrices  $g'$ and $h'$  take the following forms
$$
g'=\left(\begin{array}{cc}
g&0\\
0&I_{nm}
\end{array}\right) \quad\text{and} \quad h'=\left(\begin{array}{cc}
h&0\\
0&I_{nm}
\end{array}\right).
$$
Now apply the identity
$$
\left(\begin{array}{cc}ghg^{-1}h^{-1}&0\\0&1\end{array}\right)=\left(\begin{array}{cc}gh&0\\0&h^{-1}g^{-1}\end{array}\right)
\left(\begin{array}{cc}g^{-1}&0\\0&g\end{array}\right)\left(\begin{array}{cc}h^{-1}&0\\0&h\end{array}\right).
$$
By Lemma~\ref{[gg]=e}, all of the factors on the right lie in $\operatorname{E}_{2nm}(A)(\alpha_1,\ldots,\alpha_n)^{2m}$. This shows that
$$[g,h]\in \operatorname{E}(A)(\alpha_1,\cdots,\alpha_n).$$
Hence
$$[\operatorname{GL}(A)(\alpha_1,\cdots,\alpha_n),\operatorname{GL}(A)(\alpha_1,\cdots,\alpha_n)]= \operatorname{E}(A)(\alpha_1,\cdots,\alpha_n).$$
It follows that $\operatorname{E}(A)(\alpha_1,\cdots,\alpha_n)$ is a normal subgroup of $\operatorname{GL}(A)(\alpha_1,\cdots,\alpha_n)$. This finishes the proof.
\end{proof}
Theorem~\ref{White} amounts to saying that the quotient group  $\operatorname{GL}(A)(\alpha_1,\cdots,\alpha_n)/\operatorname{E}(A)(\alpha_1,\cdots,\alpha_n)$ is an abelian group for every finite family  $\{\alpha_1,\cdots,\alpha_n\}$.
\begin{Lem}\label{lem:4.7}
Let $S$ be a finite family of elements of $\Gamma$. Then
$$
K_1(A)(S)\simeq \operatorname{GL}(A)(S)/\operatorname{E}(A)(S).
$$
\end{Lem}
The lemma is an easy generalization  of the corresponding result in classical $K$-theory. We skip its proof and refer the interested reader to \cite[Theorem~5.4]{Lam} or \cite[Theorem~3.1.7]{Rosen}.

It follows by Lemma~\ref{lem:4.7} that for any $\Gamma$ graded ring $A$,  $K_1^{gr}(A)$ is a direct limit of matrix  rings, i.e.,
$$
K_1^{gr}(A)\simeq \varinjlim_{S}K_1(A)(S)\simeq\varinjlim_{S} \operatorname{GL}(A)(S)/\operatorname{E}(A)(S) .
$$
This gives  ``locally'' a matrix description for $K_1^{gr}(A)$.

\section{Relative $K_1^{gr}$ and an exact sequence}\label{s:5}
An important property of $K$-theory groups is the existence of  exact sequences relating the $K_i(A)$, $K_i(A/I)$ and the relative $K$-group $K_i(A,I)$ for each $i\in \mathbb Z$, where $A$ is an ungraded ring containing $I$ as a two-sided ideal. In the present section, we establish such exact sequence for $K_1^{gr}$ of graded rings.

\begin{Def}
Let $A$ be a $\Gamma$ graded ring and $I$ a graded two-sided ideal of $A$. The {\em double of $A$ along $I$} is the subring of the Cartesian product $A\times A$ given by
$$
D(A,I)=\{(x,y)\in A\times A\mid x-y\in I\}.
$$
Its grading is induced from the grading of $A\times A$.
\end{Def}
We denote the projection onto the $i$th coordinate of $D(A,I)$ by $p_i$. Then there is a  split exact sequence of graded rings
$$
\xymatrix@1{0\ar[r]& I\ar[r]& D(A,I)\ar[r]^{\hspace{15pt} p_1}& A\ar[r]&0.}
$$

\begin{Def}
The {\em relative $K_{1}^{gr}$ group}  of a graded ring $A$ and a graded ideal $I$ is defined by
$$
K_1^{gr}(A,I)=\ker \Big( (p_1)_*: K_1^{gr}(D(A,I))\to K_1^{gr}(A)\Big).
$$
\end{Def}
As in the case of ungraded rings, we need a more computable construction of   $K_{1}^{gr}(A,I)$. This will be done locally.
\begin{Def}
Let $S$ be a finite family of elements  of $\Gamma$. We define
$$
K_1(A,I)(S)=\ker \Big( (p_1)_*: K_1(D(A,I))(S)\to K_1(A)(S)\Big).
$$
\end{Def}

\begin{The}\label{relative-White}
Let $A$ be a $\Gamma$ graded ring with graded two-sided ideal $I$. Then $\operatorname{E}(A,I)(\alpha_1,\cdots,\alpha_n)$ is normal in $\operatorname{GL}(A,I)(\alpha_1,\cdots,\alpha_n)$, and
$$
\operatorname{GL}(A,I)(\alpha_1,\cdots,\alpha_n)/\operatorname{E}(A,I)(\alpha_1,\cdots,\alpha_n)\cong K_1(A,I)(\alpha_1,\cdots,\alpha_n),
$$
for every finite family $\{\alpha_1,\cdots,\alpha_n\}$ of elements of $\Gamma$.
\end{The}
\begin{proof}
Suppose
$e\in \operatorname{E}(A,I)(\alpha_1,\cdots, \alpha_n)$ and $g\in \operatorname{GL}(A)(\alpha_1,\cdots, \alpha_n)$. Then there exists an $m$, such that $e\in \operatorname{E}_{nm}(A)(\alpha_1,\cdots, \alpha_n)^m$ and  $g\in \operatorname{GL}_{nm}(A)(\alpha_1,\cdots, \alpha_n)^m$.
Then we have the identity
$$
\left(\begin{array}{cc}
geg^{-1}&0\\0&I_{nm}
\end{array}\right)=\left(\begin{array}{cc}
g&0\\0&g^{-1}
\end{array}\right)
\left(\begin{array}{cc}
e&0\\0&I_{nm}
\end{array}\right)
\left(\begin{array}{cc}
g^{-1}&0\\0&g
\end{array}\right).
$$
By Lemma~\ref{[gg]=e}, the first and third factors on the right are in $\operatorname{E}_{2nm}(A)(\alpha_1,\cdots, \alpha_n)^{2m}$. This proves the first assertion.

Suppose $(g_1,g_2)\in  \operatorname{GL}(D(A,I))(\alpha_1,\cdots,\alpha_n)$ and maps to the identity of $K_1^{gr}(A,I)$ under $(p_1)_*$.  This means that $g_1\in \operatorname{E}(A)(\alpha_1,\cdots,\alpha_n)$.
Thus there is an $m\in\mathbb N$ such that
$
g_1,g_2\in \operatorname{GL}_{nm}(A)(\alpha_1,\cdots, \alpha_n)^m.
$
Thus $g_1$ can be write as a finite product of elementary matrices, namely we have $g_1=\prod_k e_{i_k,j_k}(a_k)$. But
$$
(g_1,g_1)=\prod_k e_{i_k,j_k}(a_k,a_k)\in \operatorname{E}(D(A,I)(\alpha_1,\cdots, \alpha_n)^m).
$$
Multiplying $(g_1,g_2)$ by $(g_1,g_1)^{-1}$ does not change the class of $(g_1,g_2)$ in $K_1(D(A,I))(\alpha_1,\cdots,\alpha_n)$, and
$$
(g_1,g_2)(g_1, g_1)^{-1}=(1, g_2g_1^{-1}).
$$
Since $(1, g_2g_1^{-1})\in \operatorname{GL}(D(A,I))(\alpha_1,\cdots,\alpha_n)$, we deduce that $g_2g_1^{-1}\in \operatorname{GL}(A,I)(\alpha_1,\cdots,\alpha_n)$.

Conversely every $g\in \operatorname{GL}(A,I)(\alpha_1,\cdots,\alpha_n)$ corresponds to $(1,g)$ in $\operatorname{GL}(D(A,I))(\alpha_1,\cdots,\alpha_n)$, which defines an element in $K_1(D(A,I))(\alpha_1,\cdots,\alpha_n)$.

To show that $\operatorname{GL}(A,I)(\alpha_1,\cdots,\alpha_n)/\operatorname{E}(A,I)(\alpha_1,\cdots,\alpha_n)\cong K_1(A,I)(\alpha_1,\cdots,\alpha_n)$, we need only to  check that $(1, g)\in \operatorname{E}(D(A,I))(\alpha_1,\cdots,\alpha_n)$ if and only if $ g\in \operatorname{E}(A,I)(\alpha_1,\cdots,\alpha_n)$ for any $g\in \operatorname{GL}(A,I)(\alpha_1,\cdots,\alpha_n)$.

For one direction, suppose that $g\in \operatorname{E}(A,I)(\alpha_1,\cdots,\alpha_n)$. Then there exists an $m\in\mathbb N$
such that $g\in \operatorname{E}_{nm}(A,I)(\alpha_1,\cdots,\alpha_n)^m$.  Then $g=\prod_k S_ke_{i_k,j_k}(a_k)S_k^{-1}$ where $S_k\in \operatorname{E}_{nm}(A)(\alpha_1,\alpha_2,\cdots,\alpha_n)^m$ and $a_k\in I$. It suffices to show that $(1, S_ke_{i_k,j_k}(a_k)S_k^{-1})\in \operatorname{E}_{nm}(D(A,I))(\alpha_1,\alpha_2,\cdots,\alpha_n)^m$ for each $k$. But
$$
(1, S_ke_{i_k,j_k}(a_k)S_k^{-1})=(S_k,S_k)e_{i,j}(0,a_k)(S_k^{-1},S_k^{-1})
$$
and all the three factors on the right lie in $\operatorname{E}(D(A,I))(\alpha_1,\alpha_2,\cdots,\alpha_n)$.

For the other direction, suppose that $(1, g)\in \operatorname{E}(D(A,I))(\alpha_1,\cdots,\alpha_n)$. There exists an $m\in\mathbb Z$ such that $(1,g)\in \operatorname{E}_{nm}(A,I)(\alpha_1,\cdots,\alpha_n)^m)$. Thus
$$
(1,g) =\prod_{k=1}^re_{i_k,j_k}(a_k,b_k)\in \operatorname{E}_{nm}(A,I)(\alpha_1,\cdots,\alpha_n)^m,
$$
where $\prod_{k=1}^re_{i_k,j_k}(a_k)=1$. Note for each $k$,
$$
e_{i_k,j_k}(a_k,b_k)=e_{i_k,j_k}(a_k,a_k)e_{i_k,j_k}(0,b_k-a_k)=(S_k,S_k)(1, T_{k}),
$$
where
$$
S_k=e_{i_k,j_k}(a_k), \quad T_k=e_{i_k,j_k}(b_k-a_k),  \quad b_k-a_k\in I.
$$
Thus we have
\begin{eqnarray*}
(1, g)&=&\prod_{k=1}^re_{i_k,j_k}(a_k,b_k)\\
&=&\prod_{k=1}^r (S_k, S_kT_k)\\
&=&(S_1,S_1T_1S_1^{-1})(S_2,S_1S_2T_1S_2^{-1}S_1^{-1})\\
&&\qquad \cdots (S_2,S_1S_2\cdots S_rT_1)\\
&=&\Big(I_n, (S_1T_1S_1^{-1})(S_1S_2T_1S_2^{-1}S_1^{-1})\\
&&\qquad\cdots(S_1S_2\cdots S_rT_1 S_r^{-1}\cdots S_2^{-1}S_1^{-1})
\Big)
\end{eqnarray*}
since $S_1S_2\cdots S_r=1$, and we have written $g$ as a product of generators of $E_n(A,I)(\alpha_1,\dots,\alpha_n)$. This finishes the proof.
\end{proof}

\begin{The}\label{Exact}
Let $A$ be a graded ring with a graded two-sided ideal $I$ and $S$ a finite family of elements of $\Gamma$. Then there is a natural exact sequence of groups
$$
\xymatrix@1{K_1(A,I)(S)\ar[r]^{\quad(p_2)_*}& K_1(A)(S)\ar[r]^{q_*}& K_1(A/I)(S)},
$$
where $(p_2)_*$ is induced by the projection $p_2:D(A,I)\to A$ and $q$ is induced by the quotient map $A\to A/I$.
\end{The}
\begin{proof}
We have seen that any class in $K_1(A,I)(S)$ is represented by
$$(1,g)\in \operatorname{GL}(D(A,I))(S)$$
with $g\in \operatorname{GL}(A,I)(S)$. We obtain immediately that $q_*\circ (p_2)_*([g])=1$ in $K_1(A/I)(S)$.

Conversely, if $g\in K_1(A)(S)$ and $q_*(g)=1$ in $K_1(A/I)(S)$, then  $g$ is represented by
$g'\in \operatorname{GL}(A)(S)$.
$q(g')\in \operatorname{E}(A/I)(S)$. Since $\operatorname{E}(A)(S)$ maps surjectively  onto $\operatorname{E}(A/I)(S)$, we may find $h\in \operatorname{E}(A)$ such that $q(g')=q(h)$. Therefore $q(gh^{-1})=1\in \operatorname{GL}(A/I)(S)$. Now $(1, gh^{-1})\in \operatorname{GL}(D(A,I))(S)$ and we  finish the proof by Theorem~\ref{relative-White}. \end{proof}

Since taking direct limits preserve exactness, we obtain from  Theorem~\ref{Exact} the usual $K$-theory sequence for  graded $K_1^{gr}$:
\begin{The}\label{Exact2}
Let $A$ be a graded ring with a graded two-sided ideal $I$. Then there is a natural exact sequence of $\mathbb Z[\Gamma]$-modules
$$
\xymatrix@1{K_1^{gr}(A,I)\ar[r]^{\quad(p_2)_*}& K_1^{gr}(A)\ar[r]^{q_*}& K_1^{gr}(A/I)},
$$
where $(p_2)_*$ is induced by the projection $p_2:D(A,I)\to A$ and $q$ is induced by the quotient map $A\to A/I$.
\end{The}
\begin{Rem}
As in classical $K$-theory, the sequence above can be extended to graded $K_0$ groups
$$
\xymatrix@1{K_1^{gr}(A,I)\ar[r]^{\quad(p_2)_*}& K_1^{gr}(A)\ar[r]^{q_*}& K_1^{gr}(A/I)\ar[r]&K_0^{gr}(A,I)\ar[r]& K_0^{gr}(A)\ar[r]& K_0^{gr}(A/I)}.
$$
 It is a straightforward adoption of the proof for the classical result, see \cite{Bass} or \cite{Rosen} and see \cite{RH2} for the exactness of  the extended sequence in classical $K$-theory.
\end{Rem}

\section{$K_1^{gr}$ of cross products}\label{s:final}
It is known that for any $\Gamma$ strongly graded  ring $A$, the category $\mM A$ of graded $A$-modules  is equivalent to the category $\mM {A_0}$ of $A_0$-modules  \cite[Theorem~2.8]{Dade}. Therefore $K_i^{gr}(A)$ is isomorphic to $K_i(A_0)$ as a group \cite[Section~3.8]{RH2}.  In this section, we will give an elementary proof of the fact  $K_1^{gr}(A)\cong K_1(A_0)$ as groups when $A$ is strongly graded, and then show that the action of $\Gamma$ on $K_1^{gr}(A)$ is always trivial, when $A$ is a cross product.

\begin{Lem}
Let $A$ be a $\Gamma$ strongly graded ring. For any free graded module $A(\alpha)$ of rank $1$, there exist a finitely generated graded projective module $P$ and a free graded module $A(0)^m$ of rank $m$ such that
$$
A(\alpha)\oplus P \cong _{gr} A(0)^m.
$$
\end{Lem}
\begin{proof}
By definition, $A(\alpha)$ is an $\alpha$-suspended module of $A$. Since $A(\alpha)$ is $\alpha$-suspended,  the identity $1$ of $A$ is of level $-\alpha$ in $A(\alpha)$. To make things clear, we denote the $\alpha$-suspended identity $1$ by $1(\alpha)$.

Since $A$ is strongly graded, we can find elements $s_1,\ldots, s_m\in \operatorname{A}_{-\alpha}$ and $t_1, \ldots, t_m\in A_{\alpha}$ such that
$$
\sum_{i=1}^m s_it_i=1.
$$
Since all $t_i$'s are  in $A_{\alpha}$, it follows immediately that the $t_i1(\alpha)$ 's are in $A(\alpha)_0$. Furthermore, the equation
$$
\sum_{i=1}^m s_it_i1(\alpha)=1(\alpha)
$$
implies that $t_11(\alpha),\ldots,t_m1(\alpha)$ generated $A(\alpha)$ as graded module.

The free graded module $A(0)^m$ has a basis $(a_1, \ldots,a_m)$ with $a_1,\ldots,a_m\in A(0)_0$. For each $i$, assign  $a_i$ in $A(0)^m$ to $ t_i1(\alpha)$ in $A(\alpha)$. Extending the assignment linearly, we get a graded homomorphism $\varphi: A(0)^m\to A(\alpha)$, which is  surjective. The graded homomorphism $\varphi$ induces
 an exact sequence of graded modules
$$
\xymatrix@1{0\ar[r]& \ker(\varphi)\ar[r]& A(0)^m\ar[r]^{\varphi}&A(\alpha)\ar[r]&0 },
$$
which is split by Proposition~\ref{gr-projective}. We obtain that
$$A(\alpha)\oplus \ker(\varphi) \cong _{gr} A(0)^m,
$$
and   the proof is finished.
\end{proof}
A direct consequence of the above lemma is the following.

\begin{Lem}
Let $A$ be a strongly graded $\Gamma$ graded ring. Then
every free graded $A$-module of finite rank is a direct summand of a free graded module $A(0,\ldots,0)$ of finite rank. Furthermore, every finitely generated graded projective $A$-module is a direct summand of a free graded module $A(0,\ldots,0)$ of finite rank.
\end{Lem}
 This phenomenon reduces $K_1^{gr}(A)$ to $K_1(A)(0)$ which is $\operatorname{GL}(A)(0)/\operatorname{E}(A)(0)$ by Lemma~\ref{lem:4.7}. By the definition of the $K_1$ of ungraded rings, $\operatorname{GL}(A)(0)/\operatorname{E}(A)(0) \cong K_1(A_0)$. Hence we have
\begin{Lem}
Let $A$ be a strongly graded $\Gamma$ graded ring. Then
$$
K_1^{gr}(A)\cong K_1(A_0)
$$
as groups.
\end{Lem}
In  case  $A$ is a cross product, the action of $\Gamma$ is always trivial.

\begin{Lem}\label{lem:7.1}
Let $\{\alpha_1,\cdots,\alpha_n\}$ be a finite family of elements of  $\Gamma$ and $\lambda$ an element in $\Gamma$. For any invertible homogeneous element $r\in A_\lambda$, we have
$$
\left(\begin{array}{cc}
0&-r^{-1}I_n\\rI_n&0
\end{array}\right)\in \operatorname{E}_{2n}(A)(\alpha_1,\cdots,\alpha_n,\lambda+\alpha_1,\cdots,\lambda+\alpha_n).
$$
\end{Lem}
\begin{proof}
The assertion directly follows from the matrix identity
$$
\left(\begin{array}{cc}
0&-r^{-1}I_n\\rI_n&0
\end{array}\right)=\left(\begin{array}{cc}
I_n&-r^{-1}I_n\\0&I_n
\end{array}\right)\left(\begin{array}{cc}
I_n&0\\rI_n&I_n
\end{array}\right)
\left(\begin{array}{cc}
I_n&-r^{-1}I_n\\0&I_n
\end{array}\right)
$$
where all three factors on the right  are in $\operatorname{E}(A(\alpha_1,\cdots,\alpha_n,\lambda+\alpha_1,\cdots,\lambda+\alpha_n))$.
\end{proof}
\begin{Lem}\label{lem:7.2}
For any $h\in \operatorname{E}_n(A)(\alpha_1,\cdots,\alpha_n)$ and any invertible homogeneous element $r\in A_\lambda$, we have
$$
\left(\begin{array}{cc}
h&0\\0&rh^{-1}r^{-1}
\end{array}\right)\in\operatorname{E}_{2n}(A)(\alpha_1,\cdots,\alpha_n,\lambda+\alpha_1,\cdots,\lambda+\alpha_n).
$$
\end{Lem}
\begin{proof}
We have the matrix equation
$$
\left(\begin{array}{cc}
h&0\\0&rh^{-1}r^{-1}
\end{array}\right)=\left(\begin{array}{cc}
I_n&hr^{-1}\\0&I_n
\end{array}\right)
\left(\begin{array}{cc}
I_n&0\\-rh^{-1}&I_n
\end{array}\right)
\left(\begin{array}{cc}
I_n&hr^{-1}\\0&I_n
\end{array}\right)
\left(\begin{array}{cc}
0&-r^{-1}I_n\\rI_n&0
\end{array}\right)
$$
where the first three factors are in $\operatorname{E}_{2n}(A)(\alpha_1,\cdots,\alpha_n,\lambda+\alpha_1,\cdots,\lambda+\alpha_n)$ and the last factor is also there by Lemma~\ref{lem:7.1}.
\end{proof}
\begin{The}\label{cross}
Suppose the $\Gamma$ graded ring $A$ is a cross product. Then the action of $\Gamma$ on $K_1^{gr}(A)$ is trivial.
\end{The}
\begin{proof}
Let $[(P,\varphi)]$ be an element in $K_1^{gr}(A)$. Then there is a finite family $\{\alpha_1,\cdots,\alpha_n\}$ of elements of $\Gamma$ such that $[(P,\varphi)]\in K_1^{gr}(A)(\alpha_1,\cdots,\alpha_n)$. Take $h$ to be a matrix which represents $[(P,\varphi)]$ in
$K_1^{gr}(A)(\alpha_1,\cdots,\alpha_n)$. For any $\gamma\in \Gamma$, the result of $\gamma$ acting  on $[P,\varphi]$ is $[(P(\gamma),\mathcal S_\gamma(\varphi))]$, which lies in $K_1^{gr}(A)(\lambda+\alpha_1,\cdots,\lambda+\alpha_n)$.  It is easy to see  that $[(P(\gamma),\mathcal S_\gamma(\varphi))]$ is represented by $g=r hr^{-1}$ in $K_1^{gr}(A)(\lambda+\alpha_1,\cdots,\lambda+\alpha_n)$ with $r$ an invertible element of degree $\lambda$. The images of $h$ and $g^{-1}$ in the larger group
$K_1^{gr}(A)(\alpha_1,\cdots,\alpha_n,\lambda+\alpha_1,\cdots,\lambda+\alpha_n)$ are
$$h'=\left(\begin{array}{cc}
h&0\\0&I_n
\end{array}\right)\qquad\text{ and }\qquad g'=\left(\begin{array}{cc}
I_n&0\\0&rh^{-1}r^{-1}
\end{array}\right),$$ respectively. By Lemma~\ref{lem:7.2}, the product of $h'$ and $g'$ is zero in $K_1^{gr}(A)(\alpha_1,\cdots,\alpha_n,\lambda+\alpha_1,\cdots,\lambda+\alpha_n)$. This shows that $g=h$ in $K_1^{gr}(A)$. Therefore the action of $\Gamma$ is trivial.
\end{proof}
\begin{Rem}
Theorem~\ref{cross} can also be concluded using the properties of cross product directly. See \cite[Example 3.1.9]{RH2}.
\end{Rem}
\noindent{\bf Acknowledgement.}
I would like to thank Roozbeh Hazrat  for very productive discussions and for sharing me generously his  unpublished  manuscript \cite{RH2} and to Prof.~Anthony Bak for constructive suggestions. I would like also to thank anonymous referee for very careful reading the manuscript and detailed comments.

K-Theory and Homology (math.KT); Rings and Algebras (math.RA)


Loss Landscapes of Regularized Linear Autoencoders

Abstract: Autoencoders are a deep learning model for representation learning. When trained to minimize the distance between the data and its reconstruction, linear autoencoders (LAEs) learn the subspace spanned by the top principal directions but cannot learn the principal directions themselves. In this paper, we prove that $L_2$-regularized LAEs are symmetric at all critical points and learn the principal directions as the left singular vectors of the decoder. We smoothly parameterize the critical manifold and relate the minima to the MAP estimate of probabilistic PCA. We illustrate these results empirically and consider implications for PCA algorithms, computational neuroscience, and the algebraic topology of learning.

\section{Introduction}
\label{introduction}

Consider a data set consisting of points $x_1, \ldots, x_n$ in $\mathbb{R}^m$. Let $X \in \mathbb{R}^{m \times n}$ be the data matrix with columns $x_i$. We will assume throughout that $k \le\min\{m,n\}$ and that the singular values of $X$ are positive and distinct.

An \textit{autoencoder} consists of an encoder $f: \mathbb{R}^m \to \mathbb{R}^k$ and decoder $g: \mathbb{R}^k \to \mathbb{R}^m$; the latter maps the latent representation $f(x_i)$ to the reconstruction $\hat{x}_i = g(f(x_i))$ \citep{goodfellow}. The full network is trained to minimize reconstruction error, typically the squared Euclidean distance between the dataset $X$ and its reconstruction $\hat{X}$ (or equivalently, the Frobenius norm of $X - \hat{X}$). When the activations of the network are the identity, the model class reduces to that of one encoder layer $W_1 \in \mathbb{R}^{k \times m}$ and one decoder layer $W_2 \in \mathbb{R}^{m \times k}$. We refer to this model as a \textit{linear autoencoder} (LAE) with loss function defined by

$$\mathcal{L}(W_1, W_2) = ||X - W_2 W_1 X||_F^2.$$

Parameterizing $\mathcal{L}$ by the product $W = W_2 W_1$, the Eckart-Young Theorem \citep{young} states that the optimal $W$ orthogonally projects $X$ onto the subspace spanned by its top $k$ \textit{principal directions}\footnote{The principal directions of $X$ are the eigenvectors of the $m \times m$ covariance of $X$ in descending order by eigenvalue, or equivalently the left singular vectors of the mean-centered $X$ in descending order by (squared) singular values.}.

Without regularization, LAEs learn this subspace but cannot learn the principal directions themselves due to the symmetry of $\mathcal{L}$ under the action of the group $\mathrm{GL}_k(\mathbb{R})$ of invertible $k \times k$ matrices defined by $(W_1, W_2) \mapsto (G W_1, W_2 G^{-1})$:

\begin{equation}
\label{symg}
X - (W_2 G^{-1})(G W_1) X = X - W_2 W_1X.
\end{equation}

Indeed, $\mathcal{L}$ achieves its minimum value on a smooth submanifold of $\mathbb{R}^{k \times m} \times \mathbb{R}^{m \times k}$ diffeomorphic to $\mathrm{GL}_k(\mathbb{R})$; the learned latent representation is only defined up to deformation by invertible linear maps; and the $k$-dimensional eigenspace of $W$ with eigenvalue one has no preferred basis.

In light of the above, the genesis of this work was our surprise at the theorem\footnote{The theorem was retracted following our correspondance.} and empirical observation in \citet{plaut} that the principal directions of $X$ are recovered from a trained LAE as the left singular vectors of the decoder (or as the right singular vectors of the encoder). We realized by looking at the code that training was done with the common practice of $L_2$-regularization:

$$\mathcal{L}_\sigma(W_1, W_2) = \mathcal{L}(W_1, W_2) + \lambda\left(||W_1||^2_F + ||W_2||^2_F\right).$$

In this paper, we prove that LAEs \textit{with} $L_2$-regularization do in fact learn the principal directions in this way, while shrinking eigenvalues in the manner of probabilistic PCA \citep{bishop99}. The key idea is that regularization reduces the symmetry group from $\mathrm{GL}_k(\mathbb{R})$ to the orthogonal group $\mathrm{O}_k(\mathbb{R})$, which preserves the structure of SVD. We further prove that the encoder and decoder are transposes at all critical points, with implications for whether the brain could plausibly implement error backpropagation.

\begin{figure*}[t]
\label{vectorfields}
\vskip 0.2in
\begin{center}
\begin{subfigure}{0.49\columnwidth}
  \centerline{
  \includegraphics[width=\linewidth]{img/1d/noreg.jpeg}
  }
  \caption{Unregularized}
\end{subfigure}
\begin{subfigure}{0.49\columnwidth}
  \centerline{
  \includegraphics[width=\linewidth]{img/1d/product.jpeg}
  }
  \caption{Product ($\lambda = 2$)}
\end{subfigure}
\begin{subfigure}{0.49\columnwidth}
  \centerline{
  \includegraphics[width=\linewidth]{img/1d/sum1.jpeg}
  }
  \caption{Sum ($\lambda = 2$)}
\end{subfigure}
\begin{subfigure}{0.49\columnwidth}
  \centerline{
  \includegraphics[width=\linewidth]{img/1d/sum2.jpeg}
  }
  \caption{Sum ($\lambda = 4$)}
\end{subfigure}
\caption{Scalar loss landscapes with $x^2=4$. Yellow points are saddles and red curves and points are global minima.}
\label{one dimensional}
\end{center}
\vskip -0.2in
\end{figure*}

\subsection{Related work}

Building on the original work of \citet{young} on low-rank matrix approximation, \citet{izenman} demonstrated a connection between a rank-reduced regression model similar to an LAE and PCA. \citet{bourlard} characterized the minima of an unregularized LAE; \citet{baldi1989} extended this analysis to all critical points.

Several studies of the effect of regularization on LAEs have emerged of late. The rank-reduced regression model was extended in \citet{mukherjee} to the study of rank-reduced ridge regression. A similar extension of the LAE model was given in \citet{josse}. An in depth analysis of the linear denoising autoencoder was given in \citet{pretorius} and most recently, \citet{mianjy} explored the effect of dropout regularization on the minima of an LAE.

While $L_2$-regularization is a foundational technique in statistical learning, its effect on autoencoder models has not been fully characterized. Recent work of \citet{mehta} on $L_2$-regularized deep linear networks applies techniques from algebraic geometry to highlight how algebraic symmetries result in ``flat" critical manifolds and how $L_2$-regularization breaks these symmetries to produce isolated critical points. We instead apply elementary linear algebra, with intuition from algebraic topology, to completely resolve dynamics in the special case of LAEs.

\subsection{Our contributions}

The contributions of our paper are as follows.
\begin{compactitem}
    \item In Section \ref{secloss} we consider LAEs with (i) no regularization, (ii) $L_2$-regularization of the composition of the encoder and decoder, and (iii) $L_2$-regularization of the encoder and decoder separately as in $\mathcal{L}_\sigma$. We build intuition by analyzing the scalar case, consider the relationship between regularization and orthogonality, and deduce that the encoder and decoder are transposes at all critical points of $\mathcal{L}_\sigma$.
    \item In Section \ref{bayesian models}, we realize all three LAE models as generative processes, most notably relating the minimum of $\mathcal{L}_\sigma$ and the MAP estimate of  probabilistic PCA.
    \item In Section \ref{seccrit}, we characterize all three loss landscapes. To build intuition, we first leave the overparameterized world of coordinate representations to think geometrically about the squared distance from a plane to a point cloud. We expand on this topological viewpoint in Appendix \ref{morse}.
    \item In Section \ref{secemp}, we illustrate these results empirically, with all code and several talks available on \href{https://github.com/danielkunin/Regularized-Linear-Autoencoders}{GitHub}\footnote{\href{https://github.com/danielkunin/Regularized-Linear-Autoencoders}{github.com/danielkunin/Regularized-Linear-Autoencoders}}.
    \item In Section \ref{secdisc}, we discuss implications for eigendecomposition algorithms, computational neuroscience, and deep learning.
\end{compactitem}
The connections we draw between regularization and orthogonality, LAEs and probabilistic PCA, and the topology of Grassmannians are novel and provide a deeper understanding of the loss landscapes of regularized linear autoencoders.

\section{Regularized LAEs}
\label{secloss}

In the Appendix \ref{bias and mean centering}, we provide a self-contained derivation of the fact that an LAE with bias parameters is equivalent to an LAE without bias parameters trained on mean-centered data \citep{bourlard}. So without loss of generality, we assume $X$ is mean centered and consider the following three LAE loss functions for fixed $\lambda > 0$:
\begin{align*}
    \mathcal{L}(W_1,W_2) &= ||X - W_2W_1X||_F^2 \\
    \mathcal{L}_\pi(W_1,W_2) &= \mathcal{L}(W_1,W_2) + \lambda||W_2W_1||_F^2\\
    \mathcal{L}_\sigma(W_1,W_2) &= \mathcal{L}(W_1,W_2) + \lambda(||W_1||_F^2 + ||W_2||_F^2)
\end{align*}
We call these the \textbf{unregularized}, \textbf{product}, and \textbf{sum} losses, respectively.

The product and sum losses mirror the loss functions of a linear \textit{denoising autoencoder} (DAE) \citep{vincent} and linear \textit{contractive autoencoder} (CAE) \citep{rifai} respectively. See Appendix \ref{denoising} for details.

\subsection{Visualizing LAE loss landscapes}
\label{sec1d}

We can visualize these loss functions directly in the case $n = m = k = 1$, as shown in Figure \ref{one dimensional}. In fact, working out the critical points in this scalar case led us to conjecture the general result in Section $\ref{seccrit}$. We invite the reader to enjoy deriving the following results and experimenting with these loss landscapes using our online \href{https://danielkunin.github.io/Regularized-Linear-Autoencoders/}{visualization tool}.

For all three losses, the origin $w_1 = w_2 = 0$ is the unique rank-0 critical point. For $\mathcal{L}$ and $\mathcal{L}_\pi$, the origin is always a saddle point, while for $\mathcal{L}_\sigma$ the origin is either a saddle point or global minimum depending of the value of $\lambda$.

For $\mathcal{L}$, the global minima are rank-1 and consist of the hyperbola\footnote{Identified with the components of $\mathrm{GL}_1(\mathbb{R}) \cong \mathbb{R} \backslash \{0\}$.}
$$w_2 w_1 = 1.$$
For $\mathcal{L}_\pi$, the global minima are rank-1 and consist of this hyperbola shrunk toward the origin as in ridge regression,
$$w_2 w_1 = (1 + \lambda x^{-2})^{-1}.$$
For $\mathcal{L}_\sigma$ the critical points depend on the scale of $\lambda$ relative to $x^2$. For $\lambda < x^2$, the origin is a saddle point and the global minima are the two isolated rank-1 critical points\footnote{Identified with the components of $\mathrm{O}_1(\mathbb{R}) \cong \{\pm 1 \}$.} cut out by the equations
$$w_2 w_1 = 1 - \lambda x^{-2}, \quad w_1 = w_2.$$
As $\lambda$ increases toward $x^2$, these minima move toward the origin, which remains a saddle point. As $\lambda$ exceeds $x^2$, the origin becomes the unique global minimum. This loss of information was our first hint at the connection to probabilistic PCA formalized in Theorem \ref{thmppca}.

\subsection{Regularization and orthogonality}

Adding $L_2$-regularization to the encoder and decoder separately reduces the symmetries of the loss from $\mathrm{GL}_k(\mathbb{R})$ to the orthogonal group $O_k(\mathbb{R})$. Two additional facts about the relationship between regularization and orthogonality have guided our intuition:

\begin{asparaenum}[(a)]
\item Orthogonal matrices are the determinant $\pm 1$ matrices of minimal Frobenius norm\footnote{Geometrically: the unit-volume parallelotope of minimal total squared side length is the unit hypercube.},
$$\argmin_{A} ||A||^2_F \ \ \ \mathrm{ s.t. } \ \ \det(A)^2 = 1.$$
\item Orthogonal matrices are the inverse matrices of minimum total squared Frobenius norm,
$$\argmin_{A,B} ||A||_F^2 + ||B||_F^2 \ \ \ \mathrm{ s.t. } \ \ AB = I,$$
and in particular $A = B^\intercal$ at all minima.
\end{asparaenum}
Both facts follow from the inequality of arithmetic and geometric means after casting the problems in terms of squared singular values\footnote{The squared Frobenius norm is their sum and the squared determinate is their product.}. While it was not immediately clear to us that the transpose relationship in (b) also holds at the minima of $\mathcal{L}_\sigma$, in fact, all critical points of an L$_2$-regularized linear autoencoder are symmetric:

\begin{theorem}[Transpose Theorem]
\label{orthogonality} All critical points of $\mathcal{L}_\sigma$ satisfy $W_1 = W_2^\intercal.$
\end{theorem}

Our proof uses elementary properties of positive definite matrices as reviewed in Appendix \ref{definite}. Note that without regularization, all critical points are psuedoinverses, $W_1 = W_2^+$, as is clear in the scalar case and derived in Section \ref{seccrit}.

\begin{proof}
Critical points of $\mathcal{L}_\sigma$ satisfy:
\begin{align*}
\frac{\partial \mathcal{L}_\sigma}{\partial W_1} &= 2W_2^\intercal(W_2W_1 - I)XX^\intercal + 2\lambda W_1 = 0,\\
\frac{\partial \mathcal{L}_\sigma}{\partial W_2} &= 2(W_2W_1 - I)XX^\intercal W_1^\intercal + 2\lambda W_2 = 0.
\end{align*}

We first prove that the matrix $$C = (I - W_2 W_1)XX^\intercal$$ is positive semi-definite\footnote{Intuitively, we expect this property so long as $W_2 W_1$ shrinks the principal directions of $X$, so that $I - W_2 W_1$ does as well.}. Rearranging $\frac{\partial \mathcal{L}_\sigma}{\partial W_2} W_2^\intercal$ gives
$$XX^\intercal(W_2 W_1)^\intercal = (W_2 W_1)XX^\intercal (W_2W_1)^\intercal + \lambda W_2 W_2^\intercal.$$
Both terms on the right are positive semi-definite, so their sum on the left is as well and therefore
$$XX^\intercal(W_2W_1)^\intercal \succeq (W_2W_1) XX^\intercal (W_2W_1)^\intercal.$$ Cancelling $(W_2 W_1)^\intercal$ via Lemma \ref{psdprops} gives $C \succeq 0$.

We now show the difference $A = W_1 - W_2^\intercal$ is zero. Rearranging terms using the symmetry of $C$ gives
$$0 = \frac{\partial \mathcal{L}_\sigma}{\partial W_1} - \frac{\partial \mathcal{L}_\sigma}{\partial W_2}^\intercal = 2 A (C + \lambda I).$$
Since $C \succeq 0$ and $\lambda > 0$ imply $C + \lambda I \succ 0$, we conclude from $$A (C + \lambda I) A^T = 0$$
that $A = 0$.
\end{proof}

\section{Bayesian Models}
\label{bayesian models}

In this section, we identify Bayesian counterparts of our three loss functions and derive a novel connection between (regularized) LAEs and (probabilistic) PCA. This connection enables the application of any LAE training method to Bayesian MAP estimation of the corresponding model.

Consider the rank-$k$ (self-)regression model
$$x_i = W_2 W_1 x_i + \varepsilon_i = W x_i + \varepsilon_i$$
where $W_1$ and $W_2$ act through their product $W$ and $\varepsilon_i \sim \mathcal{N}_m(0, 1)$.

\begin{compactitem}
    \item $\mathcal{L}$ is \textit{rank-k regression}. The prior on $W$ is the uniform distribution on $\mathbb{R}^{m \times m}$ restricted to rank-$k$ matrices\footnote{Note that rank-(k-1) matrices are a measure zero subset of rank-$k$ matrices.}.

    \item $\mathcal{L}_\pi$ is \textit{rank-k ridge regression}. The prior on $W$ is $\mathcal{N}_{m \times m}(0, \lambda^{-1})$ restricted to rank-$k$ matrices.

    \item $\mathcal{L}_\sigma$ is the model with $W_1$ and $W_2^\intercal$ independently drawn from $\mathcal{N}_{k \times m}(0, \lambda^{-1})$.
\end{compactitem}

Theorem \ref{unified statement} shows that the minima of $\mathcal{L}_\sigma$, or equivalently the MAP of the Bayesian model, are such that $W_2$ is the orthogonal projection onto the top $k$ principal directions followed by compression in direction $i$ via multiplication by the factor $(1 - \lambda \sigma_i^{-2})^\frac{1}{2}$ for $\sigma_i^2 > \lambda$ and zero otherwise. Notably, for principal directions with eigenvalues dominated by $\lambda$, all information is lost no matter the number of data points. The same phenomenon occurs for pPCA with respect to eigenvalues dominated by the variance of the noise, $\sigma^2$. Let's consider these Bayesian models side by side, with $W_0 \in \mathbb{R}^{m \times k}$ the parameter of pPCA\footnote{See Chapter 12.2 of \citet{bishop} for background on pPCA.}:

\begin{table}[H]
\centering
\begin{tabular}{@{}cc@{}} \toprule
Bayesian $\mathcal{L}_\sigma$ & pPCA\\ \midrule
$\begin{aligned}[t]
W_1, W_2^\intercal &\sim \mathcal{N}_{k \times m}(0, \lambda^{-1})\\
\varepsilon_i &\sim \mathcal{N}_m(0, 1)\\
x_i &= W_2 W_1 x_i + \varepsilon_i
\end{aligned}$ &
$\begin{aligned}[t]
z_i &\sim \mathcal{N}_k(0, 1)\\
\varepsilon_i &\sim \mathcal{N}_m(0, \sigma^2)\\
x_i &= W_0 z_i + \varepsilon_i\\
\end{aligned}$\\ \bottomrule
\end{tabular}
\end{table}

Comparing the critical points of $\mathcal{L}_\sigma$ in Theorem \ref{unified statement},
\begin{equation}
\label{ppca1}
W_1^T = W_2 = U_\mathcal{I} (I_\ell - \lambda \Sigma^{-2}_\mathcal{I})^{\frac{1}{2}} O^\intercal,
\end{equation}
and pPCA \citep{bishop99},
\begin{equation}
\label{ppca2}
W_0 = U_\mathcal{I} \Sigma_\mathcal{I}(I_\ell - \sigma^2 \Sigma^{-2}_\mathcal{I})^{\frac{1}{2}}O^\intercal,
\end{equation}
where $O \in \mathbb{R}^{k \times \ell}$ has orthonormal columns, we see that $\lambda$ corresponds to $\sigma^2$ (rather than the precision $\sigma^{-2}$) in the sense that principal directions with eigenvalues dominated by either are collapsed to zero. The critical points only differ in the factor by which the the remaining principal directions are shrunk. More precisely:

\begin{theorem}[pPCA Theorem]
\label{thmppca}
With $\sigma^2 = \lambda$, the critical points of $$\mathcal{L}^0_\sigma(W_1, W_2) = \mathcal{L}_\sigma(W_1(XX^\intercal)^{-\frac{1}{2}}, (XX^\intercal)^{-\frac{1}{2}} W_2)$$ coincide with the critical points of pPCA.
\end{theorem}
\begin{proof}
Multiplying the expression for $W_2$ in \eqref{ppca1} on the left by $(XX^\intercal)^{\frac{1}{2}}$ gives the expression for $W_0$ in \eqref{ppca2}.
\end{proof}
Interestingly, the generative model for $\mathcal{L}^0_\sigma$ differs from that of pPCA. For example, in the scalar case $\mathcal{L}^0_\sigma(w_1, w_2)$ is
$$x^2 \left(1-x^{-2} w_2w_1\right)^2 + \lambda x^{-2}(w_1^2 + w_2^2)$$
whereas the negative log likelihood of pPCA is
$$\frac{1}{2}\left(\ln(2\pi) + \ln(w_0^2+\sigma^2) + x^2 (w_0^2 + \sigma^2)^{-1}\right).$$

\section{Loss Landscapes}
\label{seccrit}

Having contextualized LAE models in a Bayesian framework, we now turn to understanding their loss landscapes. Symmetries such as \eqref{symg} exist because the model is expressed in an ``overparameterized" coordinate form rooted in classical linear algebra. This results in ``flat" critical manifolds rather than a finite number of critical points. In Section \ref{secgrass}, we remove all symmetries by expressing the loss geometrically over a topological domain. This results in $\binom{m}{k}$ critical points, and in particular a unique minimum. This intuition will pay off in Sections \ref{alltogether} and \ref{curvature}, where we fully characterize the critical manifolds and local curvatures of all three LAE loss landscapes.

\subsection{Critical points}
\label{secgrass}

We now consider reconstruction loss over the domain of $k$-dimensional planes through the origin in $\mathbb{R}^m$. This space has the structure of a $k(m-k)$-dimensional smooth, compact manifold called the Grassmannian of $k$-planes in $\mathbb{R}^m$ and denoted $\mathrm{Gr}_k(\mathbb{R}^m)$ \citep{hatcher}. We'll build intuition with a few simple examples.

\begin{figure}[tb]
\vskip 0.2in
\begin{center}
\begin{subfigure}{\columnwidth}
  \centerline{
  \includegraphics[width=\linewidth]{img/grassmannian/r2.jpeg}
  }
  \caption{Lines in the plane.}
\end{subfigure}
\begin{subfigure}{\columnwidth}
  \centerline{
  \includegraphics[width=\linewidth]{img/grassmannian/r3.jpeg}
  }
  \caption{Lines in space.}
\end{subfigure}
\caption{\textit{Left:} Principal directions of a point cloud $X$. \textit{Middle:} $\mathcal{L}_X$ as height function on the manifold of lines through the origin. \textit{Right:} Negative gradient flow of $\mathcal{L}_X$.}
\label{grassman}
\end{center}
\vskip -0.2in
\end{figure}

\begin{compactitem}
\item $\mathrm{Gr}_1(\mathbb{R}^2)$ is the space of lines through the origin in the plane, which may be smoothly parameterized by a counterclockwise angle of rotation of the $x$-axis modulo the half turn that maps a line to itself.

\item $\mathrm{Gr}_1(\mathbb{R}^3)$ is the space of lines through the origin in 3-space, also known as the real projective plane. We can visualize $\mathrm{Gr}_1(\mathbb{R}^3)$ as the northern hemisphere of the 2-sphere with equator glued by the antipodal map.

\item $\mathrm{Gr}_2(\mathbb{R}^3)$ is identified with $\mathrm{Gr}_1(\mathbb{R}^3)$ by mapping a plane to its 1-dimensional orthogonal complement.
\end{compactitem}

A point cloud $X$ in $\mathbb{R}^m$ determines a smooth function $$\mathcal{L}_X: \mathrm{Gr}_k(\mathbb{R}^m) \to \mathbb{R}$$ whose value on a $k$-plane is the sum of square distances from the points to the plane. Figure \ref{grassman} depicts $\mathcal{L}_X$ as a height function for $\mathrm{Gr}_1(\mathbb{R}^2)$ and $\mathrm{Gr}_1(\mathbb{R}^3)$. Note that the min and max in (a) are the principal directions $u_1$ and $u_2$, while the min, saddle, and max in (b) are the principal directions $u_1$, $u_2$, and $u_3$. At right, we depict the negative gradient flow $-\nabla \mathcal{L}_X$ on each space, with (b) represented as a disk with glued boundary. In (a), $u_1$ may descend to $u_2$ by rotating clockwise or counterclockwise\footnote{Formally, we mean there are two gradient trajectories (modulo time translation) that converge to $u_1$ and $u_2$ in each time direction asymptotically, namely the left and right halves of the circle.}. In (b), $u_1$ may descend to $u_2$ or $u_3$ by rotating in either of two directions in the plane they span, and $u_2$ may similarly descend to $u_3$.

The following theorem requires our assumption that the singular values of $X$ are distinct.

\begin{theorem}[Grassmannian Theorem]
\label{perfect}
$\mathcal{L}_X$ is a smooth function with $\binom{m}{k}$ critical points given by all rank-$k$ principal subspaces. In local coordinates near the critical point with principal directions $i_1 < \ldots < i_k$, $\mathcal{L}_X$ takes the form of a standard non-degenerate saddle with
\begin{equation}
\label{morseindex}
d_\mathcal{I} = \sum_{j=1}^k (i_j - j).
\end{equation}
descending directions.
\end{theorem}
The latter formula counts the total number of pairs $i < j$ with $i \in \mathcal{I}$ and $j \notin \mathcal{I}$. These correspond to directions to flow along $-\nabla\mathcal{L}_X$ by rotating one principal direction $u_i$ to another $u_j$ of higher eigenvalue, fixing the rest.

In Appendix \ref{morse}, we prove a stronger form of the Grassmannian Theorem by combining Theorem \ref{unified statement}, a commutative diagram \eqref{commutegrass} relating $\mathcal{L}_X$ and $\mathcal{L}$, and techniques from algebraic topology.

\subsection{Critical manifolds}
\label{alltogether}

Translating the Grassmannian Theorem back to the coordinate representation $\mathbb{R}^{k \times m} \times \mathbb{R}^{m \times k}$ introduces two additional phenomena we saw in the scalar case in Section \ref{sec1d}.

\begin{compactitem}
\item Each critical point on $\mathrm{Gr}_k(\mathbb{R}^m)$ corresponds to a manifold $\mathrm{GL}_k(\mathbb{R})$ or $\mathrm{O}_k(\mathbb{R})$ of rank-$k$ critical points.

\item Critical manifolds appear with rank less than $k$. In particular, $(0, 0)$ is a critical point for all three losses.
\end{compactitem}

Now let's now combine our topological and scalar intuition to understand the the loss landscapes of LAEs in all dimensions and for all three losses.

Theorem \ref{unified statement} requires our assumption\footnote{For the sum loss, we also assume $\lambda$ is distinct from all $\sigma_i^2$.} that $X$ has distinct singular values $\sigma_1 > \dots > \sigma_m > 0$. Let $u_1, \dots, u_m$ denote the corresponding left singular vectors (or principal directions) of $X$. For an index set $\mathcal{I} \subset \{1, \ldots, m \}$ we define:
\begin{compactitem}
    \item[$\bullet$] $\ell = |\mathcal{I}|$ and increasing indices $i_1 < \dots < i_\ell$,
    \item[$\bullet$] $\Sigma_\mathcal{I} = \mathrm{diag}(\sigma_{i_1}, \dots, \sigma_{i_\ell})\in \mathbb{R}^{\ell \times \ell}$,
    \item[$\bullet$] $U_\mathcal{I} \in \mathbb{R}^{m \times \ell}$ consisting of columns $i_1, \ldots, i_\ell$ of $U$,
    \item $F_\mathcal{I}$, the open submanifold of $\mathbb{R}^{k \times \ell}$ whose points are matrices $G$ with independent columns\footnote{Also known as the manifold of $l$-frames in $\mathbb{R}^k$.},
    \item $V_\mathcal{I}$, the closed submanifold of $\mathbb{R}^{k \times \ell}$ whose points are matrices $O$ with orthonormal columns\footnote{Also known as the Stiefel manifold $V_l(\mathbb{R}^k)$.}.
\end{compactitem}

\begin{theorem}[Landscape Theorem]
\label{unified statement}
For each loss, the critical points form a smooth submanifold  of $\mathbb{R}^{k \times m} \times \mathbb{R}^{m \times k}$.

For $\mathcal{L}$ and $\mathcal{L}_\pi$, this submanifold is diffeomorphic to the disjoint union of $F_\mathcal{I}$ over all $\mathcal{I} \subset \{1, \ldots, m \}$ of size at most $k$.

For $\mathcal{L}_\sigma$, this submanifold is diffeomorphic to the disjoint union of $V_\mathcal{I}$ over all $\mathcal{I} \subset \{1, \ldots, m_0 \}$ of size at most $k$, where $m_0$ is the largest index such that $\sigma_{m_0}^2 > \lambda$.

These diffeomorphisms map $G \in F_\mathcal{I}$ or $O \in V_\mathcal{I}$ to a critical point $(W_1, W_2)$ as follows:

\begin{table}[H]
\hspace*{-0.25cm}
\centering
\begin{tabular}{@{}lcc@{}} \toprule
    & $W_2$  & $W_1$ \\ \midrule
    $\mathcal{L}$ & $U_\mathcal{I} G^+$ & $G U_\mathcal{I}^\intercal$ \\
    $\mathcal{L}_\pi$ & $U_\mathcal{I} (I_\ell + \lambda \Sigma^{-2}_\mathcal{I})^{-\frac{1}{2}}G^+$ & $G (I_\ell + \lambda \Sigma^{-2}_\mathcal{I})^{-\frac{1}{2}} U_\mathcal{I}^\intercal$ \\
    $\mathcal{L}_\sigma$ & $U_\mathcal{I} (I_\ell - \lambda \Sigma^{-2}_\mathcal{I})^\frac{1}{2} O^\intercal$ & $O (I_\ell - \lambda \Sigma^{-2}_\mathcal{I})^\frac{1}{2} U_\mathcal{I}^\intercal$\\ \bottomrule
\end{tabular}
\end{table}
\end{theorem}

The proof of the Landscape Theorem \ref{unified statement} follows quickly from the Transpose Theorem \ref{orthogonality} and Proposition \ref{diagonal critical points}.

\begin{prop}
\label{diagonal critical points}
Let $D, S \in \mathbb{R}^{m \times m}$ be diagonal matrices such that $S$ is invertible and the diagonal of $D^2 S^{-2}D^2$ has distinct non-zero elements. Then the critical points of
$$\mathcal{L}_*(Q_1, Q_2) = \tr(Q_2 Q_1 S^2Q_1^\intercal Q_2^\intercal - 2Q_2Q_1D^2)$$
are smoothly parameterized as the disjoint union of $F_\mathcal{I}$ over all $\mathcal{I} \subset \{1, \ldots, m \}$ of size at most $k$. The diffeomorphism maps $G \in F_\mathcal{I}$ to a critical point $(Q_1, Q_2)$ as follows\footnote{Here $D_\mathcal{I}$ and $S_\mathcal{I}$ are defined like $\Sigma_\mathcal{I}$. $I_\mathcal{I}$ is defined like $U_\mathcal{I}$.}:
\begin{table}[H]
\centering
\begin{tabular}{@{}ccc@{}} \toprule
    & $Q_2$  & $Q_1$ \\ \midrule
    $\mathcal{L}_*$ & $D_\mathcal{I} S_\mathcal{I}^{-1} I_\mathcal{I} G^+$ & $G I_\mathcal{I}^\intercal D_\mathcal{I} S_\mathcal{I}^{-1}$\\ \bottomrule
\end{tabular}
\end{table}
\end{prop}

\begin{proof}[Proof of Theorem \ref{unified statement}]

Given the singular value decomposition $X = U\Sigma V^\intercal$, let $Q_1 = W_1U$ and $Q_2 = U^\intercal W_2$. By invariance of the Frobenius norm under the smooth action of the orthogonal group, we may instead parameterize the critical points of the following loss functions and then pull the result back to $W_1 = Q_1U^\intercal$ and $W_2 = UQ_2$:
\begin{align*}
\mathcal{L}(Q_1, Q_2) &= ||\Sigma - Q_2Q_1\Sigma||_F^2\\
\mathcal{L}_\pi(Q_1, Q_2) &= \mathcal{L}(Q_1, Q_2) + \lambda ||Q_2Q_1||_F^2\\
\mathcal{L}_\sigma(Q_1, Q_2) &= \mathcal{L}(Q_1, Q_2) + \lambda(||Q_1||_F^2 + ||Q_2||_F^2)
\end{align*}

$\mathcal{L}$ expands to
$$\tr(Q_2 Q_1\Sigma^2 Q_1^\intercal Q_2^\intercal -2Q_2Q_1\Sigma^2 + \Sigma^2).$$
By Proposition \ref{diagonal critical points} with $S = \Sigma$ and $D = \Sigma$, the critical points have the form
\begin{table}[H]
\centering
\begin{tabular}{@{}ccc@{}} \toprule
    & $Q_2$  & $Q_1$ \\ \midrule
    $\mathcal{L}$ & $I_\mathcal{I}G^{+}$ & $GI_\mathcal{I}^\intercal$\\ \bottomrule
\end{tabular}
\end{table}

$\mathcal{L}_\pi$ expands to
$$\tr(Q_2 Q_1 (\Sigma^2 + \lambda I) Q_1^\intercal Q_2^\intercal -2Q_2Q_1\Sigma^2 + \Sigma^2).$$
By Proposition \ref{diagonal critical points} with $S = (\Sigma^2 + \lambda I)^{\frac{1}{2}}$ and $D = \Sigma$, the critical points have the form
\begin{table}[H]
\centering
\begin{tabular}{@{}ccc@{}} \toprule
    & $Q_2$  & $Q_1$ \\ \midrule
    $\mathcal{L}_\pi$ & $I_\mathcal{I}(I_\ell + \lambda \Sigma^{-2}_\mathcal{I})^{-\frac{1}{2}}G^{+}$ & $G(I_\ell + \lambda \Sigma^{-2}_\mathcal{I})^{-\frac{1}{2}}I_\mathcal{I}^\intercal$\\ \bottomrule
\end{tabular}
\end{table}
By Lemma \ref{sum of fro} with $A = Q_2$ and $B = Q_1$, $\mathcal{L}_\sigma$ expands to the sum of two functions:
\begin{align*}
\mathcal{L}_1(Q_1, Q_2) &= \tr(Q_2 Q_1 \Sigma^2 Q_1^\intercal Q_2^\intercal -2Q_2Q_1(\Sigma^2 - \lambda I) + \Sigma^2)\\
\mathcal{L}_2(Q_1, Q_2) &= \lambda ||Q_1 - Q_2^\intercal||_F^2.
\end{align*}
So at a critical point, $\nabla \mathcal{L}_1(Q_1, Q_2) = -\nabla \mathcal{L}_2(Q_1, Q_2)$ and $Q_1 = Q_2^\intercal$ by the Transpose Theorem \ref{orthogonality}. The latter also implies $\nabla \mathcal{L}_2(Q_1, Q_2) = 0$. So the critical points of $\mathcal{L}_\sigma$ coincide with the critical points of $\mathcal{L}_1$ such that $Q_1 = Q_2^\intercal$.

By Proposition \ref{diagonal critical points} with $S = \Sigma$ and $D = (\Sigma^2 - \lambda I)^{\frac{1}{2}}$, these critical points have the form
\begin{table}[H]
\centering
\begin{tabular}{@{}ccc@{}} \toprule
    & $Q_2$  & $Q_1$ \\ \midrule
    $\mathcal{L}_\sigma$ & $I_\mathcal{I}(I_\ell - \lambda \Sigma^{-2}_\mathcal{I})^{\frac{1}{2}}O^\intercal$ & $O(I_\ell - \lambda \Sigma^{-2}_\mathcal{I})^{\frac{1}{2}}I_\mathcal{I}^\intercal$\\ \bottomrule
\end{tabular}
\end{table}
In particular, real solutions do not exist for $\sigma_i^2 < \lambda$.
\end{proof}

\subsection{Local curvature}
\label{curvature}

By Theorem \ref{unified statement}, the critical landscape is a disjoint union of smooth manifolds, each at some height. We now prove that the Hessian is non-degenerate in the normal directions to the critical landscape. As discussed in Appendix \ref{morse}, such functions are called \textit{Morse-Bott} and studied extensively in differential and algebraic topology.

\begin{theorem}[Curvature Theorem]
\label{morse-bott}
In local coordinates near any point on the critical manifold indexed by $\mathcal{I}$, all three losses take the form of a standard degenerate saddle with $d_\mathcal{I} + (k-\ell)(m-\ell)$ descending directions.
\begin{compactitem}
\item $\mathcal{L}$ and $\mathcal{L}_\pi$ have $k\ell$ flat directions.
\item  $\mathcal{L}_\sigma$ has $k\ell - \binom{\ell + 1}{2}$ flat directions.
\end{compactitem}
The remaining directions are ascending.
\end{theorem}
\begin{proof}
In addition to the descending directions of $\mathcal{L}_X$, there are $(k-\ell)(m-\ell)$ more that correspond to scaling one of $k-\ell$ remaining slots in $G$ or $O$ toward one of $m-\ell$ available principal directions\footnote{In Figure \ref{one dimensional}c, these two gradient trajectories descend from the yellow saddle at $0$ to the two red minima at $\pm u_1$.}. For $\mathcal{L}$ and $\mathcal{L}_\pi$, the ascending directions are the $k(m-k) - d_\mathcal{I}$ ascending directions of $\mathcal{L}_X$; for $\mathcal{L}_\sigma$, an additional $\binom{\ell + 1}{2}$ ascending directions preserve the reconstruction term while increasing the regularization term by decreasing orthogonality. The remaining (flat) directions are tangent to the critical manifold, itself diffeomorphic to the manifold of (orthonormal) $l$-frames in $\mathbb{R}^k$.
\end{proof}

\section{Empirical Illustration}
\label{secemp}

In this section, we illustrate the Landscape Theorem \ref{unified statement} by training an LAE on synthetic data and visualizing properties of the learned weight matrices. See Appendix \ref{mnist experiments} for experiments on real data. Corollary \ref{morse-bott} implies that gradient descent and its extensions will reach a global minimum, regardless of the initialization, if trained for a sufficient number of epochs with a small enough learning rate \citep{zhu}.

\subsection{Synthetic data}
\label{synetheic experiments}

\begin{figure}[tb]
\begin{center}
\centerline{\includegraphics[width=\columnwidth]{img/synthetic/difference/difference.jpeg}}
\caption{Distance between $W_1$ and $W_2^\intercal$ during training.}
\label{difference}
\end{center}
\vskip -0.2in
\end{figure}

\begin{figure}[tb]
\begin{center}
\begin{subfigure}{0.32\linewidth}
  \includegraphics[width=\columnwidth]{img/synthetic/outer_product/noreg.png}
  \caption{Unregularized}
\end{subfigure}
\begin{subfigure}{0.32\linewidth}
  \includegraphics[width=\columnwidth]{img/synthetic/outer_product/product.png}
  \caption{Product}
\end{subfigure}
\begin{subfigure}{0.32\linewidth}
  \includegraphics[width=\columnwidth]{img/synthetic/outer_product/sum.png}
  \caption{Sum}
\end{subfigure}
\caption{Heat map of the matrix $\begin{bmatrix}U & V_*\end{bmatrix}^\intercal \begin{bmatrix}U & U_*\end{bmatrix}$. Black and white correspond to $-1$ and $1$, respectively.}
\label{singular vectors}
\end{center}
\vskip -0.2in
\end{figure}

\begin{figure}[tb]
\begin{center}
\begin{subfigure}{0.49\linewidth}
  \centerline{
  \includegraphics[width=\columnwidth]{img/synthetic/shrinkage/noreg.jpeg}
  }
  \caption{Unregularized}
\end{subfigure}
\begin{subfigure}{0.49\linewidth}
  \centerline{
  \includegraphics[width=\columnwidth]{img/synthetic/shrinkage/product.jpeg}
  }
  \caption{Product}
\end{subfigure}
\begin{subfigure}{0.74\linewidth}
  \centerline{
  \includegraphics[width=\columnwidth]{img/synthetic/shrinkage/sum.jpeg}
  }
  \caption{Sum}
\end{subfigure}
\begin{subfigure}{0.25\linewidth}
  \centerline{
  \includegraphics[width=0.8\columnwidth]{img/synthetic/shrinkage/legend.jpeg}
  }
    \caption*{}
\end{subfigure}
\caption{Illustration of the relationship between the eigenvalues of the weight matrix ($\tau^2$) and data matrix ($\sigma^2$) for various values of $\lambda$. Points are the empirical and lines are theoretical.}
\label{singular values}
\end{center}
\vskip -0.2in
\end{figure}

In the following experiments, we set $k = \lambda = 10$ and fix a data set $X \in \mathbb{R}^{20\times20}$ with singular values $\sigma_i = i$ and random left and right singular vectors under Haar measure. We train the LAE for each loss using the Adam optimizer for $4000$ epochs with random normal initialization, full batch, and learning rate $0.05$.

Figure \ref{difference} tracks the squared distance between $W_1$ and $W_2^\intercal$ during training. Indeed, only the the sum loss pulls the encoder and (transposed) decoder together as claimed in the Transpose Theorem \ref{orthogonality}.

Let $W_*$ be the product $W_2 W_1$ after training and fix the singular value decompositions
$$X = U \Sigma V^\intercal, \quad W_* = U_* \Sigma_* V_*^\intercal.$$
For each loss, the heat map of
$$\begin{bmatrix}
U^\intercal U & U^\intercal U_* \\
V_*^\intercal U & V_*^\intercal U_*
\end{bmatrix}$$
in Figure \ref{singular vectors} is consistent with $W_*$ approximating a global minimum defined by the Landscape Theorem. Namely, the lower right quadrant shows $U_* \approx V_*$ for each loss and the upper right and lower left quadrants show $U \approx U_*$ and $U \approx V_*$ up to column sign for the product and sum losses, \textit{but not for the unregularized loss}. That is, for the product and sum losses, the left singular vectors of $X$ are obtained as the right and left singular vectors of $W_*$.

The Landscape Theorem also gives explicit formulae for the eigenvalues of $W_*$ at convergence. Letting $\sigma_i^2$ and $\tau_i^2$ be the $i^{th}$ largest eigenvalues of $XX^\intercal$ and $W_*$, respectively, in Figure \ref{singular values} we plot the points
$\left(\sigma_i^2, \tau_i^2\right)$
for many values of $\lambda$. We superimpose a curve for each value of $\lambda$ defined by the theoretical relationship between $\sigma_i^2$ and $\tau_i^2$ in the Landscape Theorem. The (literal) alignment of theory and practice is visually perfect.

\section{Implications}
\label{secdisc}

\subsection{PCA algorithms}

The Landscape Theorem for $\mathcal{L}_\sigma$ implies those principal directions of $X$ with eigenvalues greater than $\lambda$ coincide with the top left singular vectors of the trained decoder. Hence we can perform PCA by training a regularized LAE and computing SVD of the decoder. For example, full-batch gradient descent corresponds to the following algorithm, which is guaranteed to converge to a (global) minimum for sufficiently small learning rate by the Curvature Theorem.

\begin{algorithm}[H]
  \caption{LAE--PCA}
  \label{lae-pca}
    \begin{algorithmic}
      \STATE {\bfseries input} $X \in \mathbb{R}^{m \times n}$; $k \leq m$; $\lambda, \alpha > 0$
      \STATE {\bfseries initialize} $W_1, W_2^\intercal \in \mathbb{R}^{k \times m}$
      \STATE {\bfseries while not converged}
      \STATE \quad $W_1 \mathrel{-}= \alpha \left( W_2^\intercal(W_2W_1 - I)XX^\intercal + \lambda W_1\right)$
      \STATE \quad $W_2 \mathrel{-}= \alpha \left( (W_2W_1 - I)XX^\intercal W_1^\intercal + \lambda W_2\right)$
      \STATE $U, \Sigma, \underline{\hspace{2mm}} = \mathrm{SVD}(W_2)$
\STATE {\bfseries return} $U, \ \lambda(I - \Sigma^2)^{-1}$
    \end{algorithmic}
\end{algorithm}

Note the SVD step is trivial because the decoder is only $m \times k$ dimensional with $k \ll n$. Hence optimizing this formulation of PCA reduces to optimizing the training of a regularized LAE. We have so far explored several simple ideas for optimizing performance\footnote{NumPy implementations and benchmarks of these ideas are available on \href{https://github.com/danielkunin/Regularized-Linear-Autoencoders}{GitHub}.}:

\begin{compactitem}
    \item \textit{First order.} The Transpose Theorem suggests ``tying" $W_1 = W_2^\intercal$ \textit{a priori}. In fact, for $\lambda=0$ and $k=1$, the gradient update for $\mathcal{L}_\sigma$ with tied weights is equivalent to Oja's rule for how neurons in the brain adapt synaptic strength \citep{Oja}. See Appendix \ref{oja proof} for a derivation.
    \item \textit{Second order.} The loss is convex in each parameter fixing the other, so $W_1$ and $W_2$ may be updated exactly per iteration by solving a system of $m$ or $k$ linear equations respectively.
\end{compactitem}

While many other methods have been proposed for recovering principal components with neural networks, they all require specialized algorithms for iteratively updating weights, similar to classical numerical SVD approaches \citet{Warmuth,Feng1,Feng2}. By contrast, reducing PCA to a small SVD problem by training a regularized LAE more closely parallels randomized SVD \citep{Halko}. We hope others will join us in investigating how far one can push the performance of \text{LAE-PCA}.

\subsection{Neural alignment}

In gradient descent via backpropagation, the weight matrix before a layer is updated by an error signal that is propagated by the transposed weight matrix after the layer. In the brain, there is no known physical mechanism to reuse feedforward connections for feedback or to enforce weight symmetry between distinct forward and backward connections. The latter issue is known as the \textit{weight transport problem} \citep{Grossberg} and regarded as central to the implausibility of backpropagation in the brain. Recently \citet{feedback} showed that forward weights can sufficiently align to fixed, random feedback weights to support learning in shallow networks, but \citet{Bartunov} demonstrated that this \textit{feedback alignment} and other biologically-plausible architectures break down for deep networks.

We are now investigating two approaches to weight transport inspired by the results herein. First, the Transpose Theorem \ref{orthogonality} suggests that weight symmetry emerges dynamically when feedback weights are updated to optimize forward-backward reconstruction between consecutive layers with weight decay. This \textit{information alignment} algorithm balances task prediction, information transmission, and energy efficiency. Second, Lemma \ref{sum of fro} expresses weight symmetry as balancing weight decay and self-amplification, lending greater biological plausibility to \textit{symmetric alignment}. We have verified that both information and (not surprisingly) symmetric alignment indeed align weights and are competitive with backprop when trained on MNIST and CIFAR10. In parallel, \citet{lillicrap} has independently shown that similar forms of local dynamic alignment scale to ImageNet.

\subsection{Morse homology}

In Appendix \ref{morse} we expand from the algebraic topology of learning PCA to that of learning by gradient descent in general. For example, we explain how Morse homology provides a principled foundation for the empirical observation of low-lying valley passages between minima in \citet{Garipov}. We are hopeful that this perspective will yield insights for efficient training and improved robustness and interpretation of consensus representations and ensemble predictions, for non-convex models arising in matrix factorization and deep learning.

\section{Conclusion}

In 1989, \citet{baldi1989} characterized the loss landscape of an LAE. In 2018, \citet{zhou} characterized the loss landscape of an autoencoder with ReLU activations on a single hidden layer. This paper fills out and ties together the rich space of research on linear networks over the last forty years by deriving from first principles a unified characterization of the loss landscapes of LAEs with and without regularization, while introducing a rigorous topological lens. By considering a simple but fundamental model with respect to regularization, orthogonality, and Morse homology, this work also suggests new principles and algorithms for learning.

\clearpage

Machine Learning (cs.LG); Machine Learning (stat.ML)