Abstract: Deep neural networks generalize well on unseen data though the number of parameters often far exceeds the number of training examples. Recently proposed complexity measures have provided insights to understanding the generalizability in neural networks from perspectives of PAC-Bayes, robustness, overparametrization, compression and so on. In this work, we advance the understanding of the relations between the network's architecture and its generalizability from the compression perspective. Using tensor analysis, we propose a series of intuitive, data-dependent and easily-measurable properties that tightly characterize the compressibility and generalizability of neural networks; thus, in practice, our generalization bound outperforms the previous compression-based ones, especially for neural networks using tensors as their weight kernels (e.g. CNNs). Moreover, these intuitive measurements provide further insights into designing neural network architectures with properties favorable for better/guaranteed generalizability. Our experimental results demonstrate that through the proposed measurable properties, our generalization error bound matches the trend of the test error well. Our theoretical analysis further provides justifications for the empirical success and limitations of some widely-used tensor-based compression approaches. We also discover the improvements to the compressibility and robustness of current neural networks when incorporating tensor operations via our proposed layer-wise structure.

\subsection{Generalization with Tensor Methods}
The theoretical analyses in~\citep{arora2018stronger} connect the generalizability of a neural network to its compressibility. In this work, we show how tensor methods can provide further insight into the understanding of this connection. We propose a series of measures to study the properties of such decomposed networks.

In our theoretical analysis, we present a rigorous proof showing the connection of these factors to the generalization error of a network. We will see in experiment section that this calculated bound is very effective at predicting the generalization error after some proper rescaling.

Although our theoretical analysis is derived for neural networks with {CPL}\xspace, our proposed generalization bounds are extended to any fully connected neural network (Theorem~\ref{thm:main_thm}), convolutional neural network (Theorem~\ref{thm:cnn_thm}), and neural network with skip connections (Theorem~\ref{thm:sc_thm}) via performing a CP decomposition on each layer of the network (see ``CNN-Project'' in algorithm \ref{alg:cnn_project}, and ``TNN-Project'' in algorithm \ref{alg:tfc_project}). For the extensions to hold, we only need to assume the weight matrices/tensors of the given neural network has some CP-decomposition with no reconstruction error; this is a mild assumption as we can increase the number of components needed to obtain such a CP decomposition. This practical concern prompts for the development of a simple layer-wise structure which we call {CP Layer}\xspace ({CPL}\xspace):

While the theoretical analysis works for any network, it is not straightforward to calculate the generalization bound on a deep and large network since during the calculation, an explicit tensor decomposition procedure has to be used (see ``CNN-Project'' in algorithm \ref{alg:cnn_project}, and ``TNN-Project'' in algorithm \ref{alg:tfc_project}). This practical concern prompts for the development of a simple layer-wise structure which we call {CP Layer}\xspace ({CPL}\xspace):

\begin{definition} [{CPL}\xspace]\label{def:cpl}
  A \textit{{CP Layer}\xspace}
  represents a given $N$-order weight tensor $\mytensor{K} \in \mathbb{R}^{d_0 \times d_1 \times \cdots \times d_N}$ with $R$ sets of components denoted by $\Big( \lambda^{(r)}, \{\myvector{v}^{(r)}_j\}_{j=1}^{N}\Big)_{r=1}^R$ where
  $\myvector{v}^{(r)}_j$ is a vector in $\mathbb{R}^{d_j}$ with $l_2$ norm equal to 1. To be more specific, $\mytensor{K} = \sum_{r=1}^{R} \lambda^{(r)} \myvector{v}^{(r)}_1 \otimes \cdots \otimes \myvector{v}^{(r)}_N$.

  \textit{Remark.} In a {CP Layer}\xspace, $R$ denotes the number of components and $\{\lambda^{(r)}\}_{r=1}^R$ denotes the CP spectrum of this layer.
 \end{definition}

Our proposed {CP layers}\xspace re-parametrize the networks' weights such that a CP-decomposed form can be learned easily in an end-to-end fashion.
We demonstrate by experiments that {CP layers}\xspace helps the network to earn a low-rank structure more easily, and thus helps with compression. This low-rankness also in return leads to a better generalization bound.
In short, our theory results in a practical method, which we show to be of great practical and theoretical use.

\section{Related Works}
\vspace{-0.25em}
\label{sec:other_related}
\textbf{Existing Metrics to Characterizing Generalization.} Classical and recent works have analyzed the generalizability of neural networks
from different perspective such as VC-dimension~\citep{bartlett1999almost, harvey2017nearly}, sharpness of the solution~\citep{keskar2016large}, robustness of the algorithm~\citep{xu2012robustness}, stability and robustness of the model~\citep{Hardt2016, Kuzborskij2018, Gonen2017, sokolic2016generalization} and over-parameterization~\citep{neyshabur2018towards, du2018power},
or using various approaches such as PAC-Bayes theory~\citep{mcallester1999some, mcallester1999pac, langford2002not, neyshabur2015norm, neyshabur2017exploring, dziugaite2017computing, golowich2018}, norm-based analysis~\citep{bartlett2002rademacher, neyshabur2015path, kawaguchi2017generalization, golowich2017size}, compression based approach~\citep{arora2018stronger}, and combinations of the above approaches~\citep{neyshabur2017exploring, neyshabur2017pac, bartlett2017spectrally, zhou2018non} (see~\citep{jakubovitz2018generalization} for a complete survey).
While these works provide deep theoretical insights to the understanding of the generalizability in neural networks, they did not provide practical techniques to improve generalization.

For the progress on non-vacuous generalization bounds, \cite{dziugaite2017computing} use non-convex optimization and PAC-Bayesian analysis to obtain a non-vacuous sample bound on MNIST, and ~\cite{zhou2018non} use a PAC-Bayesian compression approach to obtain non-vacuous generalization bounds on both MNIST and ImageNet via smart choices of the prior.
While being creative, both bounds are less intuitive and provide little insight into what properties are favorable for networks to have better generalizability.
In addition, the tensor-based compression methods are complementary to the compression approach used in~\citep{zhou2018non}, which combines pruning, quantization and huffman coding~\citep{han2015deep};
the tensor-based  compression methods can be combined with the approaches used in~\citep{han2015deep} to potentially tighten the generalization bound obtained in~\citep{zhou2018non}.

\textbf{Improving generalization in practice.} Authors of~\citep{neyshabur2015path} proposed an optimization method PATH-SGD which improves the generalization performance empirically. While~\citep{neyshabur2015path} focuses on the optimization approach, we provide a different practical approach that helps the understanding of the relations between the network architecture and its generalization ability.

\textbf{Comparison with Arora et al.~\citep{arora2018stronger}.} Besides practical improvements of generalization error, our work improves the results obtained by~\citep{arora2018stronger}: 1) we provide a tightened layer-wise analysis using tensor method to directly bound the operator norm of the weight kernel (e.g. Lemma~\ref{lem:rcpl-fc-operator-norm} and Lemma~\ref{lem:rcpl-conv-operator-norm}). The interlayer properties introduced by~\citep{arora2018stronger} are orthogonal to our proposed layer-wise properties and they can be well-combined; 2) in practice, our bound outperforms that of~\citep{arora2018stronger} in terms of the achieved degree of compression (detailed discussions in Section~\ref{sec:exp_gen_imp} and Section~\ref{sec:comp_arora});  3) for fully connected (FC) neural networks, our proposed reshaping factor (definition~\ref{def:rf}) further tightens the generalization bound as long as the inputs to the FC layers have some low-rank structures; 4) we extend our theoretical analysis to neural networks with skip connections, while the theoretical analysis in~\cite{arora2018stronger} only applies to FC and CNN.

\textbf{Comparison with existing CP decomposition for network compression.} While CP decomposition has been commonly used in neural network compression~\citep{denton2014exploiting, lebedev2014speeding, kossaifi2017tensor2}, our proposed compression method is very different from theirs. First, the the tensor contraction layer~\cite{kossaifi2017tensor2} is a special case of our {CPL}\xspace for FC layers when we set the number of components to be 1. Second, the number of components in our proposed {CPL}\xspace can be arbitrarily large (as it is a tunable hyper-parameter), while the number of components of layers in~\citep{denton2014exploiting, lebedev2014speeding, kossaifi2017tensor2} are determined by the compression ratio. Third, no tensor decomposition is needed for evaluating the generalizability and compressing neural networks with {CP layers}\xspace as the components from the CP decomposition are already stored as model parameters.
Moreover, as the smaller components in {CPL}\xspace are pruned during the compression, the performance of the compressed neural net is often preserved and thus no expensive fine tuning is required (see Table~\ref{tbl:compression}).  The depthwise-separable convolution used in MobileNet~\citep{howard2017mobilenets} is a specific implementation of {CPL}\xspace; thus, our theoretical analysis can provide generalization guarantees for the MobileNet architecture.

\section{Introduction}
\label{sec:intro}
Deep neural networks recently have made major breakthroughs in solving many difficult learning problems, especially in image classification~\citep{simonyan2014very, szegedy2015going, he2016deep, zagoruyko2016wide} and object recognition~\citep{krizhevsky2012imagenet,sermanet2013overfeat,simonyan2014very,zeiler2014visualizing}.
The success of deep neural networks depends on the high expressive power and the ability to generalize.
The high expressive power has been demonstrated empirically~\citep{he2016deep, zagoruyko2016wide} and theoretically~\citep{hornik1989multilayer, mhaskar2016deep}.
Yet, fundamental questions on why deep neural networks generalize and what enables their generalizability remain unsettled.

A recent work by~\cite{arora2018stronger} characterizes the generalizability of a neural network from a compression perspective ---
the capacity of the network is characterized through its compressed version.
The compression algorithm in~\cite{arora2018stronger} is based on random projection: each weight matrix of the compressed network are represented by a linear combination of basis matrices with entries i.i.d. sampled from $\pm 1$.
The effective number of parameters in the weight matrix is the number of coefficients in this linear combination obtained via projection --- the inner product between the original weight matrix and these basis matrices.
Though the idea of using compression in deriving the generalization bounds is novel,
the compression scheme in~\cite{arora2018stronger} could be made more practical since
(1) the cost of forwarding pass in the compressed network still remains the same as the cost in the original one, even though the effective number of parameters to represent the original weight matrices decreases;
(2) storing these random projection matrices could require more spaces than storing the original set of parameters.
We propose a new theoretical analysis based on a more practical, well-developed, and principled compression scheme using tensor methods.
Besides, we use tensor analysis to derive a much tighter bound for the layer-wise error propagation by exploiting additional structures in the weight tensors of neural networks,
which as a result significantly tightens the generalization error bound in~\cite{arora2018stronger}.

Our approach aims to characterize the network's compressibility by measuring the low-rankness of the weight kernels.
Existing compression methods in~\citep{jaderberg2014speeding, denton2014exploiting, lebedev2014speeding, kim2015compression, garipov2016ultimate, wang2018wide, su2019tensorial} implement low-rank approximations by performing matrix/tensor decomposition on weight matrices/kernels of well-trained models.
However, the layers of SOTA networks, such as VGG~\citep{simonyan2014very} and WRN~\citep{zagoruyko2016wide}, are not necessarily low-rank:
we apply CP-tensor decompositions~\citep{kolda2009tensor, anandkumar2014tensor,huang2015online,li2018guaranteed} to the weight tensors of well-trained VGG-16 and WRN-28-10, and \emph{the amplitudes of the components from the CP decomposition} (a.k.a \emph{\textbf{CP spectrum}}) are demonstrated by the brown curves in Figure~\ref{fig:eig_compare}, which indicate that the layers of these pre-trained networks are not low-rank.
Therefore a straightforward compression of the network cannot be easily achieved and computationally expensive fine tuning is often needed.

\input{figure_eig_compare}

To overcome this limitation, we propose a layer-wise structure design, \emph{{CP Layer}\xspace} ({CPL}\xspace), by incorporating the variants of CP decompositions in~\citep{jaderberg2014speeding, kossaifi2017tensor2, howard2017mobilenets}. {CPL}\xspace re-parametrizes the weight tensors such that a \emph{Polyadic form} (CP form)~\citep{kolda2009tensor} can be easily learned in an end-to-end fashion.

We demonstrate that empirically, {CPL}\xspace allows the network to learn a low-rank structure more easily, and thus helps with compression.
For example, from the pink curves in Figure~\ref{fig:eig_compare}, we see that neural networks with {CPL}\xspace have a spiky CP spectrum, which is an indication of low-rankness.
We rigorously prove that this low-rankness in return leads to a tighter generalization bound.
Moreover, we are the first to provide theoretical guarantees for the usage of CP decomposition in deep neural networks in terms of compressibility and generalizability.

\vspace{-0.5em}
\begin{definition}[\textbf{Proposed Architecture Layer}] \label{def:cpl}
A {CP Layer}\xspace ({CPL}\xspace) with width $R$ consists of $R$ set of parameters $\Big\{ \lambda^{(r)}, \big\{\myvector{v}^{(r)}_j\big\}_{j=1}^{N}  \Big\}_{r=1}^R$ where
  $\myvector{v}^{(r)}_j$ is a vector in $\mathbb{R}^{d_j}$ with unit norm.
  The weight kernel of this {CPL}\xspace is a $N$-order tensor defined as $\mytensor{K} := \sum_{r=1}^{R} \lambda^{(r)} \myvector{v}^{(r)}_1 \otimes \cdots \otimes \myvector{v}^{(r)}_N$, where $\otimes$ denotes the vector outer-product (tensor product) defined in Appendix \ref{def:tensor_product})~\footnote{The $(i_1,i_2,\ldots,i_N)^{{\mbox{\tiny th}}}$ element of the weight kernel is $\sum_{r = 1}^{R} \lambda^{(r)} \myvector{v}^{(r)}_1(i_1) \times \cdots \times \myvector{v}^{(r)}_N(i_N)$.}. Note that $\mytensor{K} \in \mathbb{R}^{d_1 \times \cdots \times d_N}$.
\end{definition}
\vspace{-0.5em}
\textit{Remark.} {CPL}\xspace allows for flexible choices of the structures since the number of components $R$ is a tunable hyper-parameter that controls the number of parameters in {CPL}\xspace.
The CP spectrum of this layer is denoted by $\{\lambda^{(r)}\}_{r=1}^R$ in a descending order.
The size of the weight kernel is $d_0 \times d_1 \times \cdots \times d_N$, while the number of parameters in {CPL}\xspace is $(d_0+d_1+\cdots+d_N+1) \times R$.

In contrast with existing works which apply CP decomposition to each layer of a reference network, no CP decomposition is needed since the components are explicitly stored as model parameters so that they can be learned from scratch via back-propagation.
Moreover, compression in {CP layers}\xspace is natural -- simply picking the top $\hat{R}$ components to retain and pruning out the rest of them.
Thus, the compression procedure using {CPL}\xspace does not require any costly fine-tuning while existing works on tensor-based compression may use hundreds of epochs for fine-tuning.

We further propose a series of simple, intuitive, data-dependent and easily-measurable \emph{properties} to measure the low-rankness in current neural networks. These properties not only guide the selection of the number of components to generate a good compression, but also tighten the bound of the layer-wise error propagation via tensor analysis.
The proposed properties
\begin{itemize}[leftmargin=*]
\vspace{-0.5em}
\item characterize the compressibility of the neural network,
 i.e., how much the original network can be compressed without compromising the performance on a training dataset more than certain range.
\vspace{-0.5em}
\item characterize the generalizability of the compressed network, i.e. tell if a neural network is trained using normal data or corrupted data.
\end{itemize}
\vspace{-0.5em}
In our theoretical analysis, we derive generalization error bounds for neural networks with {CP layers}\xspace, which take both the input distribution and the compressibility of the network into account.
We present a rigorous proof showing the connection of our proposed properties to the generalization error of a network.
We will see in experiment section that our proposed bound is very effective at predicting the generalization error.

Notice that, in this paper, the Polyadic form is chosen simply as a demonstration on how tensor methods could be used to improve the analysis of generalization bounds of deep neural networks. Therefore, follow-ups works could potentially analyze the effects of other tensor decomposition methods using our theoretical framework.

\textbf{Summary of Contributions}
\begin{enumerate}[leftmargin=*,itemsep=0em,topsep=0em]
	\item \textbf{Better generalization bound of practical use.}
	We verify that our generalization bounds can be used to guide the training of neural networks, since the calculated bound matches the trend of the test error on unseen data during the training process as shown in Figure~\ref{fig:bound}.
Moreover, we demonstrate that our generalization bound is in practice tighter than the bound proposed by~\citep{arora2018stronger} as shown in Figure~\ref{fig:bound_compare} and Table~\ref{tbl:effective}.
Notice that the generalization bound in~\citep{arora2018stronger} is already orders of magnitude better than previous norm-based or compression based bounds.
	\item \textbf{Intuitive measurements of compressibility and generalizability.}
	We propose a set of properties to characterize the low-rankness in the weight tensors of neural networks in Section~\ref{sec:cnn_properties}.
	Our theoretical analysis connects the measured low-rankness with the generalizability of the model, and such connections are verified in Figure~\ref{fig:properties}.
	\item \textbf{First theoretical guarantee} on the generalizability and robustness for neural network architectures that allow fast and real time predictions on devices with limited memory (e.g. the architecture designs proposed in~\citep{jaderberg2014speeding, kossaifi2017tensor2, howard2017mobilenets}, which uses variants of the Polyadic form).

	\vspace{-0.25em}
	\item \textbf{Practical improvements.}
	We demonstrate that pruning out the smaller components of CP decomposition in {CP layers}\xspace roughly preserves the test performance without computationally expensive fine tuning (see Section~\ref{sec:exp_cpl_compression} and Table~\ref{tbl:compression}) as our proposed layer-wise structure is easily compressible.
	Moreover, we discover that incorporating tensor operations via {CPL}\xspace reduces the generalization error of some well-known neural network architectures, and further improves the robustness of SOTA methods for learning under noisy labels (see Table~\ref{tbl:memorization}, Table~\ref{tbl:mnist}, Figure~\ref{fig:pf_sym_cr}, and Figure~\ref{fig:learning curve}).
\end{enumerate}

\section{Notations and Preliminaries}
\vspace{-0.25em}
\label{sec:setup}
In this paper, we use $S$ to denote the set of training samples drawn from a distribution $D$ with $|S| = m$. Let $n$ denote the number of layers in a given neural network, and superscripts of form ${ }^{(k)}$ denote properties related to the $k^{{\mbox{\tiny th}}}$ layer. We put ``CP'' in front of a network's name to denote such a network with {CP layers}\xspace (e.g. CP-VGG denotes a VGG with {CP layers}\xspace).
For any positive integer $n$, let $[n] := \{1,2, ..., n\}$. Let $\abs{a}$ denote the absolute value of a scalar $a$.
Given a vector $\myvector{a} \in \mathbb{R}^d$, a matrix $\mymatrix{A} \in \mathbb{R}^{d\times k}$, and a tensor $\mytensor{A} \in \mathbb{R}^{d_1\times d_2 \times d_3}$,  their norms are defined as follows:
(1) \textbf{Vector norm:} $\norm{\myvector{a}}$ denotes the $\ell_2$ norm.
(2) \textbf{Matrix norms:} Let $\nucnorm{\mymatrix{A}}$
denote its nuclear norm,  $\fronorm{\mymatrix{A}}$
denote its Frobenius norm, and $\norm{\mymatrix{A}}$
denote its operator norm (spectral norm), where $\sigma_i(\mymatrix{A})$ denotes the $i^{{\mbox{\tiny th}}}$ largest singular value of $\mymatrix{A}$.
(3) \textbf{Tensor norms:} Let $\norm{ \mytensor{A}} = \max_{x \in \mathbb{R}^{d_1}, y \in \mathbb{R}^{d_2}, z \in \mathbb{R}^{d_3}} \frac{\abs{A(x,y,z)} }{\norm{x}\norm{y}\norm{z}}$ denote its operator norm, and $\fronorm{ \mytensor{A}}$ its Frobenius norm.
Moreover, we use $\otimes$ to denote the \textbf{outer product operator}, and $\Conv$ to denote \textbf{the convolution operator}.
We use \textbf{$\ourfft{}{m}$} to denote \textbf{$m$-dimensional discrete Fourier transform}, and use tilde symbols
to denote tensors after DFT (e.g. $\Tilde{\mytensor{T}} = \ourfft{}{m}(\mytensor{T})$).
A \textbf{Polyadic decomposition (CP decomposition)}~\citep{kruskal1989rank, kolda2009tensor} of a $N$-order tensor $\mytensor{K} \in \mathbb{R}^{d_1 \times d_2 \times \cdots \times d_N}$ is a linear combination of rank-one tensors that is equal to $\mytensor{K}$:
$\mytensor{K} = \sum_{r=1}^R \lambda^{(r)} \myvector{v}^{(r)}_1 \otimes \cdots \otimes \myvector{v}^{(r)}_N$
 where $\forall r \in [R], \forall j \in [N], \norm{\myvector{v}^{(r)}_{j}} = 1$.
\textbf{Margin loss~\cite{arora2018stronger}:} we use $L_{\gamma}(\mathbb{M})$ and $\hat{L}_{\gamma}(\mathbb{M})$ to denote the expected and empirical margin loss of a neural network $\mathbb{M}$ with respect to a margin $\gamma \geq 0$. The expected margin loss of a neural network $\mathbb{M}$ is defined as
 $L_{\gamma}(\mathbb{M}):=\mathbb{P}_{(\myvector{x},y)\in D}\big[\mathbb{M}(\myvector{x})[y]\leq \gamma+\max_{i\neq y}\mathbb{M}(\myvector{x})[i]\big].$

\section{CNNs with {CPL}\xspace: Compressibility and Generalization}
\label{sec:cnn}
In this section, we derive the generalization bound for a convolutional neural network (denoted as $\mathbb{M}$) using tensor methods and standard Fourier analysis.
The complete proof is in Appendix Section~\ref{app:cnn_proofs}.
For simplicity, we assume that there is no pooling layer (e.g.~max pooling) in $\mathbb{M}$ since adding pooling layer will only lead to a smaller generalization bound (the perturbation error in our analysis decreases with the presence of pooling layers).
The derived generalization bound can be directly extended to various neural network architectures (e.g. neural networks with pooling layers, and neural networks with batch normalization).
The generalization bounds for fully connected neural networks and neural networks with skip connections are presented in Appendix Section~\ref{app:proofs} and ~\ref{app:sc_proofs} respectively.

\subsection{Compression of a CNN with {CPL}\xspace}
 We first illustrate how to compress any given CNN $\mathbb{M}$ by presenting a compression algorithm (Algorithm~\ref{alg:cnn_compress}). We will see that this compression algorithm guarantees a good estimation of the generalization bound for the compressed network $\hat{\mathbb{M}}$.

\textbf{Original CNN $\mathbb{M}$} is of \textbf{$n$} layers with \reluname activation, its $k^{{\mbox{\tiny th}}}$ layer weight tensor $\mytensor{\mytensor{M}}^{(k)}$
    is a $4^{{\mbox{\tiny th}}}$ order tensor of size = \emph{\# of input channel} $s^{(k)}$ $\times$ \emph{\# of output channel} $o^{(k)}$  $\times$ kernel height  $k_x^{(k)}$ $\times$ kernel width $k_y^{(k)}$.
    Let the $3^{\mbox{\tiny rd}}$ order tensor $\mytensor{X}^{(k)} \in \mathbb{R}^{H^{(k)} \times W^{(k)} \times s^{(k)}}$ denote the input to the $k^{{\mbox{\tiny th}}}$ layer, and $\mytensor{Y}^{(k)} \in \mathbb{R}^{H^{(k)} \times W^{(k)} \times o^{(k)}}$ denote the output of the $k^{{\mbox{\tiny th}}}$ layer before activation.
    Therefore $\mytensor{X}^{(k)} = \relu{\mytensor{Y}^{(k-1)}}$. We use $i$ to denote the index of input channels, and $j$ to denote the index of output channels.
	We further use $f$ and $g$ to denote the indices of width and height in the frequency domain.

\vspace{-0.5em}
\begin{proposition}[\textbf{Polyadic Form of original CNN $\mathbb{M}$}]
\label{assum:CP-cnn}
For each layer $k$, the weight tensor $\mytensor{\mytensor{M}}^{(k)}$ has a Polyadic form with number of components $R^{(k)} \leq \min\{s^{(k)} o^{(k)}, s^{(k)} k_x^{(k)} k_y^{(k)}, o^{(k)} k_x^{(k)} k_y^{(k)}\}$ ~\citep{kolda2009tensor}:
$\mytensor{\mytensor{M}}^{(k)} = \sum_{r = 1}^{R^{(k)}} \lambda_{r}^{(k)} \myvector{a}_{r}^{(k)} \otimes \myvector{b}_{r}^{(k)} \otimes \mymatrix{C}_{r}^{(k)}$,
where the CP-spectrum is in a descending order, i.e., $\lambda_{1}^{(k)} \geq \lambda_{2}^{(k)} \geq \cdots \geq \lambda_{R^{(k)}}^{(k)}$. All $\myvector{a}_r^{(k)}, \myvector{b}_r^{(k)}$ are unit vectors in $\mathbb{R}^{s^{(k)}}$ and $\mathbb{R}^{o^{(k)}}$ respectively, and $\mymatrix{C}_r^{(k)}$ is a matrix in $\mathbb{R}^{k_x^{(k)}\times k_y^{(k)}}$ with $\lVert \mymatrix{C}_r^{(k)}\rVert_{\textsf{F}}=1$.
The $R^{(k)}$ required for the Polyadic Form is called tensor rank.
\end{proposition}
\vspace{-0.5em}

\textbf{Transform original CNN to a CNN with {CP layers}\xspace.} By Proposition~\ref{assum:CP-cnn}, each weight tensor $\mytensor{\mytensor{M}}^{(k)}$ in $\mathbb{M}$ can be represented in a Polyadic form (CP form) and thus is transformed to a {CPL}\xspace.
The total number of parameters in {CPL}\xspace is $R^{(k)} \times (s^{(k)} + o^{(k)} + k_x^{(k)}  k_y^{(k)}+1)$.
Thus, a smaller $R^{(k)}$ leads to fewer number of effective parameters and indicates more compression.

\textbf{Compress Original CNN $\mathbb{M}$ to $\hat{\mathbb{M}}$.} We illustrate the compression procedure in Algorithm~\ref{alg:cnn_compress}. Feeding a CNN $\mathbb{M}$ to the compression algorithm, we obtain a compressed CNN $\hat{\mathbb{M}}$, where for each layer $k$, the weight tensor in $\hat{\mathbb{M}}$ is
$
\mytensor{\hat{M}}^{(k)} = \sum_{r = 1}^{\hat{R}^{(k)}} \lambda_{r}^{(k)} \myvector{a}_{r}^{(k)} \otimes \myvector{b}_{r}^{(k)} \otimes \myvector{c}_{r}^{(k)} \text{ for some $\hat{R}^{(k)} \leq R^{(k)}$ }.
$
Similarly, we use $\hat{\mytensor{X}}^{(k)}$ to denote the input tensor of the $k^{{\mbox{\tiny th}}}$ layer in $\hat{\mathbb{M}}$ and $\hat{\mytensor{Y}}^{(k)}$ to denote the output tensor of the $k^{{\mbox{\tiny th}}}$ layer in $\hat{\mathbb{M}}$ before activation.
Therefore $\hat{\mytensor{X}}^{(k)} = \relu{\hat{\mytensor{Y}}^{(k-1)}}$.
Notice that $\hat{\mytensor{X}}^{(k)}, \hat{\mytensor{Y}}^{(k)}$ are of the same shapes as $\mytensor{X}^{(k)}, \mytensor{Y}^{(k)}$  respectively and  $\mytensor{X}^{(1)} = \hat{\mytensor{X}}^{(1)}$ since the input data to both networks $\mathbb{M}$ and $\hat{\mathbb{M}}$ is the same.

The compression Algorithm~\ref{alg:cnn_compress} is designed to compress any CNN, and therefore requires applying explicit CP decompositions to the weight tensors of traditional CNNs (the step 3 in Algorithm~\ref{alg:cnn_compress}).
However, for a CNN with {CP layers}\xspace, these CP components are already stored as weight parameters in our {CPL}\xspace structure, and thus are known to the compression algorithm in advance. Therefore, no tensor decomposition is needed when compressing CNNs with {CPL}\xspace as we can prune out the components with smaller amplitudes directly.
\setlength{\textfloatsep}{0pt}
\input{algo_compressCNN}

\subsection{Characterizing Compressibility of CNN with {CPL}\xspace: Network Properties}
\label{sec:cnn_properties}
In this section, we propose the following layer-wise properties that can be evaluated based on the training data  $S$: \textit{\tflong (\tfshort)}, \textit{\nblong (\nbshort)}, and \textit{layer cushion (\lcshort)}~\citep{arora2018stronger}. These proposed properties are very effective at characterizing the compressibility of a neural network. As Algorithm~\ref{alg:cnn_compress}'s sub-procedure \FBRC selects a set of number of components $\{\hat{R}^{(k)}\}_{k=1}^{n}$ to obtain a compressed network $\hat{\mathbb{M}}$ whose output is similar to that of the original network (i.e., $\fronorm{\mathbb{M}(\mytensor{X}) -  \hat{\mathbb{M}}(\mytensor{X})} \leq \epsilon \fronorm{\mathbb{M}(\mytensor{X})}$ for any input $\mytensor{X} \in S$), our proposed properties will assist the selections of $\{\hat{R}^{(k)}\}_{k=1}^{n}$  to guarantee that Algorithm~\ref{alg:cnn_compress} returns a ``good'' compressed network.

\input{cnn_notations.tex}

Our proposed properties, orthogonal to the interlayer properties introduced in~\citep{arora2018stronger},  provide better measurements of the compressibility in each individual convolutional layer via the use of tensor analysis and Fourier analysis, and thus lead to a tighter bound of the layer-wise error propagation.

\subsection{Generalization Guarantee of CNNs}
\label{sec:cnn_gen_thm}

Based on Algorithm~\ref{alg:cnn_compress} and our proposed properties in section~\ref{sec:cnn_properties},
we obtain a generalization bound for the compressed convolutional neural network $\hat{\mathbb{M}}$ and, in section~\ref{sec:experiments}, we will evaluate this bound explicitly.

\begin{theorem} [\textbf{Main Theorem}]
\label{thm:cnn_thm}
For any convolutional neural network $\mathbb{M}$ with $n$ layers, Algorithm~\ref{alg:cnn_compress} generates a compressed CNN $\hat{\mathbb{M}}$ such that with high probability, the expected error $L_0(\hat{\mathbb{M}})$ is bounded by the empirical margin loss $\hat{L}_{\gamma}(\mathbb{M})$ (for any margin $\gamma \geq 0$) and a complexity term defined as follows
\begin{equation}
\begin{aligned}
& L_0(\hat{\mathbb{M}}) \leq \hat{L}_{\gamma}(\mathbb{M}) + \\
~ & \tilde{O}\left(\sqrt{\frac{ \sum_{k=1}^{n} \hat{R}^{(k)}(s^{(k)} + o^{(k)} + k_x^{(k)}k_y^{(k)} + 1)}{m}}\right)
\label{eq:bound}
\end{aligned}
\end{equation}
given that for all layer $k$, the number of components $\hat{R}^{(k)}$ in the compressed network satisfies that
\begin{equation}
\label{eq:main_eq}
\hat{R}^{(k)} = \min \Big\{ j \in [R^{(k)}] | \nb^{(k)}_j \Pi_{i=k+1}^{n} \tf^{(i)}_j \leq C \Big\}
\end{equation}
 $$ \text{ with }C = \frac{\gamma}{2 n \max_{\mytensor{X}  \in S } \fronorm{\mathbb{M}(\mytensor{X})}} \Pi_{i=k}^n \lc^{(i)} \fronorm{\mytensor{M}^{(i)}}$$
where $\tf^{(k)}_j$,  $\nb^{(k)}_j$ and $\lc^{(k)}$ are data dependent measurable properties --- \tflong, \nblong, and layer cushion of the $k^{{\mbox{\tiny th}}}$ layer in definitions~\ref{def:cnn_tf},~\ref{def:cnn_tnb} and~\ref{def:cnn_lc} respectively.
\end{theorem}
\textit{Remark.} How well the compressed neural network approximates the original network is related to the choice of $\hat{R}^{(k)}$.  Inside equation~\eqref{eq:main_eq}, $C$ is some value independent of the choice of $j$ in the inequality. Therefore, the number of components for the $k^{{\mbox{\tiny th}}}$ layer in the compressed network, $\hat{R}^{(k)}$, is the smallest $j \in [R^{(k)}] $ such that the inequality $\nb^{(k)}_j \Pi_{i=k+1}^{n} \tf^{(i)}_j \leq C$ holds. Hence, smaller {\tflong}s and {\nblong}s will make the LHS smaller, and larger layer cushions will make the RHS, $C$, larger. As a result, if the above inequality for each layer can be satisfied by a smaller $j$, the obtained generalization bound will be tighter as we can obtain a smaller $\hat{R}^{(k)}$.

\textbf{Analysis of generalization bounds in Theorem~\ref{thm:cnn_thm}}:
This proposed generalization error bound is proportional to the number of components in the {{CP layers}\xspace} of the compressed neural network.
Therefore, when the original neural network is highly compressible or very low-rank, the number of components needed will be lower, which thus renders a smaller generalization error bound.

The proof of Theorem~\ref{thm:cnn_thm} is in Appendix Section~\ref{app:cnn_proofs}), and the proof sketch is as follows.

\textbf{Proof sketch of Theorem~\ref{thm:cnn_thm}:}
We first establish that the difference of the outputs between the compressed CNN $\hat{\mathbb{M}}$ and the original CNN $\mathbb{M}$ is bounded by $\frac{\gamma}{2\max_{\mytensor{X}} \fronorm{\mathbb{M}(\mytensor{X})}}$ using Lemma~\ref{lem:cnn_lemma}.
Then we show the covering number of the compressed network $\hat{\mathbb{M}}$ is $\tilde{O}(d)$ via Lemma~\ref{lem:cnn_covering_number}, where $d$ denotes the total number of parameters in the compressed network. Bounding the covering number of CNNs with {CPL}\xspace to be of order $\tilde{O}(d)$ is non-trivial as we need careful handlings of the error propagations to avoid a dependence on the product of number of components.
After bounding the covering number, the rest of the proof follows from conventional learning theory and Theorem 2.1 in~\citep{arora2018stronger}.

\section{Experiments}
\label{sec:experiments}
\input{table_exp_power}
\input{table_label_noise}
\input{table_mnist}
\textbf{Architecture and optimization setting.} The architectures we use in the experiments consist of VGG-16~\citep{simonyan2014very},  {CP}-VGG-16, WRN-28-10~\citep{zagoruyko2016wide} and {CP}-WRN-28-10 (all with batch normalization). Details of the optimization settings are in~\ref{app:opt_setting}.

\subsection{Evaluation of Proposed Properties and Generalization Bounds}
\label{sec:exp_compare}
\textbf{Tighter Generalization Bound.} As shown in Fig~\ref{fig:bound_compare}, our bound is much tighter than the the state-of-the-art bound achieved in~\cite{arora2018stronger}. The effective number of parameters in~\cite{arora2018stronger} is orders of magnitude tighter than other capacity measures, such as $\ell_{1,\infty}$~\citep{bartlett2002rademacher}, Frobenius~\citep{neyshabur2015norm}, spec $\ell_{1,2}$~\citep{bartlett2017spectrally} and spec-fro~\citep{neyshabur2017pac} as shown in their Figure 4 Left.
The use of a more effective and practical compression approach allows us to achieve better compression (detailed discussions are in Appendix Section~\ref{sec:comp_arora}).

\input{figure_bound_and_compare}
\textbf{Generalization Bounds Correlated with Test Error.} We demonstrate how our generalization bound in Theorem~\ref{thm:cnn_thm} is practically useful in characterizing the generalizability during training.
In Figure~\ref{fig:bound},
\textbf{(1)} our calculated generalization bound matches well with the trend of the generalization error: after 140 epochs, the training error is almost zero but the test error continues to decrease in later epochs and our computed generalization bound captures these improvements especially well since epoch 150;
\textbf{(2)}
our calculated bound in Figure~\ref{fig:bound} for the well-trained {CP}-VGG-16 at epoch 300 is around $10$ while the total number of parameters in this {CP}-VGG-16 is around 14.7M.

\textbf{Compressibility of {CPL}\xspace: Property Evaluation.}
\label{exp:prop}
We evaluate and compare our proposed properties measuring compressibility, \emph{\tflong (\tfshort), \nblong (\nbshort) and layer cushion (\lcshort)}, on two different sets of models --- well-trained models with small generalization errors (thus expected to obtain small $\{\hat{R}^{(k)}\}_{k=1}^n$) vs. corrupted models with large generalization errors (thus expected to obtain large  $\{\hat{R}^{(k)}\}_{k=1}^n$).
In Figure~\ref{fig:properties}\textbf{(a)}, the number of components $\{\hat{R}^{(k)}\}_{k=1}^n$ returned by the compression algorithm is much smaller for well-trained models than that for corrupted models, which indicates that well-trained models have higher compressibility compared to corrupted ones as expected in our theory. Moreover, in Figure~\ref{fig:properties}\textbf{(b-d)}, we can indeed tell if the model is trained using ``good'' data or corrupted data by evaluating our proposed properties.

\input{figure_properties}

We further apply Algorithm~\ref{alg:cnn_compress} to these well-trained and corrupted models to investigate the consistency between the compression performance of Algorithm~\ref{alg:cnn_compress} and our theoretical results: on average, Algorithm~\ref{alg:cnn_compress} achieves a 31.83\% compression rate on the well-trained models, but only an 89.7\% compression rate on the corrupted models (lower compression rate is better as it implies a smaller generalization error bound). Clearly, the low-rank structures in well-trained models allow them to be compressed much further, consistent with our theoretical analysis of Algorithm~\ref{alg:cnn_compress}.

\subsection{Generalization Improvement on Real Data Experiments
\label{sec:exp_gen_imp}

\textbf{Expressive Power of Neural Networks with {CP layers}\xspace.} As shown in Table~\ref{table:exp_power}, neural networks equipped with {CP layers}\xspace maintain competitive training and test accuracies.

\textbf{Generalization Improvements under Label Noise.}
The memorization effect is directly linked to the deteriorated generalization performance of the network~\citep{zhang2016understanding}. Therefore we study how our proposed {CPL}\xspace structure affects the generalizability of a neural network with presence of strong memorization effect --- under label noise setting.
We assign random labels to a proportion of the training data and train the neural network until convergence. Then we test the network's performance on the uncorrupted test data.
As shown in Table~\ref{tbl:memorization},  {CP}-VGG consistently achieves better generalization performance compared to the traditional VGG under various label corruption ratios.

Our {CPL}\xspace, combined with co-teaching (CT)~\citep{han2018co} (the SOTA method for defeating label noise) further improves its performance as shown in Table~\ref{tbl:mnist} where we also compare our method CT+{CPL}\xspace against other different label-noise methods~\citep{han2018co}.
Besides, in Figure~\ref{fig:pf_sym_cr}, our method CT+{CPL}\xspace consistently outperforms the SOTA method (CT) with various choices of number of components.

\subsection
{CPL}\xspace Is Natural for Compression}
\label{sec:exp_cpl_compression}

Applying {CPL}\xspace for neural network compression is extensively studied in~\cite{su2019tensorial}, therefore we focus on explaining why {CPL}\xspace is natural for compression and analyzing the compressibility of {{CPL}\xspace}s.

\textbf{Low Rankness in Neural Networks with {CPL}\xspace vs Traditional Neural Networks}. The low rankness of a {CP}-VGG and a traditional VGG is demonstrated by Figure~\ref{eig_spectrum} where we display the ratios of the number of components with amplitudes above a given threshold 0.2. We clearly see that VGG with {CPL}\xspace exhibits low rankness consistently for all layers while the traditional VGG is not low-rank. Notice that the CP spectrum in each {CPL}\xspace is normalized by dividing the largest amplitude and the CP components of traditional VGG are obtained via explicit CP decompositions with reconstruction error set to 1e-3.

\input{figure_eigspec_layers}

\textbf{No Fine-tuning Needed for {CPL}\xspace.} Many works using tensor methods for neural network compression require computationally expensive fine-tuning (e.g. 200 epochs end-to-end training on the compressed networks) to recover the compressed network's test performance~\cite{jaderberg2014speeding, denton2014exploiting, lebedev2014speeding, kim2015compression, garipov2016ultimate, wang2018wide, su2019tensorial}.
However, the compression we perform does not require any fine tuning since it directly prunes out the components with amplitudes below some given threshold.
\emph{In experiments, we compress a {CP}-WRN-28-10, which has the same number of parameters as WRN-28-10, by 8$\times$ with only 0.56\% performance drop on CIFAR10 image classification.}
The full compression results for {CP}-WRN-28-10 under different cutting-off thresholds are shown in Table~\ref{tbl:compression}, where components whose amplitudes are under the cutting-off threshold are pruned.

\section{Conclusion and Discussion}
In this work, we derive a practical compression-based generalization bound via the proposed layerwise structure {CP layers}\xspace, and demonstrate the effectiveness of using tensor methods in theoretical analyses of deep neural networks.
With a series of benchmark experiments, we show the practical usage of our generalization bound and the effectiveness of our proposed structure {CPL}\xspace in terms of compression and generalization.
A possible future direction is studying the effectiveness of other tensor decomposition methods such as Tucker or Tensor Train.

\section*{Acknowledgement}
This research was supported by startup fund from Department of Computer Science of University of Maryland, National Science Foundation IIS-1850220 CRII Award 030742- 00001, DOD-DARPA-Defense Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD), Laboratory for Physical Sciences at University of Maryland.
This research was also supported in part by JSPS Kakenhi (26280009, 15H05707 and 18H03201), Japan Digital Design and JST-CREST.
Huang was also supported by Adobe, Capital One and JP Morgan faculty fellowships.
We thank Ziyin Liu for supporting this research with great advice and efforts.
We thank Jin-peng Liu, Kai Wang, and Dongruo Zhou for helpful discussions and comments.
We thank Jingxiao Zheng for supporting additional computing resources.

\section{Additional Experimental Results}
\label{app:experiments}

\subsection{Architecture and optimization setting}
\label{app:opt_setting}
We train these four models (VGG-16, {CP}-VGG-16, WRN-28-10 and {CP}-WRN-28-10) using standard optimization settings with no dropouts and default initializations provided by PyTorch~\citep{paszke2017automatic}. We use a SGD optimizer with momentum=0.9, weight decay=5e-4, and initial learning rate=0.05 to start the training process. The learning rate is scheduled to be divided by 2 every 30 epochs for VGG-16 and {CP}-VGG-16. While for WRN-28-10 and {CP}-WRN-28-10, the learning rate is scheduled to be divided by 5 at the $60^{{\mbox{\tiny th}}}, 120^{{\mbox{\tiny th}}}$ and $160^{{\mbox{\tiny th}}}$ epoch. We run 300 epochs to train each VGG-16 and {CP}-VGG-16, and we run 200 epochs to train each WRN-28-10 and {CP}-WRN-28-10.

\subsection{Generalization bounds comparison with~\citep{arora2018stronger}}
\label{sec:comp_arora}
The generalization bound we calculated for a well-trained {CP}-VGG-16 (with the same \# of parameters as VGG-16) on CIFAR10 dataset is around $12$ (thus, of order $10^1$) according to the transformation $f(x) = x / 20 - 0.5$ applied in Figure~\ref{fig:bound}. Our evaluated bound is much better than naive counting of \# parameters. Although we may not be able to directly compare our calculated bound with that in~\citep{arora2018stronger}, which is roughly of order $10^5$ as~\citep{arora2018stronger} uses a VGG-19 to evaluate their generalization bound while our evaluation is done using a {CP}-VGG-16, we present in Table~\ref{tbl:effective} the effective number of parameters identified by our proposed bound. Compared with the effective number of parameters in~\citep{arora2018stronger} (Table 1 of~\citep{arora2018stronger}), we can see that
\textbf{(1)} our effective number of parameters is upper bounded by the total number of parameters in original network (thus, the compression ratio is bounded by $1$), while the effective number identified by~\citep{arora2018stronger} could be several times larger than the original number of parameters (e.g. based on Table 1 of~\citep{arora2018stronger}, their effective number of parameters in layer 4 and 6 are more than 4 times of the original number of parameters);
\textbf{(2)} the effective number of parameters in~\citep{arora2018stronger} ignores the dependence on depth, log factors and constants, while our effective number of parameters in Table~\ref{tbl:effective} is exactly the actual number of parameters in the compressed network without these dependences.
\input{table_effective_num}

\subsection{Neural networks with {CPL}\xspace are natural for compression}
The compression results in Table~\ref{tbl:compression} are obtained directly without any fine tuning.
\input{table_compression}

\subsection{Improved Generalization Achieved by {CPL}\xspace}
We provide additional experimental details in the improved generalization ability achieved by {CPL}\xspace under label noise setting.
Our {CPL}\xspace combined with co-teaching (CT)~\citep{han2018co} outperforms SOTA method. Co-teaching~\citep{han2018co} is a training procedure for defeating label noise: it avoids overfitting to noisy labels by selecting clean samples out of the noisy ones and using them to update the network. Given the experimental results that neural networks with {CPL}\xspace tend to overfit less to noisy labels (Table 3), we combine Co-teaching to train networks with {CPL}\xspace on three different types of corrupted data (Table~\ref{tbl:mnist}). The hyperparameters we use in these experiments are the same as the ones in Co-teaching [2].

As shown in Table~\ref{tbl:mnist}, we compare our method CT+{CPL}\xspace against various label-noise methods~\citep{han2018co} under standard label noise setting~\citep{han2018co}.
\textbf{(1)} As shown in Figure~\ref{fig:pf_sym_cr}, our method (CT+{CPL}\xspace) consistently outperforms the SOTA method with various choices of the number of components. \textbf{(1.1)} Specifically, according to Table~\ref{tbl:mnist}, we see that combining {CPL}\xspace with co-teaching achieves the SOTA results on MNIST for PairFlip\footnote{PairFlip denotes that the label mistakes can only happen within very similar classes~\citep{han2018co}} with corruption rate $45\%$ and Symmetric\footnote{Symmetric denotes that the label mistakes may happen across different classes uniformly~\citep{han2018co}} with corruption rate $50\%$. \textbf{(1.2)} We also investigate the learning curve of our method compared with the SOTA (see Figure~\ref{fig:learning curve}.).
The models first reach best test accuracy early in the training, and then the test accuracy deteriorates as training goes on due to memorization effect. We see that our method always dominates the vanilla CT method when generalizability of the model starts to deteriorate due to memorization effect. This clearly shows that a neural network with {CPL}\xspace has better generalizability property than the plain neural network under this label noise setting.
\textbf{(2)} For the Symmetric-$20\%$ in Table~\ref{tbl:mnist}, as the label corruption rate is low, our method has a low effect in improving the generalization, which is expected.
\input{figure_pairflip_symm_cr}
\input{figure_learning_curve}

\textit{Remark.} The results displayed in Figure~\ref{fig:pf_sym_cr} and Figure~\ref{fig:learning curve} are based on our implementation of the CT method in order to achieve a fair comparison, while the results displayed in Table~\ref{tbl:mnist} are based on the reported accuracies by~\citep{han2018co} as we would like to compare our CT+{CPL}\xspace with other different label-noise methods as well.

\subsection{Compressibility of CPL: Property Evaluation {CPL}\xspace}
\input{figure_eigspec_all}
Figure~\ref{fig:cp_spec_all_n} displays the CP spectral of a well-trained, a corrupted, and a randomly initialized {CP}-VGG-16 (at the $13^{{\mbox{\tiny th}}}$ convolutional layer). For the unnormalized CP spectra of three models in Figure~\ref{fig:cp_spec_all_n}(a), we can see that the largest amplitude in the CP spectrum of the corrupted {CP}-VGG-16 is much smaller than that of well-trained and random models. Yet, a smaller leading value in the CP spectrum does not necessarily mean that the corrupted is more low rank. As shown in Figure~\ref{fig:cp_spec_all_n}(b), after normalizing the CP spectrum of each model by its largest amplitude, well-trained {CP}-VGG-16 still has the most low-rank CP spectrum (the blue curve) than that of corrupted or random models. Notice that the random model has the least low rankness since its weight tensors are the closest to random noise and thus it is hard to compress them.

We also compare our proposed properties among the three different sets of {CP}-VGG-16: well-trained, corrupted, and randomly initialized. As shown in Figure~\ref{fig:properties_full}, since random models have the least compressibility as their weight tensors are closest to random noise, properties that focus more on the compressibility of the model are larger on random models (e.g \nblong), which will lead to larger generalization bounds. In the meantime, properties that focus more on measuring the information loss after compression as well as the expressive power of the models (e.g. \fflong) are smaller for random models.
\input{figure_properties_full}
The reason why well-trained models have the largest \tflong is in Figure~\ref{fig:cp_spec_all_n} as the corrupted model usually has a very small leading value in its CP spectrum of later layers; yet as explained before, this does not necessarily indicate that corrupted models have more compressibility or low-rankness. The reason why the CP spectrum of corrupted models tend to have a small leading value is still a interesting question to study and we defer this to future work.

\textbf{Optimization settings for obtaining the well-trained, corrupted, and randomly initialized models of {CP}-VGG-16.}  We obtain well-trained {CP}-VGG-16 using the same hyperparameter settings as mentioned in Appendix Section~\ref{app:opt_setting}. For corrupted {CP}-VGG-16, we train the model under 50\% of label noise but using the same set of hyperparameters as the well-trained models. For {CP}-VGG-16 with random initialization, we just train the models for less than $1$ epoch.  For each set of these models, we obtain 200 instances using different random seeds.

\section{Common Definitions and Propositions}
\label{app:definitions}

In this section, we will briefly review three key concepts underlying all analysis in this work,
including {\em (multidimensional discrete Fourier transform)},
{\em CP decomposition} and {\em 2D-convolutional layer} in neural networks.

\subsection{Multidimensional Discrete Fourier Transform (MDFT)}
\label{sub:Fourier}

\begin{definition} \label{def:mdft}
(Multidimensional discrete Fourier transform, MDFT)
An $m$-dimensional MDFT $\ourfft{}{m}$ defines a mapping
from an $m$-order tensor $\mytensor{X} \in \mathbb{R}^{N_1 \times \cdots \times N_m}$
to another {\em complex} $m$-order tensor $\Tilde{\mytensor{X}} \in \mathbb{C}^{N_1 \times \cdots \times N_m}$ such that
\begin{equation}
\Tilde{\mytensor{X}}_{f_1,\dots, f_m}
= \left( \prod_{l = 1}^{m} N_l \right)^{-\frac{1}{2}}
\sum_{n_1=1}^{N_1} \cdots \sum_{n_m = 1}^{N_m}
\mytensor{X}_{n_1, \cdots, n_m}
\left( \prod_{l = 1}^{m} \omega_{N_l}^{f_l n_l} \right)
\label{eq:mdft}
\end{equation}
where $\omega_{N_l}=\exp \left(-\mathrm{j} 2\pi/N_l\right)$
and $\left( \prod_{l = 1}^{m} N_l \right)^{-\frac{1}{2}}$ is the {\em normalization factor} that makes $\ourfft{}{m}$ unitary.
Through out the paper, we will use symbols with tilde (e.g. $\Tilde{\mytensor{X}}$) to denote tensors after MDFT.

MDFT can also be applied on a subset of the dimensions $\mathcal{I} \subseteq [m]$,
and in this case we denote the mapping as $\ourfft{\mathcal{I}}{m}$.
\begin{equation}
\Tilde{\mytensor{X}}_{i_1,\dots, i_m}
= \left( \prod_{l \in \mathcal{I}} N_l \right)^{-\frac{1}{2}}
\sum_{\forall l \in \mathcal{I}}
\mytensor{X}_{n_1, \cdots, n_m}
\left( \prod_{l \in \mathcal{I}} \omega_{N_l}^{f_l n_l} \right)
\label{eq:mdft-partial}
\end{equation}
where $i_l = f_l$ if $l \in \mathcal{I}$ and $i_l = n_l$ for $l \notin \mathcal{I}$.

\end{definition}

\begin{fact} \label{fact:mdft-separability}
(Separability of MDFT)
An $m$-dimensional MDFT $\ourfft{}{m}$ is equivalent to a composition of $m$ unidimensional {DFT}s, i.e.\
\begin{equation}
\ourfft{}{m}=\ourfft{1}{m} \circ \ourfft{2}{m} \circ \cdots \circ \ourfft{m}{m}
\end{equation}
Similarly, $\ourfft{\mathcal{I}}{m}$ is identical to a composition of $|\mathcal{I}|$ unidimensional {DFT}s over corresponding dimensions.
\end{fact}

\begin{fact} \label{fact:mdft-unitary}
(MDFT is unitary) For an MDFT $\ourfft{}{}$, its adjoint $\ourfft{\ast}{}$ is equal to its inverse $\ourfft{-1}{}$, i.e.\ $\ourfft{\ast}{} = \ourfft{-1}{}$. An immediate corollary of this property is that the operator norm is invariant to MDFT: Given an operator $\mathcal{A}$, its operator norm of $\mathcal{A}$ is equal to $\ourfft{\ast}{} \mathcal{A} \ourfft{}{}$, i.e.\ $\|\mathcal{A}\| = \| \ourfft{\ast}{} \mathcal{A} \ourfft{}{} \|$.
\end{fact}

\subsection{CP decomposition}
\label{sub:cp-decomposition}

\begin{definition} \label{def:cp-decomposition}
(CP decomposition) Given an $m$-order tensor $\mytensor{T} \in \mathbb{R}^{N_1 \times \cdots \times N_m}$, a CP decomposition factorizes $\mytensor{T}$ into $m$ {\em core factors} $\{\mymatrix{K}^{l} \}_{l = 1}^{m}$ with $\mymatrix{K}^{l} \in \mathbb{R}^{R \times N_l}$ (with its $r^{{\mbox{\tiny th}}}$ column as $\myvector{k}^{l}_{r} \in \mathbb{R}^{N_l}$) such that
\begin{subequations}
\begin{gather}
\mytensor{T} =  \sum_{r = 1}^{R} \lambda_r \myvector{k}^{1}_{r} \otimes \cdots \otimes \myvector{k}^{m}_{r}
\label{eq:cp-decomposition} \\
\mytensor{T}_{n_1, \cdots, n_m} = \sum_{r = 1}^{R} \lambda_r \mymatrix{K}^{1}_{r, n_1} \cdots \mymatrix{K}^{m}_{r, n_m}
\end{gather}
\end{subequations}
where each column $\myvector{k}^{l}_{r}$ has unit $\ell_2$ norm,
i.e.\ $\|\myvector{k}^{l}_{r}\|_2 = 1, \forall r \in [R], l \in [m]$.
Without loss of generality, we assume the {\em CP eigenvalues} are positive and sorted in decreasing order,
i.e.\ $\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_m > 0$.
If the columns in $\mymatrix{K}^{l}$ are orthogonal, i.e.\ $\langle \myvector{k}^{l}_{r}, \myvector{k}^{l}_{r^{\prime}} \rangle = 1$ for $r \neq r^{\prime}$, the factorization is further named as {\em orthogonal CP decomposition}.
\end{definition}

\begin{lemma} \label{lem:cp-mdft}
(MDFT of CP decomposition) If an $m$-order tensor $\mytensor{T} \in \mathbb{R}^{N_1 \times \cdots \times N_m}$ takes a CP decomposition as in Eq.~\eqref{eq:cp-decomposition}, its (all-dimensional) MDFT $\mytensor{\Tilde{T}} = \ourfft{}{m}(\mytensor{T}) \in \mathbb{C}^{N_1 \times \cdots \times N_m}$ also takes a CP format as
\begin{gather}
\Tilde{\mytensor{T}} = \sum_{r = 1}^{R} \lambda_r
\tilde{\myvector{k}}^{1}_r \otimes \cdots \otimes \tilde{\myvector{k}}^{m}_r
\label{eq:cp-mdft} \\
\Tilde{\mytensor{T}}_{f_1, \cdots, f_m} = \sum_{r = 1}^{R} \lambda_r \tilde{\mymatrix{K}}^{1}_{r, f_1} \cdots \tilde{\mymatrix{K}}^{m}_{r, f_m}
\end{gather}
where $\tilde{\mymatrix{K}}^{l} = \ourfft{2}{2}(\mymatrix{K}^{l}), \forall l \in [m]$. The result can be extended to MDFT where a subset of dimensions are transformed.
\end{lemma}

\begin{proof} (of Lemma~\ref{lem:cp-mdft})
According to the definition of multidimensional discrete Fourier transform, we have
\begin{align}
\Tilde{\mytensor{T}}_{n_1, \cdots, n_m}
& = \left( \prod_{l = 1}^{m} N_l \right)^{-\frac{1}{2}}
\sum_{n_1=1}^{N_1} \cdots \sum_{n_m = 1}^{N_m}
\mytensor{T}_{n_1, \cdots, n_m}
\left( \prod_{l = 1}^{m} \omega_{N_l}^{f_l n_l} \right) \\
& = \left( \prod_{l = 1}^{m} N_l \right)^{-\frac{1}{2}}
\sum_{n_1=1}^{N_1} \cdots \sum_{n_m = 1}^{N_m}
\left( \sum_{r = 1}^{R} \lambda_r \tilde{\mymatrix{K}}^{1}_{r, n_1} \cdots \tilde{\mymatrix{K}}^{m}_{r, n_m} \right)
\left( \prod_{l = 1}^{m} \omega_{N_l}^{f_l n_l} \right) \\
& = \sum_{r = 1}^{R} \lambda_r
\left( N_1^{-1/2} \sum_{n_1 = 1}^{N_1} \mymatrix{K}^{1}_{r, n_1} \omega_{N_1}^{f_1 n_1} \right) \cdots
\left( N_m^{-1/2} \sum_{n_m = 1}^{N_m} \mymatrix{K}^{m}_{r, n_m} \omega_{N_m}^{f_m n_m} \right) \\
& = \sum_{r = 1}^{R} \lambda_r \tilde{\mymatrix{K}}^{1}_{r, f_1} \cdots \tilde{\mymatrix{K}}^{m}_{r, f_m}
\end{align}
which completes the proof.
\end{proof}

\subsection{2D-Convolutional Layer in Neural Networks}
\label{sub:cnn}

\begin{definition} \label{def:conv-layer}
(2D-convolutional layer)
In {CNN}s, a 2D-convolutional layer is
parametrized by a $4^{{\mbox{\tiny th}}}$-order tensor $\mytensor{M} \in \mathbb{R}^{k_x \times k_y \times T \times S}$ (with $k_x \times k_y$ kernels).
It defines a mapping from a $3^{{\mbox{\tiny rd}}}$-order input tensor $\mytensor{X} \in \mathbb{R}^{H \times W \times S}$ (with $S$ channels)
to another $3^{{\mbox{\tiny rd}}}$-order output tensor $\mytensor{Y} \in \mathbb{R}^{H \times W \times T}$ (with $T$ channels).
\begin{gather}
\mytensor{Y}_{\bm{:}, \bm{:}, t}
= \sum_{s = 1}^{S} \mytensor{M}_{\bm{:}, \bm{:}, t, s} \ast \mytensor{X}_{\bm{:}, \bm{:}, s} \\
\mytensor{Y}_{i, j, t} = \sum_{s = 1}^{S} \sum_{p, q} \mytensor{M}_{i - p, j - q, t, s} \mytensor{X}_{p, q, s}
\label{eq:conv-layer}
\end{gather}
where $\ast$ represents a 2D-convolution operator.
\end{definition}

\begin{lemma} \label{lem:cnn-mdft}
(Convolutional theorem of 2D-convolutional layer)
Suppose
$\mytensor{\Tilde{X}} =  \ourfft{1,2}{3}(\mytensor{X}) \in \mathbb{C}^{H \times W \times S}$,
$\mytensor{\Tilde{M}} = \ourfft{1,2}{4}(\mytensor{M}) \in \mathbb{C}^{H \times W \times T \times S}$ and
$\mytensor{\Tilde{Y}} =  \ourfft{1,2}{3}(\mytensor{Y}) \in \mathbb{C}^{H \times W \times T}$
are the MDFT of input, weights and outputs tensors
$\mytensor{X}$, $\mytensor{W}$ and $\mytensor{Y}$ respectively, then
these three tensors satisfy the following equation:
\begin{equation}
\mytensor{\Tilde{Y}}_{f, g, t} = \sqrt{H W} \sum_{s = 1}^{S} \mytensor{\Tilde{M}}_{f, g, t, s} \mytensor{\Tilde{X}}_{f, g, s}
\label{eq:cnn-mdft}
\end{equation}
Notice that the equation has a constant $\sqrt{H W}$ since we use a normalized MDFT.
\end{lemma}

\begin{proof} (of Lemma~\ref{lem:cnn-mdft})
The theorem can be easily proved by applying MDFT on both sides of Eq.~\eqref{eq:conv-layer}.
\begin{align}
\mytensor{\Tilde{Y}}_{f, g, t}
& = \frac{1}{\sqrt{H W}} \sum_{i, j} \mytensor{Y}_{i, j, t} \omega_{H}^{i f} \omega_{W}^{j g} \\
& = \frac{1}{\sqrt{H W}} \sum_{i, j} \left( \sum_{s = 1}^{S} \sum_{p, q} \mytensor{M}_{i - p, j - q, t, s} \mytensor{X}_{p, q, s} \right)  \omega_{H}^{i f} \omega_{W}^{j g} \\
& = \sqrt{H W} \sum_{s = 1}^{S}
\left( \frac{1}{\sqrt{H W}} \sum_{i, j} \mytensor{M}_{i - p, j - q, t, s} \omega_{H}^{(i - p)f} \omega_{W}^{(j - q)g} \right)
\left( \frac{1}{\sqrt{H W}} \sum_{p, q} \mytensor{X}_{p, q, s} \omega_{H}^{p f} \omega_{W}^{q g} \right) \\
& = \sqrt{H W} \sum_{s = 1}^{S} \mytensor{\Tilde{M}}_{f, g, t, s} \mytensor{\Tilde{X}}_{f, g, s}
\end{align}
\end{proof}

\begin{lemma} \label{lem:cnn-operator-norm}
(Operator norm of 2D-convolutional layer) Suppose we rewrite the tensors in matrix/vector form, i.e.\
$\mytensor{\tilde{X}}_{f, g, s} = \myvector{\tilde{x}}^{(f, g)}_s$,
$\mytensor{\tilde{M}}_{f, g, t} = \mymatrix{\tilde{M}}^{(f, g)}_{t, s}$,
$\mytensor{\tilde{Y}}_{f, g, t} = \myvector{\tilde{y}}^{(f, g)}_t$, then Eq.~\eqref{eq:cnn-mdft} can be written using matrix/vector products:
\begin{equation}
\myvector{\Tilde{y}}^{(f, g)}_{t} =
\sum_{s = 1}^{S} \mymatrix{\tilde{M}}^{(f, g)}_{t, s} \myvector{\tilde{y}}^{(f, g)}_t, ~ \forall f, g
\end{equation}
The operator norm of $\mytensor{M}$,
defined as $\|\mytensor{M}\| = \max_{\|\mytensor{X}\|_{F} = 1} \|\mytensor{Y}\|_{F}$,
can be obtained by spectral norms of $\mymatrix{\Tilde{M}}^{(f, g)}$ as:
\begin{equation}
\|\mytensor{M}\| = \sqrt{H W} \max_{f, g} \left\| \mymatrix{M}^{(f, g)} \right\|_{2}
\end{equation}
\end{lemma}

{\it Remarks.} The bound is first given by~\citet{sedghi2018singular}.
In this work, we provide a much simpler proof compared to the original one in~\citet{sedghi2018singular}.
In the next section, we show that the bound can be computed without evaluating the spectral norm
if the weights tensor $\mytensor{M}$ takes a CP format similar to Eq.~\eqref{eq:cp-decomposition}.

\begin{proof} (of Lemma~\ref{lem:cnn-operator-norm})
From Fact~\ref{fact:mdft-unitary}, we know that $\|\mytensor{M}\| = \|\mytensor{\Tilde{M}}\|$,
where $\|\mytensor{\Tilde{M}}\| = \max_{\|\mytensor{\Tilde{X}}\|_F = 1} \|\mytensor{\Tilde{Y}}\|_F$.
Next, we bound $\|\mytensor{\Tilde{Y}}\|_{F}^{2}$ (i.e.\ $\sum_{f, g} \left\|\myvector{\tilde{y}}^{(f, g)}\right\|_{F}^{2}$)
assuming $\|\mytensor{\Tilde{X}}\|_{2}^{2} = 1$ (i.e.\ $\sum_{f, g} \left\|\myvector{\tilde{x}}^{(f, g)}\right\|_{2}^{2} = 1$).
\begin{align}
\|\mytensor{\Tilde{Y}}\|_{F}^{2}
& = \sum_{f, g} \left\| \myvector{\tilde{y}}^{(f, g)} \right\|_{2}^{2} \\
& \leq {H W} \sum_{f, g} \left\| \mymatrix{\Tilde{M}}^{(f, g)} \right\|^2 \left\| \myvector{\tilde{x}}^{(f, g)} \right\|_2^2 \\
& \leq {H W} \max_{f, g} \left\| \mymatrix{\Tilde{M}}^{(f, g)} \right\|^2
\sum_{f, g} \left\| \myvector{\tilde{x}}^{(f, g)} \right\|_2^2 \\
& = H W \max_{f, g} \left\| \mymatrix{\Tilde{M}}^{(f, g)} \right\|^2 \\
\|\mytensor{\Tilde{Y}}\|_{F} & \leq \sqrt{H W} \max_{f, g} \left\| \mymatrix{\Tilde{M}}^{(f, g)} \right\|
\end{align}
We complete the proof by observing all inequalities can achieve equality simultaneously.
\end{proof}

\begin{definition} \label{def:tensor_product}
(Tensor product)
For vectors $\myvector{a} \in \mathbb{R}^n$, $\myvector{b} \in \mathbb{R}^m$,
and $\myvector{c} \in \mathbb{R}^p$, their tensor product $\myvector{a} \otimes \myvector{b} \otimes \myvector{c}$ is a 3-way tensor in $\mathbb{R}^{m \times n \times p}$, with the $(i,j,k)^{{\mbox{\tiny th}}}$ entry being $\myvector{a}_i \myvector{b}_j \myvector{c}_k$.
Similarly, for a matrix $\mymatrix{A} \in \mathbb{R}^{n \times m}$ and a vector $\myvector{c} \in \mathbb{R}^p$, their tensor product $\mymatrix{A} \otimes \myvector{c}$ is a $m \times n \times p$ tensor with the  $(i,j,k)^{{\mbox{\tiny th}}}$ entry being $\mymatrix{A}_{ij} \myvector{c}_k$.
 \end{definition}

\begin{definition}
\label{def:cnn_kp}
(Kronecker product). Let $\mymatrix{A}$ be an $n\times p$ matrix and $\mymatrix{B}$ an $m\times q$ matrix. The $mn\times pq$ matrix
 \begin{equation*}
 \mymatrix{A}\otimes\mymatrix{B}=
 \left[
 \begin{matrix}
    a_{1,1}\mymatrix{B} & a_{1,2}\mymatrix{B} & \cdots & a_{1,p}\mymatrix{B} \\
    a_{2,1}\mymatrix{B} & a_{2,2}\mymatrix{B} & \cdots & a_{2,p}\mymatrix{B} \\
    \vdots & \vdots & \vdots & \vdots \\
    a_{n,1}\mymatrix{B} & a_{n,2}\mymatrix{B} & \cdots & a_{n,p}\mymatrix{B}
  \end{matrix}
  \right]
  \end{equation*}
 is called the \textit{Kronecker product} of A and B. The outer product is an instance of \textit{Kronecker products}.
 \end{definition}

\section{CP Layers in Tensorial Neural Networks}
\label{app:cp-layers}

In this section, we will introduce three types of neural network layers,
whose parameters are factorized in CP format
as in Eq.~\eqref{eq:cp-decomposition} (with small variations).
For brevity, we omit the layer superscript and
denote the input, layer parameters and output
as $\mytensor{X}$, $\mytensor{M}$ and $\mytensor{Y}$,
and we use $\mytensor{Y} = \mytensor{M}\left( \mytensor{X} \right)$ to denote the relations between $\mytensor{X}$, $\mytensor{M}$ and $\mytensor{Y}$.

\subsection{CP 2D-convolutional Layer}
\label{sub:cpl-conv}

\begin{definition} \label{def:cpl-conv}
(CP 2D-convolutional layer)
For a given 2D-convolutional layer in Eq.~\eqref{eq:conv-layer},
a CP decomposition factorizes the weights tensor
$\mytensor{M} \in \mathbb{R}^{H \times W \times T \times S}$ into three {\em core factors}
$\mytensor{C} \in \mathbb{R}^{R \times k_x \times k_y}$, $\mymatrix{U} \in \mathbb{R}^{R \times T}$, $\mymatrix{V} \in \mathbb{R}^{R \times S}$
and a vector of {\em CP eigenvalues} $\lambda \in \mathbb{R}^{R}$ such that
\begin{gather}
\mytensor{M} = \sum_{r = 1}^{R} \lambda_r \mytensor{C}_{r} \otimes \myvector{u}_{r} \otimes \myvector{v}_r \\
\mytensor{M}_{i, j, t, s} = \sum_{r = 1}^{R} \lambda_r \mytensor{C}_{r, i, j} \mymatrix{U}_{r, t} \mymatrix{V}_{r, s}
\label{eq:cpl-conv}
\end{gather}
where $\lambda_r > 0$, $\|\mytensor{C}_r\|_F = 1$, $\|\myvector{u}_r\|_2 = 1$ and $\|\myvector{v}_r\|_2 = 1$ for all $r \in [R]$.
\end{definition}

\begin{lemma} \label{lem:cpl-conv-operator-norm}
(Operator norm of CP 2D-convolutional layer)
For a 2D-convolutional layer whose weights tensor takes
a CP format as in Eq.~\eqref{eq:cpl-conv},
the operator norm $\|\mytensor{M}\|$ is bound by the CP eigenvalues $\lambda$ as
\begin{equation}
\|\mytensor{M}\| \leq \sqrt{H W} \sum_{r = 1}^{R} |\lambda_r| \max_{f, g} \left| \Tilde{\mytensor{C}}^{(f, g)}_r \right|
\end{equation}
\end{lemma}

\begin{proof} (of Lemma~\ref{lem:cpl-conv-operator-norm})
From Fact~\ref{fact:mdft-unitary}, the operator norm of $\mytensor{M}$
is equal to the one of its MDFT $\mytensor{\Tilde{M}} = \ourfft{1, 2}{4}(\mytensor{M})$,
i.e.\ $\|\mytensor{M}\| = \|\Tilde{\mytensor{M}}\|$.
According to Lemma~\ref{lem:cnn-operator-norm},
it is sufficient to compute the spectral norm for each matrix $\Tilde{\mymatrix{M}}^{(f, g)}$ individually.
Notice that if $\mytensor{M}$ takes a CP format,
each $\Tilde{\mymatrix{M}}^{(f, g)}$ has a decomposed form as follows
\begin{subequations}
\begin{gather}
\mymatrix{\Tilde{M}}^{(f, g)}
= \sum_{r = 1}^{R}  \lambda_r
\mytensor{\Tilde{C}}^{(f, g)}_{r} \myvector{u}_{r} \myvector{v}_{r}^{\top} \\
\mymatrix{\Tilde{M}}^{(f, g)}_{t, s} =
\sum_{r = 1}^{R} \lambda_r
\mytensor{\Tilde{C}}^{(f, g)}_{r} \mymatrix{U}_{r, t} \mymatrix{V}_{r, s}
\end{gather}
\end{subequations}
where $\mytensor{\Tilde{C}} = \ourfft{2, 3}{3}(\mytensor{C})$ and $\mytensor{\Tilde{C}}^{(f, g)}_{r} = \mytensor{\Tilde{C}}_{r, f, g}$. The rest of the proof follows the definition of spectral norm of $\mymatrix{\Tilde{M}}$, i.e.\ $\|\Tilde{\mymatrix{M}}^{(f, g)}\|_2 = \max_{\|\myvector{a}\| = 1} \|\Tilde{\mymatrix{M}}^{(f, g)} \myvector{a}\|$. Let $\myvector{b} = \Tilde{\mymatrix{M}}^{(f, g)} \myvector{a}$, we can bound the $\ell_2$ norm of $\myvector{b}$:
\begin{align}
\left\|\myvector{b}\right\|_2 = \left\| \mymatrix{\Tilde{M}}^{(f, g)} \myvector{a} \right\|_2
& = \left\| \sum_{r = 1}^{R} \lambda_r \mytensor{\Tilde{C}}^{(f, g)}_{r} \myvector{u}_{r} \myvector{V}_{r}^{\top} \myvector{a} \right\|_2 \\
& \leq \sum_{r = 1}^{R} \left| \lambda_r \mytensor{\Tilde{C}}^{(f, g)}_r
\left( \myvector{v}_{r}^{\top} \myvector{a} \right) \right| \left\|\myvector{\myvector{u}_r} \right\|_2 \\
& = \sum_{r = 1}^{R} \left| \lambda_r \mytensor{\Tilde{C}}^{(f, g)}_r \left( \myvector{v}_{r}^{\top} \myvector{a} \right) \right| \\
& \leq \sum_{r = 1}^{R} |\lambda_r| \left| \mytensor{\Tilde{C}}^{(f, g)}_r \right|
\end{align}
Therefore, $\|\mytensor{M}\| = \|\Tilde{\mytensor{M}}\| = \sqrt{H W} \max_{f, g} \|\mymatrix{\Tilde{M}}^{(f, g)}\| \leq \sqrt{H W} \sum_{r = 1}^{R} |\lambda_r| \max_{f, g} \left| \mytensor{\Tilde{C}}^{(f, g)}_r \right|$.
\end{proof}

\subsection{Higher-order CP Fully-connected Layer}
\label{sub:rcpl-fc}

\begin{definition} \label{def:reshaped-fc}
(Higher-order fully-connected layer)
The layer is parameterized by a $2m^{{\mbox{\tiny th}}}$-order tensor
$\mytensor{M} \in \mathbb{R}^{T_1 \times \cdots \times T_m \times S_1 \times \cdots \times S_m}$.
It maps an $m^{{\mbox{\tiny th}}}$-order input tensor $\mytensor{X} \in \mathbb{R}^{S_1 \times \cdots \times S_m}$ to
another $m^{{\mbox{\tiny th}}}$-order output tensor $\mytensor{Y} \in \mathbb{R}^{T_1 \times \cdots \times S_m}$
with the following equation:
\begin{equation}
\mytensor{Y}_{t_1, \cdots, t_m} = \sum_{\forall l: S_l}
\mytensor{M}_{t_1, \cdots, t_m, s_1, \cdots, s_m} \mytensor{X}_{s_1, \cdots, s_m}
\label{eq:reshaped-fc}
\end{equation}
\end{definition}

\begin{definition} \label{def:rcpl-fc}
(Higher-order CP fully-connected layer)
Given a higher-order fully-connected layer in Eq.~\eqref{eq:reshaped-fc},
a CP decomposition factorizes the weights tensor
$\mytensor{M} \in \mathbb{R}^{T_1 \times \cdots \times T_m \times S_1 \times \cdots \times S_m}$
into $m$ core factors $\mytensor{K}^{m} \in \mathbb{R}^{R \times T_m \times S_m}$.
\begin{equation}
\mytensor{M}_{t_1, \cdots, t_m, s_1, \cdots, s_m} = \sum_{r = 1}^{R} \lambda_r
\mytensor{K}^{1}_{r, t_1, s_1} \cdots \mytensor{K}^{m}_{r, t_m, s_m}
\label{eq:rcpl-fc}
\end{equation}
For simplicity, we denote the $r^{{\mbox{\tiny th}}}$ slice of $\mytensor{K}^{l}$ as $\mymatrix{K}^{l}_{r} = \mytensor{K}^{l}_{r, \bm{:}, \bm{:}}$. We assume $\mymatrix{K}^{l}_r$ has unit Frobenius norm, i.e.\ $\|\mymatrix{K}^{l}_r\|_F = 1$ and $\lambda_r > 0$ for all $r \in [R]$.
\end{definition}

\begin{lemma} \label{lem:rcpl-fc-operator-norm}
(Operator norm of higher-order CP fully-connected layer)
For a higher-order fully layer whose weights tensor takes
a CP format as in Eq.~\eqref{eq:rcpl-fc},
the operator norm $\|\mytensor{M}\|$ is bound by the CP eigenvalues $\lambda$ as
\begin{equation}
\|\mytensor{M}\| \leq \sum_{r = 1}^{R} |\lambda_r|
\end{equation}
\end{lemma}

\begin{proof} (of Lemma~\ref{lem:rcpl-fc-operator-norm})
The proof follows directly the definition of operator norm
$\|\mytensor{M}\| = \max_{\|\mytensor{X}\|_F = 1} \|\mytensor{Y}\|_{F}$.
\begin{align}
\|\mytensor{Y}\|_{F} & \leq \sum_{r = 1}^{R} |\lambda_r|
\left\| \mymatrix{K}^{l}_m \right\|_2 \cdots \left\| \mymatrix{K}^{l}_1\right\|_2 \|\mytensor{X}\|_{F}
\\
& \leq \sum_{r = 1}^{R} |\lambda_r|
\left\|\mymatrix{K}^{l}_m\right\|_F \cdots \left\| \mymatrix{K}^{l}_1\right\|_F \|\mytensor{X}\|_{F} \\
& =  \sum_{r = 1}^{R} |\lambda_r| \|\mytensor{X}\|_{F} = \sum_{r = 1}^{R} |\lambda_r|
\end{align}
\end{proof}

\subsection{Higher-order 2D-convolutional layer}
\label{sub:rcpl-conv}

\begin{definition} \label{def:reshaped-conv-layer}
(Higher-order 2D-convolutional layer)
The layer is parameterized by a $(2m+2)^{{\mbox{\tiny th}}}$-order tensor
$\mytensor{M} \in \mathbb{R}^{k \times k \times T_1 \times \cdots \times T_m \times S_1 \times \cdots \times S_m}$.
It maps an $(m+2)^{{\mbox{\tiny th}}}$-order input tensor $\mytensor{X} \in \mathbb{R}^{H \times W \times S_1 \times \cdots \times S_m}$ to
another $(m+2)^{{\mbox{\tiny th}}}$-order output tensor $\mytensor{Y} \in \mathbb{R}^{H \times W \times T_1 \times \cdots \times S_m}$ as:
\begin{subequations}
\begin{gather}
\mytensor{Y}_{\bm{:}, \bm{:}, t_1, \cdots, t_m} =
\sum_{\forall l: s_l = 1}^{S_l}
\mytensor{M}_{\bm{:}, \bm{:}, t_1, \cdot, t_m, s_1, \cdots, s_m}
\ast \mytensor{X}_{\bm{:}, \bm{:}, s_1, \cdots, s_m}
\label{eq:reshaped-conv-layer} \\
\mytensor{Y}_{i, j, t_1, \cdots, t_m} = \sum_{\forall l: s_l = 1}^{S_l} \sum_{p, q}
\mytensor{M}_{i - p, j - q, t_1, \cdots, t_m, s_1, \cdots, s_m} \mytensor{X}_{p, q, s_1, \cdots, s_m}
\end{gather}
\end{subequations}
\end{definition}

\begin{definition} \label{def:rcpl-conv}
(CP decomposition of higher-order 2D-convolutional layer)
Given a higher-order 2D-convolutional layer in Eq.~\eqref{eq:reshaped-fc},
a CP decomposition factorizes the weights tensor
$\mytensor{M} \in \mathbb{R}^{H \times W \times T_1 \times \cdots \times T_m \times S_1 \times \cdots \times S_m}$
into $(m+1)$ core factors $\mytensor{C} \in \mathbb{R}^{R \times H \times W}$
and $\mytensor{K}^{l} \in \mathbb{R}^{R \times T_l \times S_l}, \forall l \in [m]$.
\begin{equation}
\mytensor{M}_{i, j, t_1, \cdots, t_m, s_1, \cdots, s_m} =
\sum_{r = 1}^{R} \lambda_r \mytensor{C}_{r, i, j}
\mytensor{K}^{1}_{r, t_1, s_1} \cdots \mytensor{K}^{m}_{r, t_m, s_m}
\label{eq:rcpl-conv}
\end{equation}
where we we assume $\mytensor{C}_r$ and
$\mymatrix{K}^{l}_r = \mytensor{K}^{l}_{r, \bm{:}, \bm{:}}$ has unit Frobenius norm,
i.e.\ $\|\mymatrix{K}^{l}_r\|_F = 1$ and $\|\mytensor{C}_r\|_F = 1$
\end{definition}

\begin{lemma} \label{lem:rcpl-conv-operator-norm}
(Operator norm of Higher-order CP 2D-convolutional layer)
For a higher-order 2D-convolutional layer layer whose weights tensor takes
a CP format as in Eq.~\eqref{eq:rcpl-conv},
the operator norm $\|\mytensor{M}\|$ is bound by the CP eigenvalues $\lambda$ as
\begin{equation}
\|\mytensor{M}\| \leq \sqrt{H W} \sum_{r = 1}^{R} |\lambda_r|
\max_{f, g} \left| \Tilde{\mytensor{C}}^{(f, g)}_r \right|
\end{equation}
\end{lemma}

\begin{proof} (of Lemma~\ref{lem:rcpl-conv-operator-norm})
The proof is a combination of Lemmas~\ref{lem:cpl-conv-operator-norm} and~\ref{lem:rcpl-fc-operator-norm}. Let $\Tilde{\mytensor{M}} = \ourfft{1, 2}{m}(\mytensor{M})$, we have
\begin{gather}
\|\mytensor{M}\| = \|\mytensor{\Tilde{M}}\| = \sqrt{HW} \max_{f, g} \|\mytensor{\Tilde{W}}^{(f, g)}\| \\
\mytensor{\Tilde{M}}^{(f, g)} = \sum_{r = 1}^{R} \lambda_r \mytensor{\Tilde{C}}^{(f, g)}_r
\mytensor{K}^{1}_{r, t_1, s_1} \cdots \mytensor{K}^{m}_{r, t_m, s_m}
\end{gather}
The operator norm is bounded using Lemma~\ref{lem:rcpl-fc-operator-norm}:
$\|\mytensor{\Tilde{M}}^{(f, g)}\| \leq \sum_{r = 1}^{R} |\lambda_r| \max_{f, g} \left| \Tilde{\mytensor{C}}^{(f, g)}_r \right|$.
\end{proof}
\section{Additional Algorithms and Algorithmic Details}
\label{app:algos}
\textbf{Details of Step 3 in Algorithm~\ref{alg:cnn_compress}.} We use the alternating least squares (ALS) in the implementation of step 3, which is the `parafac’ method of the tensorly library~\citep{kossaifi2019tensorly}, to obtain the CP decomposition. Though CP decomposition is in general NP-hard, the ALS method usually converges for most tensors, with polynomial convergence rate w.r.t. the given precision of the allowed reconstruction error~\citep{anandkumar2014provable, anandkumar2014guaranteed, anandkumar2014tensor}. In addition, step 3 obtains a CP parametrization of the weight tensor rather than recovers the true components of the weight tensor’s CP decomposition. The rank in the CP decomposition is selected in step 2 and is an upper bound of the true rank of the tensor (Proposition~\ref{assum:CP-cnn}). Thus, with the chosen rank, we can obtain a CP decomposition with a very low reconstruction error. In practice, for our cases, the CP decomposition method (ALS) used in step 3 always converges within a few iterations, with reasonable run time.
\input{./algos/algo_find_best_rank_cnn.tex}
\input{./algos/algo_find_best_rank.tex}
\input{./algos/algo_cnn_proj.tex}
\input{./algos/algo_tnn_proj.tex}

\section{Convolutional Neural Networks: Compressibility and Generalization}
\label{app:cnn_proofs}

\subsection{Complete Proofs of Convolutional Neural Networks}
\label{sub:cnn_perturbation}

\begin{definition} \label{def:cnn_tf_full}
[\tflong $\tf^{(k)}_j$]
The {\em {\tflong}s} $\left\{ \tf^{(k)}_j \right\}_{j=1}^{R^{(k)}}$ of the $k^{{\mbox{\tiny th}}}$ layer is defined as
\begin{equation}
\tf^{(k)}_j := \sum_{r = 1}^{j} \left| \lambda^{(k)}_{r} \right| \max_{f, g} \left| \Tilde{C}^{(f, g)}_r \right|
\label{eq:cnn_tf_full}
\end{equation}
where $\lambda_r^{(k)}$ is the $r^{{\mbox{\tiny th}}}$ largest value in the CP spectrum of $\mytensor{M}^{(k)}$.
\end{definition}

\begin{definition} \label{def:cnn_tnb_full}
[\nblong $\nb^{(k)}_j$]
The {\em \nblong} $\left\{ \nb^{(k)}_j \right\}_{j=1}^{R^{(k)}}$ of the $k^{{\mbox{\tiny th}}}$ layer
measures the amplitudes of the remaining components after pruning the ones
with amplitudes smaller than the $\lambda^{(k)}_j$:
\begin{equation}
\nb^{(k)}_j := \sum_{r=j+1}^{R^{(k)}} \left| \lambda_{r}^{(k)} \right| \max_{f, g} \left| \Tilde{C}^{(f, g)}_r \right|
\label{eq:cnn_tnb_full}
\end{equation}
\end{definition}

\begin{definition} \label{def:cnn_lc_full}
[layer cushion $\lc^{(k)}$]
As introduced in~\cite{arora2018stronger}, the layer cushion of the $k^{{\mbox{\tiny th}}}$ layer is defined to be the largest value $\lc^{(k)}$ such that for any $\mytensor{X}^{(k)} \in S$,
\begin{equation}
\lc^{(k)} \frac{\fronorm{\mytensor{M}^{(k)}}}{\sqrt{H^{(k)}W^{(k)}}} \fronorm{\mytensor{X}^{(k)}}  \leq \fronorm{ \mytensor{M}^{(k+1)}}
\label{eq:cnn_lc_full}
\end{equation}
Following~\citet{arora2018stronger}, the layer cushion considers
how much smaller the output $\fronorm{\mytensor{M}^{(k+1)} }$ of the $k^{{\mbox{\tiny th}}}$ layer (after activation)
compared with the product between the weight tensor $\fronorm{M^{(k)} }$ and the input $\fronorm{ \mytensor{X}^{(k)}}$. Note that $H^{(k)}$ and $W^{(k)}$ are constants and will not influence the results of the theorem and the lemmas. For simplicity, we use $H$ and $W$ to denote the maximum $H^{(k)}$ and $W^{(k)}$ over the $n$ layers for the following proofs where upper bounds are desired.
\end{definition}

Given these definitions, we can bound the difference of outputs from
a given model and its compressed counterpart. The following lemma characterizes
the relation between the difference and the factors $\tf^{(k)}_j$, $\nb^{(k)}_j$, $\lc^{(k)}$.
\begin{lemma} \label{lem:cnn_compression}
(Compression bound of convolutional neural networks)
Suppose a convolutional neural network $\mathbb{M}$ has $n$ layers,
and each convolutional layer takes a CP format as in Eq.~\eqref{eq:cpl-conv} with rank $R^{(k)}$.
If an algorithm generates a compressed network $\hat{\mathbb{M}}$ such that
only $\hat{R}^{(k)}$ components with largest $\lambda^{(k)}_r$'s are retained at the $k^{{\mbox{\tiny th}}}$ layer,
the difference of their outputs at the $m^{{\mbox{\tiny th}}}$ is bounded by $\mytensor{X}^{(m+1)}$ as
\begin{equation}
\fronorm{\mytensor{X}^{(m)} - \hat{\mytensor{X}}^{(m)})}
\leq \left(
\sum_{k=1}^{m-1} \frac{\nb^{(k)}}{\lc^{(k)} \fronorm{\mytensor{M}^{(k)}}}
\prod_{l=k+1}^{m-1}\frac{\tf^{(l)}}{\lc^{(l)} \fronorm{\mytensor{M}^{(l)}}}
\right)
\fronorm{\mytensor{X}^{(m)})}
\label{eq:cnn_compression_layer}
\end{equation}
Therefore for the whole network with $n$ layers,
the difference between $\mathbb{M}(\mytensor{X})$ and $\hat{\mathbb{M}}(\mytensor{X})$ is bounded by
\begin{equation}
\fronorm{\mathbb{M}(\mytensor{X}) - \hat{\mathbb{M}}(\mytensor{X})}
\leq \left(
\sum_{k=1}^{n} \frac{\nb^{(k)}}{\lc^{(k)} \fronorm{\mytensor{M}^{(k)}}}
\prod_{l=k+1}^{n}\frac{\tf^{(l)}}{\lc^{(l)} \fronorm{\mytensor{M}^{(l)}}}
\right)
\fronorm{\mathbb{M}(\mytensor{X})}
\label{eq:cnn_compression}
\end{equation}
\end{lemma}

\begin{proof} (of Lemma~\ref{lem:cnn_compression})
We prove this lemma by induction. For $m = 2$, the lemma holds since
\begin{align}
\fronorm{\mytensor{X}^{(2)} - \hat{\mytensor{X}}^{(2)}}
& = \fronorm{\relu{\mytensor{Y}^{(1)}} - \relu{\hat{\mytensor{Y}}^{(1)}}} \\
& \leq \fronorm{\mytensor{Y}^{(1)} - \hat{\mytensor{Y}}^{(1)}}
= \fronorm{\left(\mytensor{M}^{(1)} - \hat{\mytensor{M}}^{(1)} \right) \left(\mytensor{X}^{(1)} \right)} \\
& \leq \sqrt{HW} \nb^{(1)} \fronorm{\mytensor{X}^{(1)}}
\leq \frac{\nb^{(1)}}{\lc^{(1)}\fronorm{\mytensor{M}^{(1)}}} \fronorm{\mytensor{M}^{(2)}}
\end{align}
where $\mytensor{Y} = \mytensor{M}(\mytensor{X})$ denotes the computation of a convolutional layer.
(1) The first inequality follows the Lipschitzness of the ReLU activations;
(2) The second inequality uses Lemma~\ref{lem:cpl-conv-operator-norm};
and (3) the last inequality holds by the definition of $\lc^{(1)}$.
For $m + 1> 2$, we assume the lemma already holds for $m$
\begin{align}
& \quad \fronorm{\mytensor{X}^{(m+1)} - \hat{\mytensor{X}}^{(m+1)}}
= \fronorm{\relu{\mytensor{Y}^{(m)}} - \relu{\hat{\mytensor{Y}}^{(m)}}} \\
& \leq \fronorm{\mytensor{Y}^{(m)} - \hat{\mytensor{Y}}^{(m)}}
= \fronorm{\mytensor{M}^{(m)}\left(\mytensor{X}^{(m)}\right) -
\hat{\mytensor{M}}^{(m)}\left(\hat{\mytensor{X}}^{(m)}\right)} \\
& = \fronorm{\hat{\mytensor{M}^{(m)}}\left(\mytensor{X}^{(m)} - \hat{\mytensor{X}}^{(m)} \right) +
\left(\mytensor{M}^{(m)} - \hat{\mytensor{M}}^{(m)}\right) \left(\mytensor{X}^{(m)}\right)} \\
& \leq \sqrt{HW} \left( \tf^{(m)} \fronorm{\mytensor{X}^{(m)} - \hat{\mytensor{X}}^{(m)}} + \nb^{(m)} \fronorm{\mytensor{X}^{(m)}} \right) \\
& \leq \tf^{(m)} \left( \sum_{k=1}^{m-1} \frac{\nb^{(k)}}{\lc^{(k)} \fronorm{\mytensor{M}^{(k)}}}
\prod_{l=k+1}^{m-1}\frac{\tf^{(l)}}{\lc^{(l)} \fronorm{\mytensor{M}^{(l)}}} \right)
\fronorm{\mytensor{X}^{(m)}} + \frac{\nb^{(m)}}{\lc^{(m)}\fronorm{\mytensor{M}^{(m)}}} \fronorm{\mytensor{X}^{(m)}} \\
& \leq \left( \sum_{k=1}^{m} \frac{\nb^{(k)}}{\lc^{(k)} \fronorm{\mytensor{M}^{(k)}}}
\prod_{l=k+1}^{m}\frac{\tf^{(l)}}{\lc^{(l)} \fronorm{\mytensor{M}^{(l)}}} \right) \fronorm{\mytensor{M}^{(m+1)}}
\end{align}
which completes the induction.
\end{proof}

\begin{lemma} \label{lem:cnn_lemma}
For any convolutional neural network $\mathbb{M}$ of $n$ layers satisfying the assumptions in section~\ref{sec:setup} and any error $0\leq\epsilon\leq 1$, Algorithm~\ref{alg:cnn_compress} generates a compressed tensorial neural network $\hat{\mytensor{\mathbb{M}}}$ such that for any $\mytensor{X}\in S$:
\begin{equation}
\fronorm{\mathbb{M}(\mytensor{X}) -  \hat{\mathbb{M}}(\mytensor{X})} \leq \epsilon \fronorm{\mathbb{M}(\mytensor{X})}
\end{equation}
The compressed convolutional neural network $\hat{\mathbb{M}}$ has $\sum_{k=1}^n\hat{R}^{(k)}(s^{(k)} + o^{(k)}+ k_x^{(k)}k_y^{(k)}+1)$ total parameters, where each $\hat{R}^{(k)}$ satisfies:
\begin{equation}
\hat{R}^{(k)} = \min \Big\{ j \in [R^{(k)}] | \nb^{(k)}_j \Pi_{i=k+1}^{n} \tf^{(i)}_j \leq \frac{\epsilon}{n} \Pi_{i=k}^n \lc^{(i)} \fronorm{\mytensor{M}^{(i)}} \Big\}
\label{eq:lemma_eq}
\end{equation}
\end{lemma}
\textit{Remark.} Equation~\eqref{eq:lemma_eq} is slightly different with equation~\ref{eq:main_eq}, as the margin $\gamma$ is replaced by a perturbation error $\epsilon$. Therefore, how well the compressed tensorial neural network can approximate the original network is related to the choice of $\hat{R}^{(k)}$.
Notice that when $\hat{R^{(k)}} = R^{(k)}$, the inequality for the $k^{{\mbox{\tiny th}}}$ layer will be automatically satisfied as $\cfr^{(k)} = 0$ in this case by definition.

\begin{proof} (of Lemma~\ref{lem:cnn_lemma})
The proof is trivial by observing
\begin{align}
& \frac{\nb^{(k)}}{\lc^{(k)} \fronorm{\mytensor{M}^{(k)}}} \leq \frac{\epsilon}{n}
\prod_{i=k+1}^{n} \frac{\lc^{(i)} \fronorm{\mytensor{M}^{(i)}}}{\tf^{(i)}} \\
\implies ~ & \frac{\nb^{(k)}}{\lc^{(k)} \fronorm{\mytensor{M}^{(k)}}}
\prod_{i=k+1}^{n}\frac{\tf^{(i)}}{\lc^{(i)} \fronorm{\mytensor{M}^{(i)}}} \leq \frac{\epsilon}{n} \\
\implies ~ & \sum_{k=1}^{n} \frac{\nb^{(k)}}{\lc^{(k)} \fronorm{\mytensor{M}^{(k)}}}
\prod_{i=k+1}^{n} \frac{\tf^{(i)}}{\lc^{(l)} \fronorm{\mytensor{M}^{(i)}}} \leq \epsilon
\end{align}
\end{proof}

Before proving Theorem~\ref{thm:cnn_thm}, Lemma~\ref{lem:cnn_helper_lemma_3} (introduced below) is needed.
\begin{lemma}
\label{lem:cnn_helper_lemma_3}
For any convolutional neural network $\mathbb{M}$ of $n$ layers satisfying the assumptions in section~\ref{sec:setup} and any margin $\gamma \geq 0$, $\mathbb{M}$ can be compressed to a tensorial convolutional neural network $\hat{\mathbb{M}}$ with $\sum_{k=1}^{n} \hat{R}^{(k)}(s^{(k)}+t^{(k)}+k_x^{(k)} \times k_y^{(k)}+1)$ total parameters such that for any $\mytensor{X} \in S$, $\hat{L_0}(\hat{\mathbb{M}}) \leq \hat{L}_\gamma(\mathbb{M})$.
Here, for each layer $k$,
\begin{equation}
\hat{R}^{(k)} = \min \Big\{ j \in [R^{(k)}] | \nb^{(k)}_j \Pi_{i=k+1}^{n} \tf^{(i)}_j \leq \frac{\epsilon}{n} \Pi_{i=k}^n \lc^{(i)} \fronorm{\mytensor{M}^{(i)}} \Big\}
\end{equation}
\end{lemma}

\input{./proofs/cnn_helper_lemma_3.tex}

Now we prove the main theorem~\ref{thm:cnn_thm} by bounding the covering number given any $\epsilon$.
\input{./proofs/cnn_thm.tex}

\section{Fully Connected Networks: Compressibility and Generalization}
In this section, we derive generalization bounds for fully connected (FC) neural networks (denoted as $\mathbb{M}$) using tensor methods.

\subsection{Compression of a FC Network with {CPL}\xspace}
\label{sec:fcnn}

\textbf{Original Fully Connected Neural Network:} Let $\mathbb{M}$ denote an \textbf{$n$}-layer fully connected network with \reluname activations, where $\mymatrix{A}^{(k)} \in \mathbb{R}^{h^{(k)} \times h^{(k+1)}}$ denotes the weight matrix of the $k^{{\mbox{\tiny th}}}$ layer, $\myvector{x}^{(k)} \in \mathbb{R}^{h^{(k)}}$ denotes the input to $k^{{\mbox{\tiny th}}}$ layer, and $\myvector{y}^{(k)}$ denotes the output of the $k^{{\mbox{\tiny th}}}$ layer before activation in $\mathbb{M}$.
\textbf{Transform original FCN to a CP-FCN:} We transform the original fully connected network $\mathbb{M}$ to a network $\mathbb{M}$ with {CPL}\xspace. The $k^{{\mbox{\tiny th}}}$ layer of $\mathbb{M}$ is denoted by $\mytensor{M}^{(k)}\in \mathbb{R}^{s^{(k)}_1 \times s^{(k)}_2 \times s^{(k+1)}_1 \times s^{(k+1)}_2}$ is a 4-dimensional tensor reshaped from $\mymatrix{A}^{(k)}$ where $s^{(k)}_1 \times s^{(k)}_2 = h_{k}, \forall k \in [n]$.

\textbf{Input and Output of $\mathbb{M}$:}
The original input and output vectors of $\mathbb{M}$ are reshaped into matrices.
The input to the $k^{{\mbox{\tiny th}}}$ layer of the $\mathbb{M}$, denoted by $\mymatrix{X}^{(k)} \in \mathbb{R}^{s^{(k)}_1 \times s^{(k)}_2}$, is a matrix reshaped from the input vector $\myvector{x}^{(k)}$ of the $k^{{\mbox{\tiny th}}}$ layer in the original network $\mathbb{M}$.
Similarly, the output of the $k^{{\mbox{\tiny th}}}$ layer  before activation in $\mathbb{M}$, denoted by $\mymatrix{Y}^{(k)} \in \mathbb{R}^{s^{(k)}_1 \times s^{(k)}_2}$,  is a matrix reshaped from the output vector $\myvector{y}^{(k)}$ of the $k^{{\mbox{\tiny th}}}$ layer in the original network $\mathbb{M}$.
For prediction purposes, we reshape the output $\mymatrix{Y}^{(n)}$ of the last layer in $\mathbb{M}$ back into a vector. So the final outputs of $\mathbb{M}$ and $\mathbb{M}$ are of the same dimension.

\begin{assumption}[\textbf{Polyadic Form of $\mathbb{M}$}]\label{assum:CP-tnn}
For each layer $k$, assume the weight tensor $\mytensor{\mytensor{M}}^{(k)}$ of $\mathbb{M}$ has a Polyadic form with rank $R^{(k)} \leq \min\{s_1^{(k)}, s_2^{(k)}, s_1^{(k+1)}, s_2^{(k+1)} \}$:
\begin{equation}\label{eq:cpform-tnn-fc}
\mytensor{M}^{(k)} = \sum_{i = 1}^{R^{(k)}} \lambda_{i}^{(k)} a_{i}^{(k)} \otimes b_{i}^{(k)} \otimes c_{i}^{(k)} \otimes d_{i}^{(k)}
\end{equation}
where $\forall i, a_i, b_i, c_i, d_i$ are unit vectors in $\mathbb{R}^{s_1^{(k)}},  \mathbb{R}^{s_2^{(k)}}, \mathbb{R}^{s_1^{(k+1)}}, \mathbb{R}^{s_2^{(k+1)}}$ respectively, and $\forall 1 \leq i \leq R^{(k)}, \inp{a_i}{a_i} = 1,  \inp{b_i}{b_i} = 1,  \inp{c_i}{c_i} = 1,  \inp{d_i}{d_i} = 1 $.
Moreover, for each $\mytensor{M}^{(k)}$, $\lambda_{i}^{(k)} \geq \lambda_{i+1}^{(k)},  \forall i$, and the absolute value of the smallest $| \lambda_{R^{(k)}}^{(k)} |$ can be arbitrarily small.
\end{assumption}

The total number of parameters in $\mathbb{M}$ is $(s_1^{(k)} + s_2^{(k)} + s_1^{(k+1)} + s_2^{(k+1)}+1) R^{(k)}$ and a
smaller $R^{(k)}$ renders fewer number of parameters and thus leads to compression.
We introduce a compression mechanism that prunes out the smaller components of weight tensor of $\mathbb{M}$, i.e., a low rank approximation of each weight tensor $\mytensor{M}^{(k)}$ of the $k^{{\mbox{\tiny th}}}$ layer, and generates a compressed CP-FCN $\hat{\mathbb{M}}$. The algorithm is depicted in Algorithm~\ref{alg:tfc_compress}.

\textbf{Compression of a FC Network with {CPL}\xspace:} In~\cite{li2018guaranteed}, a tensor decomposition algorithm (procedure 1 in~\cite{li2018guaranteed}) on tensors with asymmetric orthogonal components is guaranteed to recover the top-$r$ components with the largest singular values.  To compress $\mathbb{M}$, we apply top-$\hat{R}^{(k)}$ ($\hat{R}^{(k)} \leq R^{(k)}$) CP decomposition algorithm on each $\mytensor{M}^{(k)}$, obtaining the components from CP decomposition $(\hat{\lambda}_{i}^{(k)}, \hat{a}_{i}^{(k)}, \hat{b}_{i}^{(k)}, \hat{c}_{i}^{(k)}, \hat{d}_{i}^{(k)}$), $i \in [\hat{R}^{(k)}]$.
Therefore, we achieve \textbf{a compressed network $\hat{\mathbb{M}}$} of $\mathbb{M}$, and the $j^{{\mbox{\tiny th}}}$ layer of the compressed network  $\hat{\mathbb{M}}$ has weight tensor as follows
\begin{equation}\label{eq:compressedtnn_fc}
\mytensor{\hat{T}}^{(k)} = \sum_{i = 1}^{\hat{R}^{(k)}} \hat{\lambda}_{i}^{(k)} \hat{a}_{i}^{(k)} \otimes \hat{b}_{i}^{(k)}\otimes \hat{c}_{i}^{(k)} \otimes \hat{d}_{i}^{(k)}.
\end{equation}
As each $\mytensor{M}^{(k)}$ has a low rank orthogonal CP decomposition by our assumption, the returned results $\{\hat{\lambda}_{i}^{(k)}, \hat{a}_{i}^{(k)}, \hat{b}_{i}^{(k)}, \hat{c}_{i}^{(k)}, \hat{d}_{i}^{(k)}\}_{i=1}^{\hat{R}^{(k)}}$ from procedure 1 in~\cite{li2018guaranteed} are perfect recoveries of $\{\lambda_{i}^{(k)}, a_{i}^{(k)}, b_{i}^{(k)}, c_{i}^{(k)}, d_{i}^{(k)}\}_{i=1}^{\hat{R}^{(k)}}$
according to the robustness theorem in~\cite{li2018guaranteed}.  Our compression procedure is depicted in Algorithm~\ref{alg:tfc_compress}.

\input{algo_compressFCN}
We denote  the input matrix of the $k^{{\mbox{\tiny th}}}$ layer in $\hat{\mathbb{M}}$ as $\hat{\mymatrix{X}}^{(k)}$, and the output matrix before activation as $\hat{\mymatrix{Y}}^{(k)}$. Note that $\mymatrix{X}^{(1)} = \hat{\mymatrix{X}}^{(1)}$ as the input data is not being modified.

Algorithm~\ref{alg:tfc_compress} is desigend for general neural networks. For neural networks with ${CP Layer}\xspace$, line 3 can be done by pruning out small components from CP decomposition, and only keeping top-$\hat{R}^{(k)}$ components.
For notation simplicity, assume for each layer in $\mathbb{M}$, the width of the $k^{{\mbox{\tiny th}}}$ layer is a square of some integer $s^{(k)}$.
Then the input to the $k^{{\mbox{\tiny th}}}$ layer of $\mathbb{M}$ is a ReLu transformation of the output of the $k-1^{{\mbox{\tiny th}}}$ layer as in equation~\eqref{eq:tnn_in_out}. The output of the $k^{{\mbox{\tiny th}}}$ layer of $\mathbb{M}$ is illustrated in equation~\eqref{eq:tnn_out} as the weight tensor which permits a CP forms as in equation~\eqref{eq:cpform-tnn-fc}.
\begin{align}
\mymatrix{X}^{(k)} & = \relu{\mymatrix{Y}^{(k-1)}} \label{eq:tnn_in_out}\\
\mymatrix{Y}^{(k)} &=  \sum_{i = 1}^{\hat{R}^{(k)}} \lambda_{i}^{(k)} {a_{i}^{(k)}}^\top \mymatrix{X}^{(k)} b_{i}^{(k)} c_{i}^{(k)} \otimes d_{i}^{(k)} + \mytensor{\phi}^{(k)}(\mymatrix{X}^{(k)}) \label{eq:tnn_out}
\end{align}

where $\mytensor{\phi}^{(k)} = \sum_{i = \hat{R}^{(k)} + 1}^{R^{(k)}} \lambda_{i}^{(k)} a_{i}^{(k)} \otimes b_{i}^{(k)} \otimes c_{i}^{(k)} \otimes d_{i}^{(k)}$, $\mytensor{\phi}^{(k)}(\mymatrix{X}^{(k)})$ denotes the multilinear operation of the tensor $\mytensor{\phi}^{(k)}$ on $\mymatrix{X}^{(k)}$, i.e., $\{ \mytensor{\phi}^{(k)}(\mymatrix{X}^{(k)}) \}_{i,j} = \sum_{k, l} \mytensor{\phi}^{(k)}_{i,j,k,l} \mymatrix{X}^{(k)}_{k,l}$ and $a_{i}^{(k)}, b_{i}^{(k)}, \hat{a}_{i}^{(k)}, \hat{b}_{i}^{(k)} \in \mathbb{R}^{s_k}$.
Similarly, the input and output of the $k^{{\mbox{\tiny th}}}$ layer of the compressed neural nets $\hat{\mathbb{M}}$ satisfy
\begin{align}
\hat{\mymatrix{X}}^{(k)}& = \relu{\hat{\mymatrix{Y}}^{(k-1)}}\\
\hat{\mymatrix{Y}}^{(k)} &= \sum_{i = 1}^{\hat{R}^{(k)}} \hat{\lambda}_{i}^{(k)} (\hat{a}_{i}^{(k)})^\top \hat{\mymatrix{X}}^{(k)} \hat{b}_{i}^{(k)} \hat{c}_{i}^{(k)} \otimes \hat{d}_{i}^{(k)}.
\end{align}

\subsection{Characterizing Compressibility of FC Networks with CPL}\label{sec:properties-tnn}
Now we characterize the compressibility of the fully connected network with {CPL}\xspace $\mathbb{M}$ through properties defined in the following, namely reshaping factor, tensorization factor, layer cushion and tensor noise bound.

\input{fc_notations.tex}
\label{sec:notations_fc}

\subsection{Generalization Guarantee of Fully Connected Neural Networks}
We have introduced the compression mechanism in Algorithm~\ref{alg:tfc_compress}. For a fully connected network with {CPL}\xspace $\mathbb{M}$ that is characterized by the properties such as reshaping factor, tensorization factor, layer cushion and tensor noise bound, in section~\ref{sec:properties-tnn}, we derive the generalization error bound of a compression network with any chosen ranks $\{\hat{R}^{(k)}\}_{k=1}^{n}$ as follows.

\begin{theorem}
\label{thm:main_thm}
For any fully connected network $\mathbb{M}$ of $n$ layers satisfying the Assumptions~\ref{assum:CP-tnn}, Algorithm~\ref{alg:tfc_compress} generates a compressed network $\hat{\mathbb{M}}$ such that with high probability over the training set , the expected error $L_0(\hat{\mathbb{M}})$ is bounded by
\begin{equation}
L_0(\hat{\mathbb{M}}) \le \hat{L}_{\gamma}(\mathbb{M}) + \tilde{O}\Big(\sqrt{\frac{ \sum_{k=1}^{n} \hat{R}^{(k)}(2 s^{(k)} + 2 s^{(k+1)} + 1)}{m}}\Big)
\end{equation}
for any margin $\gamma \geq 0$,
and the rank of the $k^{{\mbox{\tiny th}}}$ layer, $\hat{R}^{(k)}$,  satisfies that
\begin{equation}
\label{eq:rank-choice}
\hat{R}^{(k)} = \min \Big\{ j \in [R^{(k)}] \ \Big |\  n\rf^{(k)} \nb^{(k)}_j \Pi_{i=k+1}^{n} \tf^{(i)} \nonumber \leq \frac{\gamma}{2 \max_{\myvector{x}  \in S } \fronorm{\mathbb{M}(\myvector{x})}} \fronorm{\mymatrix{A}^{(k)}} \Pi_{i=k}^n \lc^{(i)} \Big\}
\end{equation}
and $\rf^{(k)}, \tf^{(k)}, \lc^{(k)}$ are reshaping factor, tensorization factor, layer cushion and tensor noise bound of the $k^{{\mbox{\tiny th}}}$ layer in Definitions~\ref{def:rf}, ~\ref{def:tf}, and ~\ref{def:lc} respectively. $\nb^{(k)}_j$ is defined in the same way with $\nb^{(k)}$, where $\hat{R}^{(k)}$ is replaced by $j$.
\end{theorem}

The generalization error of the compressed network $L_0(\hat{\mathbb{M}})$ depends on the compressibility of the $\mathbb{M}$.  The compressibility of the $\mathbb{M}$ determines the rank that the compression mechanism should select according to Theorem~\ref{thm:main_thm}, which depends on reshaping factor $\rf^{(k)}$, tensorization factor $\tf^{(k)}$, layer cushion $\lc^{(k)}$ and tensor noise bound  $\nb^{(k)}_j$.

\paragraph{Proof sketch of Theorem~\ref{thm:main_thm}:} To prove this theorem, we introduce the following Lemma~\ref{lem:main_lemma}, which reveals that the difference between the output of the original fully connected network  $\mathbb{M}$ and that of the compressed $\mathbb{M}$ is bounded by $\epsilon \fronorm{\mathbb{M}(\myvector{x})}$. Then we show the covering number of the compressed network $\mathbb{M}$ by approximating each parameter with some certain accuracy is $\tilde{O}(d)$ w.r.t to a given $\epsilon$. After bounding the covering number, the rest of the proof follows from conventional learning theory.

\begin{lemma}
\label{lem:main_lemma}
For any fully connected network $\mathbb{M}$ of $n$ layers satisfying Assumption~\ref{assum:CP-tnn} , Algorithm~\ref{alg:tfc_compress} generates a compressed ${Tensorial-FC}\xspace$ $\hat{\mathbb{M}}$ where for any $\myvector{x} \in S$ and any error $0 \leq \epsilon \leq 1$:
\begin{equation}
\fronorm{\mathbb{M}(\myvector{x}) -  \hat{\mathbb{M}}(\mymatrix{X})} \leq \epsilon \fronorm{\mathbb{M}(\myvector{x})}
\end{equation}
The compressed ${Tensorial-FC}\xspace$ $\hat{\mathbb{M}}$ consists of $\sum_{k=1}^{n} \hat{R}^{(k)}[2(s^{(k)} + s^{(k+1)}) + 1)]$ number of  parameters, where each $\hat{R}^{(k)}$ is defined as what is stated in Algorithm~\ref{alg:find_best_rank}: for each layer $k \in [n]$,
\begin{align*}
\hat{R}^{(k)} &= \min \Big\{ j \in [R^{(k)}] \ \Big |\  \rf^{(k)} \nb^{(k)}_j \Pi_{i=k+1}^{n}  \tf^{(i)} \leq \frac{\epsilon}{n} \fronorm{\mymatrix{A}^{(k)}} \Pi_{i=k}^n \lc^{(i)} \Big\}
\end{align*}
\end{lemma}
The complete proofs are in \ref{app:proofs}

\subsection{Complete Proofs of Fully Connected Neural Networks}
\label{app:proofs}

To prove Lemma~\ref{lem:main_lemma}, Lemma~\ref{lem:helper_lemma} (introduced below) is needed.
\begin{lemma}
\label{lem:helper_lemma}
For any fully connected network $\mathbb{M}$ of $n$ layers satisfying the assumptions in section~\ref{sec:setup}, given a list of ranks $\{\hat{R}^{(k)}\}_{i=1}^n (\forall k, \hat{R}^{(k)} \leq R^{(k)}$), after tensorizing each layer in $\mathbb{M}$ and making $\mathbb{M}$ into $\mathbb{M}$, Algorithm~\ref{alg:tfc_project} generates a compressed tensorial neural network $\hat{\mathbb{M}}$ with $\sum_{k=1}^{n} r^{(k)}[2(s^{(k)} + s^{(k+1)}) + 1)]$ total parameters where for any $\myvector{x} \in S$:
$$\fronorm{\mathbb{M}(\myvector{x}) -  \hat{\mathbb{M}}(\mymatrix{X})} \leq (\sum_{k=1}^n \frac{\rf^{(k)} \nb^{(k)} }{ \lc^{(k)} \fronorm{\mymatrix{A}^{(k)} }} \cdot \Pi_{i=k+1}^{n} \frac{\tf^{(i)} }{ \lc^{(i)}} ) \fronorm{\mathbb{M}(\myvector{x})}$$
where $\mymatrix{X}$ is the matricized version of $\myvector{x}$, and $\rf^{(k)}, \tf^{(k)}, \lc^{(k)}, \nb^{(k)}$ are reshaping factor, tensorization factor, layer cushion, and tensor noise bound of the $k^{{\mbox{\tiny th}}}$ layer in Definitions~\ref{def:rf}, ~\ref{def:tf}, ~\ref{def:lc}, and~\ref{def:t_noise_bound} respectively.
\end{lemma}
\input{./proofs/helper_lemma.tex}
Then we can proceed to prove Lemma~\ref{lem:main_lemma}:
\input{./proofs/main_lemma.tex}
Before proving Theorem~\ref{thm:main_thm}, Lemma~\ref{lem:helper_lemma_2} (introduced below) is needed.
\begin{lemma}
\label{lem:helper_lemma_2}
For any fully connected network $\mathbb{M}$ of $n$ layers satisfying the assumptions in section~\ref{sec:setup} and any margin $\gamma \geq 0$, $\mathbb{M}$ can be compressed to a fully-connected tensorial neural network $\hat{\mathbb{M}}$ with $\sum_{k=1}^{n} \hat{R}^{(k)}[2(s^{(k)} + s^{(k+1)}) + 1)]$ total parameters such that for any $\myvector{x} \in S$, $\hat{L_0}(\hat{\mathbb{M}}) \leq \hat{L}_\gamma(\mathbb{M})$.
Here, for each layer $k$,
\begin{align*}
\hat{R}^{(k)} &= \min \Big\{ j \in [R^{(k)}] \ \Big |\  \rf^{(k)} \nb^{(k)}_j \Pi_{i=k+1}^{n} \tf^{(i)}\leq \frac{\epsilon}{n} \fronorm{\mymatrix{A}^{(k)}} \Pi_{i=k}^n \lc^{(i)} \Big\}
\end{align*}
\end{lemma}
\input{./proofs/helper_lemma_2.tex}
Now we prove the main theorem~\ref{thm:main_thm} by bounding the covering number given any $\epsilon$.
\input{./proofs/main_thm.tex}

\section{Neural Networks with Skip Connections}
\label{sec:sc}
\subsection{Problem Setup}
\label{subsec:sc_notations}
For neural nets with skip connections, the current theoretical analyses consider convolutional neural networks with one skip connection used on each layer, since our theoretical results can easily extend to general neural nets with skip connections. Therefore, we used the same the notations for neural nets with skip connections as what we defined for convolutional neural networks.

\textbf{Forward pass functions}
Under the above assumptions, the only difference that we need to take into account between our analysis of CNN with skip connections and our analysis of standard CNN is the forward pass functions. In neural networks with skip connections, we have
$$\mytensor{X}^{(k)}=\relu{\mytensor{Y}^{(k-1)}}$$
$$\mytensor{Y}^{(k)}= \mytensor{M}^{(k)} \left(\mytensor{X}^{(k)}\right) +\mytensor{X}^{(k)}$$
and
$$\hat{\mytensor{X}}^{(k)}=\relu{\hat{\mytensor{Y}}^{(k-1)}}$$
$$\hat{\mytensor{Y}}^{(k)}=\hat{\mytensor{M}}^{(k)}\left(\hat{\mytensor{X}}^{(k)}\right)+\hat{\mytensor{X}}^{(k)}$$
where $\mytensor{M}^{(k)}\left(\mytensor{X}^{(k)}\right) $ and $\hat{\mytensor{M}}^{(k)}\left(\hat{\mytensor{X}}^{(k)}\right)$
compute the outputs of the $k^{{\mbox{\tiny th}}}$ convolutional layer.

Similarly, we use \textit{tensorization factor}, \textit{tensor noise bound} and \textit{layer cushion} as in convolutional neural network defined in \ref{def:cnn_tf}, \ref{def:cnn_tnb} and \ref{def:cnn_lc}. But note that the input $\mytensor{X}^{(k)}$ in the definition of \textit{layer cushion} is the input of $k^{{\mbox{\tiny th}}}$ layer after skip connection.

\subsection{Generalization Guarantee of Compressed Network Proposed}

\begin{theorem}
\label{thm:sc_thm}
For any convolutional neural network $\mathbb{M}$ of $n$ layers with skip connection satisfying the assumptions in section~\ref{sec:setup} and any margin $\gamma \geq 0$, Algorithm~\ref{alg:cnn_compress} generates a compressed network $\hat{\mathbb{M}}$ such that with high probability over the training set, the expected error $L_0(\hat{\mathbb{M}})$ is bounded by
\begin{equation}
\hat{L}_{\gamma}(\mathbb{M}) + \tilde{O}\Big(\sqrt{\frac{ \sum_{k=1}^{n} \hat{R}^{(k)}(s^{(k)} + o^{(k)} + k_x^{(k)} \times k_y^{(k)} + 1)}{m}}\Big)
\end{equation}
where
\begin{equation}
\begin{aligned}
\hat{R}^{(k)} = \min \Big\{ j \in [R^{(k)}] | \nb^{(k)}_j \Pi_{i=k+1}^{n} (\tf^{(i)}_j + 1) \leq \frac{\gamma}{2 n \max_{\mytensor{X}  \in S } \fronorm{\mathbb{M}(\mytensor{X})}} \Pi_{i=k}^n \lc^{(i)} \fronorm{\mytensor{M}^{(i)}} \Big\}
\end{aligned}
\end{equation}
\end{theorem}

To prove this theorem, Lemma~\ref{lem:sc_lemma} is needed.

\begin{lemma}
\label{lem:sc_lemma}
For any convolutional neural network $\mathbb{M}$ of $n$ layers with skip connection satisfying the assumptions in section~\ref{sec:setup} and any error $0\leq\epsilon\leq 1$, Algorithm~\ref{alg:cnn_compress} generates a compressed tensorial neural network $\hat{\mytensor{\mathbb{M}}}$ such that for any $X\in S$:
\begin{equation}
\fronorm{\mathbb{M}(\mytensor{X}) -  \hat{\mathbb{M}}(\mytensor{X})} \leq \epsilon \fronorm{\mathbb{M}(\mytensor{X})}
\end{equation}
The compressed convolutional neural network $\hat{\mathbb{M}}$ has with $\sum_{k=1}^n\hat{R}^{(k)}(s^{(k)}+o^{(k)}+k_x^{(k)} \times k_y^{(k)}+1)$ total parameters, where each $\hat{R}^{(k)}$ satisfies:
\begin{equation}
\begin{aligned}
\hat{R}^{(k)} = \min \Big\{ j \in [R^{(k)}] | \nb^{(k)}_j \Pi_{i=k+1}^{n} (\tf^{(i)}_j + 1) \leq \frac{\epsilon}{n} \Pi_{i=k}^n \lc^{(i)} \fronorm{\mytensor{M}^{(i)}} \Big\}
\end{aligned}
\end{equation}

\end{lemma}

\subsection{Complete Proofs of Neural Networks with Skip Connection}
\label{app:sc_proofs}
To prove Lemma~\ref{lem:sc_lemma}, the following Lemma~\ref{lem:sc_helper_lemma} is needed.
\begin{lemma}
\label{lem:sc_helper_lemma}
For any convolutional neural network $\mathbb{M}$ of $n$ layers with skip connection satisfying the assumptions in section~\ref{sec:setup}, Algorithm~\ref{alg:cnn_project} generates a compressed tensorial neural network $\hat{\mathbb{M}}$ where for any $\mytensor{X} \in S$:
$$
\fronorm{\mathbb{M}(\mytensor{X})-\hat{\mathbb{M}}(\mytensor{X})}
\leq \left( \sum_{k=1}^{n} \frac{\nb^{(k)}}{\lc^{(k)} \fronorm{\mytensor{M}^{(k)}}}
\prod_{l=k+1}^{n}\frac{\tf^{(l)}+1}{\lc^{(l)} \fronorm{\mytensor{M}^{(l)}}} \right) \fronorm{\mathbb{M}(\mytensor{X})}
$$
where $\nb$, $\lc$, and $\tf$ are \textit{tensor noise bound}, \textit{layer cushion}, \textit{tensorization factor} defined in \ref{def:cnn_tnb}, \ref{def:cnn_lc} and \ref{def:cnn_tf} respectively.
\end{lemma}

\input{./proofs/sc_helper_lemma.tex}

Now we can proove Lemma~\ref{lem:sc_lemma}

\input{./proofs/sc_lemma.tex}

To prove Theorem~\ref{thm:sc_thm}, the following lemma is needed.

\begin{lemma}
\label{lem:sc_helper_lemma_2}
For any convolutional neural network $\mathbb{M}$ of $n$ layers with skip connection satisfying the assumptions in section~\ref{sec:setup} and any margin $\gamma \geq 0$, $\mathbb{M}$ can be compressed to a tensorial convolutional neural network $\hat{\mathbb{M}}$ with $\sum_{k=1}^{n} \hat{R}^{(k)}(s^{(k)}+t^{(k)}+k_x^{(k)} \times k_y^{(k)}+1)$ total parameters such that for any $\mytensor{X} \in S$, $\hat{L_0}(\hat{\mathbb{M}}) \leq \hat{L}_\gamma(\mathbb{M})$.
Here, for each layer $k$,
$$
\begin{aligned}
\hat{R}^{(k)} = \min \Big\{ j \in [R^{(k)}] | \nb^{(k)}_j \Pi_{i=k+1}^{n} (\tf^{(i)}_j + 1) \leq \frac{\epsilon}{n} \Pi_{i=k}^n \lc^{(i)} \fronorm{\mytensor{M}^{(i)}} \Big\}
\end{aligned}
$$
\end{lemma}

The proof of Lemma~\ref{lem:sc_helper_lemma_2} is the same with Lemma~\ref{lem:helper_lemma_2}. And by setting $\epsilon=\frac{\gamma}{2 \max_{\mytensor{X}\in S}}$, we get the desired expression of $\hat{R}^{(k)}$ in the main theorem.

\input{./proofs/sc_thm.tex}

\section*{Supplementary Material}
Supplementary material for the paper: ``Understanding Generalization in Deep Learning via Tensor Methods''.
This appendix is organized as follows:
\begin{itemize}
    \item Appendix A: Experimental details and additional results
    \item Appendix B: Technical definitions and propositions
    \item Appendix C: Main technical contributions

    \item Appendix D, E, and F: Generalization bounds on three types of neural networks: convolutional neural networks, fully-connected neural networks, and neural networks with residual connections
    \item Appendix G: Additional algorithms and algorithmic details
\end{itemize}
\input{a1_experiments}
\input{a2_definitions}
\input{a3_cp_layers}
\input{a5_cnn}
\input{a6_fcn}
\input{a7_sc}
\newpage
\input{a4_algorithms}

\subsubsection{Covering Number Analysis for Convolutional Neural Network}
\begin{proof} (of Theorem~\ref{thm:cnn_thm})
To be more specific, let us bound the covering number of the compressed network $\hat{\mathbb{M}}$ by approximating each parameter with accuracy $\mu$.

\label{app:cnn_covering_number}
\begin{lemma}
\label{lem:cnn_covering_number}
For any given constant accuracy $\mu$, the covering number of the compressed convolutional network $\hat{\mathbb{M}}$ is of order $\tilde{O}(d)$ where $d$ denotes the total number of parameters in $\hat{\mathbb{M}}$: $d := \sum_{k=1}^{n} \hat{R}^{(k)}(s^{(k)} + o^{(k)} + k_x^{(k)} \times k_y^{(k)} + 1)$.

\end{lemma}
Let $\tilde{\mytensor{M}}$ denote the network after approximating each parameter in $\hat{\mathbb{M}}$ with accuracy $\mu$ (and $\mytensor{\tilde{\mytensor{M}}}^{(k)}$ denote its weight tensor on the $k^{th}$ layer). Based on the given accuracy, we know that
$\forall k, \ $
$\abs{\hat{\lambda}_{r}^{(k)} - \tilde{\lambda}_{r}^{(k)}} \leq \mu$,
$\norm{\hat{\myvector{a}}_{r}^{(k)} - \tilde{\myvector{a}}_{r}^{(k)}} \leq \sqrt{s^{(k)}}\mu$,
$\norm{\hat{\myvector{b}}_{r}^{(k)} - \tilde{\myvector{b}}_{r}^{(k)}} \leq \sqrt{o^{(k)}}\mu$,
$\norm{\hat{\mymatrix{C}}_{r}^{(k)} - \tilde{\mymatrix{C}}_{r}^{(k)}} \leq \sqrt{k_x^{(k)}k_y^{(k)}}\mu$, where $s$, $o$, $k_x$ and $k_y$ are the number of input channels, the number of output channels, the height of the kernel and the width of the kernel, as defined in Section~\ref{sec:setup}.
For simplicity, in this proof, let us just use $\mytensor{X}^{(k)}, \mytensor{Y}^{(k)}, \myvector{a}_{r}^{(k)}, \myvector{b}_{r}^{(k)}, \mymatrix{C}_{r}^{(k)}$ to denote $\hat{\mytensor{X}}^{(k)}, \hat{\mytensor{Y}}^{(k)}, \hat{\myvector{a}}_{r}^{(k)}, \hat{\myvector{b}}_{r}^{(k)}, \hat{\mymatrix{C}}_{r}^{(k)}$. $\mytensor{X}^{(k)} \in \mathbb{R}^{H \times W \times s^{(k)}}$, $\mytensor{Y}^{(k)} \in \mathbb{R}^{H \times W \times o^{(k)}}$.

We have
$$
\ourfft{1,2}{3}(\mytensor{Y}^{(k)})_{fgj} = \sqrt{HW} \sum_i[\ourfft{1,2}{3}(\mytensor{X}^{(k)})_{fgi}\sum_{r=1}^{\hat{R}^{(k)}}\lambda_r^{(k)} a_{ri}^{(k)}b_{rj}^{(k)}\ourfft{}{2}(C_{r}^{(k)})_{fg}]
$$
$$\ourfft{1,2}{3}(\tilde{\mytensor{Y}}^{(k)})_{fgj} = \sqrt{HW} \sum_i[\ourfft{1,2}{3}(\tilde{\mytensor{X}}^{(k)})_{fgi}\sum_{r=1}^{\hat{R}^{(k)}}\tilde{\lambda}_r^{(k)} \tilde{a}_{ri}^{(k)}\tilde{b}_{rj}^{(k)}\ourfft{}{2}(\tilde{C}_{r}^{(k)})_{fg}]$$
where $\sqrt{HW}$ is a normalization factor defined in Lemma~\ref{lem:cnn-mdft}

Let $\epsilon^{(k)} = \fronorm{\tilde{\mymatrix{Y}}^{(k)} - \mymatrix{Y}^{(k)}}$. Then for each $k$, let $\varphi=\sum_{f,g,i,j}\Big(\sum_r^{\hat{R}^{(k)}}\lambda_r^{(k)} a_{ri}^{(k)} b_{rj}^{(k)} \big(\ourfft{}{2}(C_r^{(k)})_{fg}-\ourfft{}{2}(\tilde{C}_r^{(k)})_{fg}\big)\Big)^2$ and $\psi=\sum_{f,g,i,j}\Big( \sum_r^{\hat{R}^{(k)}}\big(\lambda_r^{(k)}a_{ri}^{(k)}b_{rj}^{(k)}-\tilde{\lambda}_r^{(k)}\tilde{a}_{ri}^{(k)}\tilde{b}_{rj}^{(k)}\big) \ourfft{}{2}(\tilde{C}_r^{(k)})_{fg}\Big)^2$. We first bound $\varphi$ and $\psi$ as follows.

\textbf{Bound $\varphi=\sum_{f,g,i,j}\Big(\sum_r^{\hat{R}^{(k)}}\lambda_r^{(k)} a_{ri}^{(k)} b_{rj}^{(k)} \big(\ourfft{}{2}(C_r^{(k)})_{fg}-\ourfft{}{2}(\tilde{C}_r^{(k)})_{fg}\big)\Big)^2$:}  All calculations are based on the $k^{th}$ layer, we remove the layer number $(k)$ for ease of reading. So $a=a^{(k)}$ (the same for $b$, $c$, and $R$). Then

\begin{equation*}
\begin{split}
\MoveEqLeft \sum_{f,g,i,j}\Big(\sum_r^{\hat{R}^{(k)}}\lambda_r^{(k)} a_{ri}^{(k)} b_{rj}^{(k)} \big(\ourfft{}{2}(C_r^{(k)})_{fg}-\ourfft{}{2}(\tilde{C}_r^{(k)})_{fg}\big)\Big)^2 \\
&\leq \sum_{f,g,i,j}\Big( \sum_r^{\hat{R}}(\lambda_r a_{ri} b_{rj})^2 \sum_r^{\hat{R}}\big( \ourfft{}{2}(C_r)_{fg}-\ourfft{}{2}(\tilde{C}_r)_{fg} \big)^2 \Big) \\
&\leq \sum_r^{\hat{R}} (\lambda_r^2 \sum_i a_{ri}^2 \sum_j b_{rj}^2) \sum_r^{\hat{R}} \sum_{f,g} \big(\ourfft{}{2}(C_r)_{fg} - \ourfft{}{2}(\tilde{C}_r)_{fg} \big)^2  \\
&\leq \sum_r^{\hat{R}}\lambda_r^2 \hat{R} k_x k_y \mu^2
\end{split}
\end{equation*}

\textbf{Bound $\psi=\sum_{f,g,i,j}\Big( \sum_r^{\hat{R}^{(k)}}\big(\lambda_r^{(k)}a_{ri}^{(k)}b_{rj}^{(k)}-\tilde{\lambda}_r^{(k)}\tilde{a}_{ri}^{(k)}\tilde{b}_{rj}^{(k)}\big) \ourfft{}{2}(\tilde{C}_r^{(k)})_{fg}\Big)^2$:}  Similarly, we remove the layer number $(k)$ for ease of reading. Then we have

\begin{equation*}
\begin{split}
\MoveEqLeft \sum_{f,g,i,j}\Big( \sum_r^{\hat{R}^{(k)}}\big(\lambda_r^{(k)}a_{ri}^{(k)}b_{rj}^{(k)}-\tilde{\lambda}_r^{(k)}\tilde{a}_{ri}^{(k)}\tilde{b}_{rj}^{(k)}\big) \ourfft{}{2}(\tilde{C}_r^{(k)})_{fg}\Big)^2 \\
&\leq \sum_{f,g,i,j}\big( \sum_r^{\hat{R}}(\lambda_r a_{ri} b_{rj} - \tilde{\lambda}_r \tilde{a}_{ri} \tilde{b}_{rj})^2 \sum_r^{\hat{R}} \ourfft{}{2}(\tilde{C}_r)_{fg}^2 \big) \\
&= \sum_{f,g,i,j}\Big( \sum_r^{\hat{R}}\big(\lambda_r (a_{ri} b_{rj}-\tilde{a}_{ri} \tilde{b}_{rj}) + (\lambda_r-\tilde{\lambda}_r) \tilde{a}_{ri} \tilde{b}_{rj} \big)^2 \sum_r^{\hat{R}} \ourfft{}{2}(\tilde{C}_r)_{fg}^2 \Big) \\
&\leq \sum_{f,g,i,j} \Big( \big( 2\sum_r^{\hat{R}}\lambda_r^2(a_{ri} b_{rj}-\tilde{a}_{ri} \tilde{b}_{rj})^2 + 2\sum_r^{\hat{R}} (\lambda_r-\tilde{\lambda}_r)^2 \tilde{a}_{ri}^2 \tilde{b}_{rj}^2 \big)  \sum_r^{\hat{R}} \ourfft{}{2}(\tilde{C}_r)_{fg}^2\Big) \\
&= \sum_{f,g,i,j} \Bigg( \bigg( 2\sum_r^{\hat{R}}\lambda_r^2 \big( a_{ri} (b_{rj}-\tilde{b}_{rj}) + (a_{ri}-\tilde{a}_{ri}) \tilde{b}_{rj} \big)^2 + 2\sum_r^{\hat{R}} (\lambda_r-\tilde{\lambda}_r)^2 \tilde{a}_{ri}^2 \tilde{b}_{rj}^2 \bigg)  \sum_r^{\hat{R}} \ourfft{}{2}(\tilde{C}_r)_{fg}^2\Bigg) \\
&\leq \sum_{f,g,i,j} \Bigg( \bigg( 4\sum_r^{\hat{R}}\lambda_r^2 \big( a_{ri}^2 (b_{rj}-\tilde{b}_{rj})^2 + (a_{ri}-\tilde{a}_{ri})^2 \tilde{b}_{rj}^2 \big)^2 + 2\sum_r^{\hat{R}} (\lambda_r-\tilde{\lambda}_r)^2 \tilde{a}_{ri}^2 \tilde{b}_{rj}^2 \bigg)  \sum_r^{\hat{R}} \ourfft{}{2}(\tilde{C}_r)_{fg}^2\Bigg) \\
&= \bigg( 4\sum_r^{\hat{R}}\lambda_r^2 \big( \sum_i a_{ri}^2 \sum_j(b_{rj}-\tilde{b}_{rj})^2 + \sum_i(a_{ri}-\tilde{a}_{ri})^2 \sum_j\tilde{b}_{rj}^2 \big)^2 + 2\sum_r^{\hat{R}} (\lambda_r-\tilde{\lambda}_r)^2 \sum_i\tilde{a}_{ri}^2 \sum_j\tilde{b}_{rj}^2 \bigg)  \sum_r^{\hat{R}} \sum_{f.g}\ourfft{}{2}(\tilde{C}_r)_{fg}^2 \\
&\leq \big(4\sum_r^{\hat{R}}\lambda_r^2(o\mu^2+s\mu^2)+2\hat{R}\mu^2\big)\hat{R} \\
&= \big(4\sum_r^{\hat{R}}\lambda_r^2(o+s)\hat{R}+2\hat{R}^2\big) \mu^2
\end{split}
\end{equation*}

\textbf{Bound $\epsilon^{(k)}=\fronorm{\tilde{\mytensor{Y}}^{(k)}-\mytensor{Y}^{(k)}}$:}  Similarly, we remove the layer number $(k)$. And we let $w_i=\ourfft{1,2}{3}(\mytensor{X}^{(k)})_{fgi}$, $\tilde{w}_i=\ourfft{1,2}{3}(\tilde{\mytensor{X}}^{(k)})_{fgi}$, $u_i=\sum_r^{\hat{R}}\lambda_r^{(k)}a_{ri}^{(k)}b_{rj}^{(k)}\ourfft{}{2}(C_r^{(k)})_{fg}$ and $\tilde{u}_i=\sum_r^{\hat{R}}\tilde{\lambda}_r^{(k)}\tilde{a}_{ri}^{(k)}\tilde{b}_{rj}^{(k)}\ourfft{}{2}(\tilde{C}_r^{(k)})_{fg}$.

\begin{equation*}
\begin{split}
\MoveEqLeft \fronorm{\tilde{\mytensor{Y}}^{(k)}-\mytensor{Y}^{(k)}}^2 \\
&= \fronorm{\ourfft{1,2}{3}(\tilde{\mytensor{Y}}^{(k)})-\ourfft{1,2}{3}(\mytensor{Y}^{(k)})}^2 \\
&= \sum_{f,g,j} \absbig{[\ourfft{1,2}{3}(\tilde{\mytensor{Y}}^{(k)})]_{fgj}-[\ourfft{1,2}{3}(\mytensor{Y}^{(k)})]_{fgj}}^2 \\
&= \sum_{f,g,j} HW (\sum_i w_i u_i-\sum_i \tilde{w}_i \tilde{u}_i)^2 \\
&= HW \sum_{f,g,j} \big( \sum_i w_i(u_i-\tilde{u}_i) + \sum_i (w_i-\tilde{w}_i) \tilde{u}_i \big)^2 \\
&\leq 2HW \sum_{f,g,j} \big( \sum_i w_i(u_i-\tilde{u}_i)\big)^2 + 2 \sum_{f,g,j}\big(\sum_i (w_i-\tilde{w}_i) \tilde{u}_i \big)^2 \\
&\leq 2HW \sum_{f,g,j}\big((\sum_i w_i^2) \sum_i (u_i-\tilde{u}_i)^2\big) + 2\sum_{f,g,j}\big(\sum_i(w_i-\tilde{w}_i)^2(\sum_i \tilde{u}_i)^2\big) \\
&\leq 2HW (\sum_{f,g,i}w_i^2)\sum_{f,g,i,j}(u_i-\tilde{u}_i)^2 + 2\sum_{f,g,i}(w_i-\tilde{w}_i)^2\sum_{f,g,i,j}\tilde{u}_i^2 \\
&\leq 2HW (\sum_{f,g,i}w_i^2)(2\varphi + 2\psi)+ 2\sum_{f,g,i}(w_i-\tilde{w}_i)^2\sum_{f,g,i,j}\tilde{u}_i^2 \\
&\leq 4HW \fronorm{\mytensor{X}^{(k)}}^2 \mu^2 \big( \sum_r^{\hat{R}}\lambda_r^2 \hat{R} k_x k_y + 4\sum_r^{\hat{R}}\lambda_r^2(o+s)\hat{R}+2\hat{R}^2\big) + 2\sum_{f,g,i}(w_i-\tilde{w}_i)^2\sum_{f,g,i,j}\tilde{u}_i^2 \\
&\leq 4HW \fronorm{\mytensor{X}^{(k)}}^2 \mu^2 \big( \sum_r^{\hat{R}}\lambda_r^2 \hat{R} k_x k_y + 4\sum_r^{\hat{R}}\lambda_r^2(o+s)\hat{R}+2\hat{R}^2\big) + 2(\fronorm{\mytensor{X}^{(k)}-\tilde{\mytensor{X}}^{(k)}}^2 \fronorm{\tilde{\mytensor{M}}}^2) \\
\end{split}
\end{equation*}

When $k=1$, we know that $\mytensor{X}^{(1)}=\tilde{\mytensor{X}}^{(1)}$, so

\begin{equation*}
\begin{split}
\MoveEqLeft \fronorm{\tilde{\mytensor{Y}}^{(1)}-\mytensor{Y}^{(1)}}^2 \\
&\leq 4HW \fronorm{\mytensor{X}^{(1)}}^2 \mu^2 \big( \sum_r^{\hat{R}}\lambda_r^2 \hat{R} k_x k_y + 4\sum_r^{\hat{R}}\lambda_r^2(o+s)\hat{R}+2\hat{R}^2\big)\\
\end{split}
\end{equation*}

When $k>1$, we have

\begin{equation*}
\begin{split}
\MoveEqLeft \fronorm{\tilde{\mytensor{Y}}^{(k)}-\mytensor{Y}^{(k)}}^2 \\
&\leq 4HW \fronorm{\mytensor{X}^{(k)}}^2 \mu^2 \big( \sum_r^{\hat{R}}\lambda_r^2 \hat{R} k_x k_y + 4\sum_r^{\hat{R}}\lambda_r^2(o+s)\hat{R}+2\hat{R}^2\big) + 2(\fronorm{\mytensor{X}^{(k)}-\tilde{\mytensor{X}}^{(k)}}^2 \fronorm{\tilde{\mytensor{M}}}^2) \\
&\leq 4HW \fronorm{\mytensor{X}^{(k)}}^2 \mu^2 \big( \sum_r^{\hat{R}}\lambda_r^2 \hat{R} k_x k_y + 4\sum_r^{\hat{R}}\lambda_r^2(o+s)\hat{R}+2\hat{R}^2\big) + 2(\fronorm{\relu{\mytensor{Y}^{(k-1)}}-\relu{\tilde{\mytensor{Y}}^{(k-1)}}}^2 \fronorm{\tilde{\mytensor{M}}}^2) \\
&\leq 4HW \fronorm{\mytensor{X}^{(k)}}^2 \mu^2 \big( \sum_r^{\hat{R}}\lambda_r^2 \hat{R} k_x k_y + 4\sum_r^{\hat{R}}\lambda_r^2(o+s)\hat{R}+2\hat{R}^2\big) + 2(\fronorm{\mytensor{Y}^{(k-1)}-\tilde{\mytensor{Y}}^{(k-1)}}^2 \fronorm{\tilde{\mytensor{M}}}^2) \\
&\leq 4HW \fronorm{\mytensor{X}^{(k)}}^2 \mu^2 \big( \sum_r^{\hat{R}}\lambda_r^2 \hat{R} k_x k_y + 4\sum_r^{\hat{R}}\lambda_r^2(o+s)\hat{R}+2\hat{R}^2\big) + 2((\epsilon^{(k-1)})^2\fronorm{\tilde{\mytensor{M}}}^2) \\
\end{split}
\end{equation*}

Let $\alpha^{(k)}=4HW \fronorm{\mytensor{X}^{(k)}}^2 \big( \sum_r^{\hat{R}^{(k)}}(\lambda_r^{(k)})^2 \hat{R}^{(k)} k_x^{(k)} k_y^{(k)} + 4\sum_r^{\hat{R}^{(k)}}(\lambda_r^{(k)})^2(o^{(k)}+s^{(k)})\hat{R}^{(k)}+2(\hat{R}^{(k)})^2 \big) \mu^2$, \\
and $\beta^{(k)}=2\fronorm{\tilde{\mytensor{M}}^{(k)}}^2$. Then the difference between the final output of the two networks are bounded by:
\begin{equation*}
\begin{split}
\MoveEqLeft \fronorm{\hat{\mathbb{M}}(\mytensor{X})-\tilde{\mathbb{M}}(\mytensor{X})}^2 \\
&= \fronorm{\relu{(\hat{\mytensor{Y}})}-\relu{(\tilde{\mytensor{Y}})}}^2 \\
&\leq \fronorm{\hat{\mytensor{Y}}-\tilde{\mytensor{Y}}}^2 \\
&\leq \sum_{k=1}^n\alpha^{(k)}\prod_{i=k+1}^n\beta^{(i)}
\end{split}
\end{equation*}

Since $\forall k \in [n], \norm{\mytensor{X}^{(k)}} \leq \Pi_{i = k}^n \frac{\fronorm{\mytensor{X}^{(n+1)}}}{\lc^{(i)} \fronorm{\mytensor{M}^{(i)}}} $, to obtain an $\epsilon$-cover of the compressed network, we can first assume $\beta^{(k)} \geq 1 \ \forall k \in [n]$. Then $\mu$ need to satisfy:

$$\mu \leq \frac{\epsilon}{ 2\sqrt{HW}n\fronorm{\mytensor{X}^{(n+1)}}  \hat{R}^{(*)} (\frac{\sqrt{2}\fronorm{\tilde{\mytensor{M}}^{(*)}}}{\lc^{(*)}\fronorm{\mytensor{M}^{(*)}}})^n \sqrt{(\lambda^{(*)})^2 k_x^{(*)} k_y^{(*)} + 4(\lambda^{(*)})^2(o^{(*)}+s^{(*)})+2}}$$

where $\hat{R}^{(*)} = \max_{k} r^{(k)}$ $\lambda^{(*)} = \max_{r, k} \lambda_{r}^{(k)}$, $s^{(*)} = \max_{k} s^{(k)}$, $o^{(*)} = \max_{k} o^{(k)}$, $k_x^{(*)} = \max_{k} k_x^{(k)}$, $k_y^{(*)} = \max_{k} k_y^{(k)}$ and
$\frac{\fronorm{\tilde{\mytensor{M}}^{(*)}}} { \mu^{(*)} \fronorm{\mytensor{M}^{(*)}}} = \max_{k} \frac{\fronorm{\tilde{\mytensor{M}}^{(k)}}} {\mu^{(k)} \fronorm{\mytensor{M}^{(*)}}}$

As when $\mu$ is fixed, the number of networks in our cover will at most be $(\frac{1}{\mu})^d$ where $d$ denote the number of parameters in the compressed network. Hence, the covering number w.r.t to a given $\epsilon$ is $\tilde{O}(n d)$ ($n$ is the number of layers in the given neural network). As for practical neural networks, the number of layers $n$ is usually much less than $O(\log(d))$, thus the covering number we obtained w.r.t to a given $\epsilon$ is just $\tilde{O}(d)$ for practical neural networks.

\end{proof}

Machine Learning (cs.LG); Machine Learning (stat.ML)