Tensor-structured sketching for constrained least squares

Abstract: Constrained least squares problems arise in many applications. Their memory and computation costs are expensive in practice involving high-dimensional input data. We employ the so-called "sketching" strategy to project the least squares problem onto a space of a much lower "sketching dimension" via a random sketching matrix. The key idea of sketching is to reduce the dimension of the problem as much as possible while maintaining the approximation accuracy.Tensor structure is often present in the data matrices of least squares, including linearized inverse problems and tensor decompositions. In this work, we utilize a general class of row-wise tensorized sub-Gaussian matrices as sketching matrices in constrained optimizations for the sketching design's compatibility with tensor structures. We provide theoretical guarantees on the sketching dimension in terms of error criterion and probability failure rate. In the context of unconstrained linear regressions, we obtain an optimal estimate for the sketching dimension. For optimization problems with general constraint sets, we show that the sketching dimension depends on a statistical complexity that characterizes the geometry of the underlying problems. Our theories are demonstrated in a few concrete examples, including unconstrained linear regression and sparse recovery problems.

\section{Introduction}
Constrained optimization plays an important role in the intersection of machine learning \cite{BCN18}, computational mathematics \cite{NW06}, theoretical computer science \cite{B15} and many other fields. We consider the least squares problem of the following form:
\begin{equation}
\label{QP}
\mathbf{x}^*:= \arg\min_{\mathbf{x}\in\bm{\mathcal{C}}}\|\mathbf{A}\mathbf{x}-\mathbf{b}\|_2^2,
\end{equation}
where $\bm{\mathcal{C}}\subset \mathbb{R}^p$ is the convex constraint set, and $\mathbf{A}\in\mathbb{R}^{n\times p}$ and $\mathbf{b}\in \mathbb{R}^n$ are respectively the coefficient matrix and vector. When the optimal solutions are not unique, we denote $\mathbf{x}^*\in\mathbb{R}^p$ as one of the minimizers.

A naive approach to solve this over-determined problem \@ifstar\@eqref\@@eqref{QP} $(m \ll n)$ requires polynomial time in the ambient dimension $n$, which is not ideal as $n$ is remarkably high in large-scale optimization settings. \emph{Sketching} is a leading alternative to approximate a high-dimensional system with lower-dimensionional representations for less data memory, storage and computation complexity. It introduces a random matrix $\mathbf{S}$ of size $m \times n$, called the sketching matrix, to reduce the original program \@ifstar\@eqref\@@eqref{QP} in a smaller problem:
\begin{equation}
\label{SQP}
\hat{\mathbf{x}}:=\arg\min_{\mathbf{x}\in\bm{\mathcal{C}}}\|(\mathbf{S}\mathbf{A})\mathbf{x}-(\mathbf{S}\mathbf{b})\|_2^2,
\end{equation}
where the sketched coefficients $\mathbf{S}\mathbf{A}, \mathbf{S}\mathbf{b}$ are merely of $m$ dimensions rather than $n.$

In terms of error guarantees, we consider the following prediction error
\begin{equation}
\label{eqn:err}
\|\mathbf{A}\hat{\mathbf{x}}-\mathbf{b}\|_2^2 \leq (1+ \varepsilon)^2\|\mathbf{A}\mathbf{x}^*-\mathbf{b}\|_2^2
\end{equation}
with high probability for a pre-specified error criterion $\varepsilon\in (0,1)$.
It is an interesting question how to design a sketching matrix $\mathbf{S}$ with limited sketching dimension $m$ that can achieve small prediction error in the sense of \@ifstar\@eqref\@@eqref{eqn:err}. This question has been well investigated in a line of work \cite{DMM06, RT08, MM13, CW13, ANW14, PW15, PW16} based on common dimensionality reduction methods like CountSketch \cite{CCC04}, sparse $0$-$1$ matrices \cite{DKS10}, Gaussian and sub-Gaussian matrices \cite{RV08, MPT07}, and FFT-based fast constructions \cite{AC06,T11}.

In many applications, the coefficient data $\mathbf{A},\mathbf{b}$ admit multi-linear (tensor) structures, for example in: spatio-temporal data analysis \cite{ BYL14, H15}, higher-order tensor decompositions \cite{DDV00, KB09}, approximating polynomial kernels \cite{KK12, PP13}, linearized PDE inverse problems \cite{CLNW19, JCCL19} and so on. In the sketching setting, we can utilize the tensor structure in the original objective function to speed up forming the sketched problem by making the sketching matrix have a corresponding tensor structure. In this paper, we study the sketching matrix whose rows are tensor products of sub-Gaussian vectors. This design is natural to process the tensor data in the program. Previous works that share similar set-ups are \cite{SGTU18, CLNW19, RR20}. We include the details of comparisons in later discussions.

Apart from the row-wise tensorized sketches, there are other multi-linear random projections and sampling strategies that work well in practice.
For sketching constructions employing fast matrix-vector multiplications, Pagh et al. \cite{P11, PP13} develop and analyze the TensorSketch method which uses fast Fourier transform (FFT) and CountSketch \cite{CCC04} techniques. This method is efficient while being applied to Kronecker products of vectors in the context of kernel machines. Later on, \cite{DSSW18} provides applications of TensorSketch to Kronecker product regressions as well as multimodal $p$-spline tensor sketching.
Another line of works is represented by \cite{JKW19, MB19}. The authors respectively consider the so-called Kronecker FJLT. The analysis in both papers focuses on the subspace embedding property which can be seen as a stepping stone for sketching linear regressions.
In the importance sampling regime, an efficient sublinear algorithm is provided by \cite{CPLP16} to sample the tensor CP alternating least squares (CP-ALS) problem \cite{KB09} by estimating the statistical leverage scores. \cite{LHW17} analyzes sparse sketching techniques in low-rank tensor regressions for tensor CP and Tucker decompositions.
For computational advantage from the tensor structure in the sketching matrices, please see the works \cite{CLNW19,PP13,JKW19} for details.

The aforementioned previous works mostly rely on the Johnson-Lindenstrauss property \cite{JL84} to embed pairwise distances and derive concentrations. While such strategy is easy to apply and powerful for subspace embedding, it falls short of capturing the essential dimension of a subset and thus is suboptimal in most constrained optimization problems. Recently, Pilanci et al. \cite{PW15,PW16} obtain sharp guarantees for constrained convex programs via the \textit{Gaussian width} of the constraint set. Gaussian width is a widely used notion in learning theory, statistics and geometric analysis to estimate reconstruction errors of sub-Gaussian measurements \cite{KM05,BBM05,MPT07}. It is capable of providing sharper bound than common concentration techniques like the union bound and $\varepsilon$-net \cite{N16}.

\subsection{Our contributions}
The current work aims to capture the essential geometry of the constraint set in \@ifstar\@eqref\@@eqref{QP} and gain computational advantage from the tensor structure of the sketching matrix at the same time. In particular, our result adopts the same Gaussian width measure as in prior works \cite{MPT07,PW15} and extends the theory to tensorized sub-Gaussian cases. To the best of our knowledge, this paper presents the state-of-the-art sketching dimension for the row-wise tensor sketching matrices.

For the sake of simplicity of the exposition, we first present the main result \cref{informal} for the sketching design constructed with standard Gaussian \footnote{A standard Gaussian vector has i.i.d. $\mathcal{N} (0,1)$ entries.} and Rademacher \footnote{A random Rademacher vector has i.i.d. entries which take the values $-1, 1$ with equal probability $1/2.$} vectors in the example of unconstrained linear regression problems. We would like to remark that a more comprehensive result later shown in \cref{sketch_dimension} holds for a wide class of sub-Gaussian matrices with flexible density level and for all types of constrained least squares. For the proof of \cref{informal}, we refer the readers to the proof of \cref{sketch_dimension} shown in \cref{sketch_dimension proof} and the proof of \cref{unconstrained}.
\begin{theorem}
\label{informal}
Let $n_1, n_2, p, m \in\mathbb{N}^+.$ Consider the linear regression problem \@ifstar\@eqref\@@eqref{QP} $(n=n_1n_2)$ with coefficient matrix $\mathbf{A}\in \mathbb{R}^{n_1n_2\times p}$ and the constraint $\bm{\mathcal{C}}=\mathbb{R}^p$. Fix the error criterion $ \varepsilon\in(0,1)$ and the failure probability $\delta \in (0,\frac{1}{2}).$ Let $\mathbf{S}\in\mathbb{R}^{m \times n_1n_2}$ be a sketching matrix such that each row of $\mathbf{S}$ is an independent tensor product of a standard Gaussian vector of length $n_1$ and a Rademacher vector of length $n_2$. If the sketching dimension $m$ satisfies
\begin{equation}
\label{opt_rank(A)}
m = \mathcal{O}\left(\frac{\max\left(4\, \text{rank}(\mathbf{A}), |\log\delta| \right)}{\varepsilon^2}\right),
\end{equation}
then the sketched solution $\hat{\mathbf{x}}\in\mathbb{R}^{p}$ in \@ifstar\@eqref\@@eqref{SQP} satisfies the accuracy as in \@ifstar\@eqref\@@eqref{eqn:err} with probability exceeding $1-\delta.$
\end{theorem}
The estimate \@ifstar\@eqref\@@eqref{opt_rank(A)} indicates that for an unconstrained least squares \cref{QP} of large dimension $n$, one can obtain an accurate approximated solution with high probability by solving the sketched problem \cref{SQP} of dimension $m$, which can be as small as $\mathcal{O}(\text{rank}(\mathbf{A}))$.
Our result \@ifstar\@eqref\@@eqref{opt_rank(A)} matches the result of the optimal unstructured sub-Gaussian sketching matrix in terms of their dependences on $\text{rank}(\mathbf{A})$, $\varepsilon$ and $\delta$. We conclude that a row-wise tensor sketching strategy possesses both the computational advantage and the same accuracy as unstructured sketchings.

In addition, we manage to derive an estimate for constrained least squares, see \cref{sketch_dimension}. To this end, the generic chaining method is employed to bound the supreme embedding error over a convex cone generated by the true solution $\mathbf{x}^*$ and the constraint set $\bm{\mathcal{C}}$ in \@ifstar\@eqref\@@eqref{QP}. Such geometric description for constrained problems results in applications for various optimizations, such as $\ell_1$-constrained problems, see \cref{sparse recovery}.

\subsection{Related work}
Sketching designs of row-wise tensor structure are previously studied in \cite{SGTU18, CLNW19, RR20} with applications for data memory reductions, tensor decompositions and linear inverse problems. Through the lens of unconstrained linear regressions, by applying a standard covering net technique, their results are suboptimal. The sketching sizes $m$ in the mentioned works are bounded below by $\text{rank}^8(\mathbf{A}), \text{rank}^6(\mathbf{A}), \text{rank}^4(\mathbf{A})$ respectively for a qualified sketching to the linear regression in \@ifstar\@eqref\@@eqref{QP}. Our work improves the estimate of $m$ to a linear dependence on the rank of $\mathbf{A}.$

A similar construction of the sketching matrix is considered in Meister et al. \cite{MSW19}, in which they form the sketching as the composition of a tensorized Rademacher matrix and a sparse $\{0,\pm1\}$-valued random matrix called CountSketch \cite{CCC04}. Compared to their sketching design, the construction of $\mathbf{S}$ in this work considers a wider class of sub-Gaussian random variables. Moreover, our construction is easy to implement with adjustable sparsity.
In terms of accuracy, the authors of \cite{MSW19} show their design is nearly optimal in vector-based embeddings. Particularly, by applying their result to the context of subspace embedding, their work gives a necessary lower bound of the sketching dimension
\begin{equation}
\label{eqn:MSW19}
m \geq \max\left(\mathcal{O}\left(\frac{\text{rank}(\mathbf{A})+|\log \delta\, \varepsilon|}{\varepsilon^2}\right), \mathcal{O}\left(\frac{\text{rank}^2(\mathbf{A})+|\log \delta\, \varepsilon|^2}{\varepsilon}\right)\right),
\end{equation}
to restrict the sketching error $\varepsilon$ in the sense of \@ifstar\@eqref\@@eqref{eqn:err} with probability $1-\delta.$ The estimate is shown to be optimal for their specific sparse sketching but unknown for other dense matrices.
In comparison, we introduce a density level parameter $q$ to adjust the number of nonzeros in the sketching matrix. In addition, our estimate $\max\left(\mathcal{O}\left(\frac{\text{rank}(\mathbf{A})}{\varepsilon^2\, q^2}\right), \mathcal{O}\left(\frac{|\log \delta|}{\varepsilon^2\, q}\right)\right)$ derived from \cref{unconstrained} is able to match the first term in \@ifstar\@eqref\@@eqref{eqn:MSW19} in small $\varepsilon$ regime, if setting the density $q$ as a constant. In the large $\varepsilon$ regime, it can be shown that our result can match the second term in \@ifstar\@eqref\@@eqref{eqn:MSW19}, if we choose $q$ larger than $\min (\frac{1}{\sqrt{\varepsilon\, \text{rank}(\mathbf{A})}}, \frac{1}{\varepsilon\, |\log \delta\, \varepsilon|}).$

\subsection{Notations}
In the paper, we denote by $\|~ \|_{2}$ and $\|~\|_{\infty}$ respectively the $\ell_2$ and $\ell_\infty$ norms of a vector $\mathbf{y}$, and $\|~ \|, ~\|~\|_{F}$ respectively the spectral and Frobenius norm of a matrix. The symbol $\mathbf{Pr}$ denotes the probability of an event, and the notion $\mathbb{E}$ is the expectation of a random variable. A $n$-dimensional vector following the distribution $\text{Ber}(q)^n$ has independent $\{0,1\}$-valued entries which take the value $1$ with probability $q.$
We use the calligraphic font $\bm{\mathcal{X}}$ to denote vector sets, the Roman script uppercase letter $\mathbf{X}$ to denote matrices, the Roman script lowercase letter $\mathbf{x}$ to denote vectors, and simple lowercase letter $x$ to denote a scalar entry. We write $\mathbf{I}_n \in \mathbb{R}^{n \times n}$ as the $n$ by $n$ identity matrix. We denote $c, \tilde{c}, C_1, C_2, C', C''$ some universal absolute constants. The usage of the constant notations are consistent throughout the paper.

\subsection{Organization of the paper}
The rest of the paper is organized as follows.
\cref{sec: 2} starts with the reasoning for the sketching design and follows the main result \cref{sketch_dimension} about the sketching dimension estimation, along with direct applications for two common classes of optimization problems.
In \cref{sec: 3}, we show an important intermediate result \cref{sup_error} about the supremum embedding error of the tensorized processes and give the complete proof of \cref{sketch_dimension} which is essentially built upon \cref{sup_error}.
The key ingredients and a simplified proof for \cref{sup_error} are then illustrated in \cref{sec: 4}.
\cref{sec: numerics} contains the numerical performance of the proposed sketching construction and provides empirical support for the theory.
We conclude and mention the future research directions in \cref{sec: conclusion}.
The major volume of the technical proofs of this paper, including the proofs for all the supporting results shown in \cref{sec: 4}, are in the appendix.

\section{Sketching dimension estimation}
\label{sec: 2}
The main goal of the paper is to develop a sketching size estimation for the constrained least squares \@ifstar\@eqref\@@eqref{SQP}.
In this section, we first introduce the tensor data structure in two examples as the motivation of our sketching matrix design. We present the main result \cref{sketch_dimension} on a nearly sharp sketching dimension bound in \cref{main result}. We conclude this section with classical optimization cases as direct results of the main theorem.

We begin with some linear algebra and probability background.
\begin{definition}
Given matrices $\mathbf{X}\in\mathbb{R}^{M_1\times N_1}$ and $\mathbf{Y}\in\mathbb{R}^{M_2\times N_2}$, the Kronecker product of $\mathbf{X}$ and $\mathbf{Y}$ is defined as
\begin{equation}
\mathbf{X} \otimes \mathbf{Y}= \left[
\begin{array}{cccc}
x_{1,1}\mathbf{Y} &x_{1,2}\mathbf{Y}  &\dots&x_{1,N_1}\mathbf{Y} \\
\vdots&\vdots &\ddots&\vdots\\
x_{M_1,1}\mathbf{Y} &x_{M_1,2}\mathbf{Y}  &\dots&x_{M_1,N_1}\mathbf{Y}  \\
\end{array}
\right]\in\mathbb{R}^{M_1M_2\times N_1N_2}.
\end{equation}
The Kronecker product satisfies the distributive property:
\begin{equation}
\label{distributive}
\mathbf{W}\mathbf{X}\otimes \mathbf{Y}\mathbf{Z} = (\mathbf{W}\otimes \mathbf{Y})(\mathbf{X}\otimes \mathbf{Z}).
\end{equation}

Assuming $M_1 = M_2$ and $N_1 = N_2,$ the Hadamard product is defined as
\begin{equation}
\mathbf{X}\circ \mathbf{Y} \in \mathbb{R}^{M_1 \times N_1},
\end{equation}
which is the element-wise product of the matrices.
\end{definition}

\begin{definition}
\label{psi_2}
The sub-Gaussian norm of a random variable $x \in \mathbb{R}$, denoted by $\|x\|_{\psi_2},$ is defined as:
$$
\|x\|_{\psi_2} = \inf\{t>0: ~~ \mathbb{E}\, {\text{exp}}(x^2/t^2) \leq 2\}.
$$
A random variable is sub-Gaussian if it has a bounded $\psi_2$ norm.
\end{definition}
We note that all normal random variables and all bounded r.v.s are sub-Gaussian r.v.s.

\begin{definition}
Given a set $\bm{\mathcal{Y}} \subset \mathbb{R}^n,$ we use $\overline{\bm{\mathcal{Y}}}$ to denote the normalized version of $\bm{\mathcal{Y}}:$
\begin{equation*}
\overline{\bm{\mathcal{Y}}} = \left\{\frac{\mathbf{y}}{\|\mathbf{y}\|_2}\in \mathbb{R}^n~~\Big\vert ~~\text{for}~\mathbf{y}\in\bm{\mathcal{Y}}\subset \mathbb{R}^n\right\}.
\end{equation*}
We define the Gaussian width of $\overline{\bm{\mathcal{Y}}}$ as:
\begin{equation}
\label{WY}
\mathbb{W}(\overline{\bm{\mathcal{Y}}}) = \mathbb{E} \sup_{\mathbf{y} \in \overline{\bm{\mathcal{Y}}}} \langle \mathbf{n},\mathbf{y}\rangle,
\end{equation}
where $\mathbf{n}\in \mathbb{R}^{n}$ is a sequence of independent standard normal variables.
\end{definition}

\subsection{Design of the sketching matrix}
\label{design}
Tensor structure is ubiquitous in applied mathematics \cite{KB09}, statistics \cite{AGHKT14} and deep learning \cite{CSS16}. Many datasets are naturally arranged with several attributes and can be represented as multi-arrays, namely tensors. Such structure also appears in the optimization problem \@ifstar\@eqref\@@eqref{QP} in practical applications. We initiate our sketching matrix design with two motivating examples: CANDECOMP/PARAFAC (CP) tensor decomposition and linearized PDE inverse problems.

CP tensor decomposition is an important tool for large-scale data analysis. The workhorse algorithm in fitting CP tensor decomposition is the alternating least squares, which solves the following convex optimization problem:
\begin{equation}
\label{eqn:als}
\arg\min_{\mathbf{X}\in \mathbb{R}^{R\times n }}\| \mathbf{A} \mathbf{X} - \mathbf{B} \|_F \,,
\end{equation}
where $\mathbf{A}$ and $\mathbf{B}$ are data matrices that are flattened from certain tensors. The flattening process forces columns of $\mathbf{A}$ to have tensor structures. In particular, let $\mathbf{a}$ be a column of $\mathbf{A}$, then it can be rewritten as a Kronecker product:
\[
\mathbf{a} = \bigotimes_{\ell=1}^d \mathbf{a}^{(\ell)}\,,
\]
where $d+1$ is the order of the original tensor from which $\mathbf{A}$ is unfolded. Due to the high computation cost of solving \@ifstar\@eqref\@@eqref{eqn:als}, sketching is used as a tool of solving CP tensor decomposition, see \cite{BBK18}. To relate the example to our work, we consider the $d=2$ case.

Another interesting example is the class of linearized PDE inverse problems. One famous case is the Electrical Impedance Tomography (EIT), which infers the body conductivity images from boundary measurements of voltage and current density on the surface. The EIT problem generates the Fredholm equation of the kind:
\begin{equation}\label{eqn:ip}
\int f_{i_1}(y)\, g_{i_2}(y)\, \sigma(y) \,dy = \text{data}_{i_1,i_2}\,, \quad \forall 1\leq i_1\leq n_1,\,  1\leq i_2\leq n_2.
\end{equation}
Here $y$ is the spatial variable and $\sigma$ is the image to reconstruct. The functions $f_{i_1},g_{i_2}$ are known from physical understanding of EIT, and $\text{data}_{i_1,i_2}$ are observed data. The subscript $i_1$ and $i_2$ are indices for where the voltage is applied on the body surface and where the current is measured. In practice, one has to place electric nodes $y_j$ on various places to produce sufficient data for the reconstruction, leading to a large number of equations with the same structure as in \@ifstar\@eqref\@@eqref{eqn:ip}. A numerical discretization of \@ifstar\@eqref\@@eqref{eqn:ip} then produces a linear equation
$
\mathbf{A} \mathbf{x}  = \mathbf{b},
$
where $a_{i,j} = f_{i_1}(y_j)\, g_{i_2}(y_j)$, $x_j = \sigma(y_j), b_i = \text{data}_{i_1,i_2}$ and the index $i$ is associated with a pair of indices $(i_1, i_2).$ The above over-determined system is usually solved as an optimization problem \@ifstar\@eqref\@@eqref{QP}. Each entry of $\mathbf{A}$ is a product of functions $f_{i_1}$ and $g_{i_2},$ and entries in one column share the same dependence on the spatial variable $y_j.$ Due to this, it can be shown that such data structure is equivalent to having Kronecker structure for all columns of $\mathbf{A}$, that is, a column $\mathbf{a}$ in $\mathbf{A}$ can be written as:
\[
\mathbf{a} = \mathbf{f} \otimes \mathbf{g}\in\mathbb{R}^{n_1n_2}
\]
for some column vectors $\mathbf{f} \in \mathbb{R}^{n_1}$ and $\mathbf{g}\in\mathbb{R}^{n_2}.$ For more details of tensor structures in linearized inverse problem, we refer the readers to \cite{CLNW19}.

In the aforementioned examples, we see that tensor structure appears in the the columns of input matrix $\mathbf{A}$ in the least squares objectives. When implementing the sketching strategy on the data matrix, the cost of the matrix multiplication $\mathbf{S}\mathbf{A}$ can be drastically reduced, if $\mathbf{S}$ has a consistent tensor structure in its rows, due to the distributive property of Kronecker product \@ifstar\@eqref\@@eqref{distributive}. To further reduce the time complexity, we consider the sketching constructed with only a portion of nonzero entries.
To this end, we design the sketching matrix $\mathbf{S}$ in the following form:
\begin{equation}
\label{sketch}
\mathbf{S} =
\frac{1}{\sqrt{m}}\,
\left[
\begin{array}{c}
\bm{\eta}_1^\top\otimes {\bm{\xi}}_1^\top\\
\vdots\\
\bm{\eta}_k^\top\otimes {\bm{\xi}}_k^\top\\
\vdots\\
\bm{\eta}_m^\top\otimes {\bm{\xi}}_m^\top\\
\end{array}
\right]\in \mathbb{R}^{m \times n_1n_2}\,,
\end{equation}
where $\bm{\eta}_k$ and ${\bm{\xi}}_k$ are independent copies of $\bm{\eta} \in \mathbb{R}^{n_1}, {\bm{\xi}}\in \mathbb{R}^{n_2}.$
The random vectors $\bm{\eta}$ and ${\bm{\xi}}$ are set to be
\begin{equation}
\label{sketch ii}
\bm{\eta} = \frac{1}{\sqrt{q}}\, \left(\bm{\phi}^{(1)} \circ \bm{\sigma}^{(1)}\right)\in \mathbb{R}^{n_1}, \quad {\bm{\xi}} = \frac{1}{\sqrt{q}}\, \left(\bm{\phi}^{(2)} \circ \bm{\sigma}^{(2)}\right)\in \mathbb{R}^{n_2},
\end{equation}
where $\bm{\phi}^{(1)} \in \mathbb{R}^{n_1}, \bm{\phi}^{(2)} \in \mathbb{R}^{n_2}$ are vectors with two sets of i.i.d. zero-mean, unit variance sub-Gaussian variables. We additionally assume the entries of $\bm{\phi}^{(2)}$ are uniformly bounded. The vectors $\bm{\sigma}^{(1)}$ and $\bm{\sigma}^{(2)}$ follow the distributions $\text{Ber}(q)^{n_1}$ and $\text{Ber}(q)^{n_2}$ respectively with $q\in(0,1]$.
We call $q$ as the density level. \footnote{We note that the density level $q$ is the percentage of nonzeros in expectation for each tensor factor of the sketching matrix $\mathbf{S}.$ As a result, the nonzero percentage of $\mathbf{S}$ is $q^2$ in expectation.

\begin{remark}
The bounded assumption can be imposed on either $\bm{\phi}^{(1)}$ or $\bm{\phi}^{(2)}$ due to the symmetry of the sketching construction.
\end{remark}

\subsection{Main result}
\label{main result}
In convex analysis, convex cones help represent the optimality condition for the program. The tangent cone of the optimum $\mathbf{x}^*$ with the constraint $\bm{\mathcal{C}}$ is defined as:
\begin{equation}
\label{cone}
\bm{\mathcal{K}}:= \{\mathbf{\Delta} = t(\mathbf{x}-\mathbf{x}^*)\in \mathbb{R}^p, \quad\text{for}~t\geq 0 ~\text{and}~ \mathbf{x} \in \bm{\mathcal{C}}\}.
\end{equation}
We focus on the transformed cone $\mathbf{A}\bm{\mathcal{K}}\in\mathbb{R}^{n_1n_2}$ to measure the gap between $\|\mathbf{A}\hat{\mathbf{x}}-\mathbf{b}\|_2^2$ and $\|\mathbf{A}\mathbf{x}^*-\mathbf{b}\|_2^2,$ due to the optimality of $\mathbf{x}^*.$ Of particular interest, our analysis exploits the Gaussian width of the cone $\mathbf{A}\bm{\mathcal{K}}$ after normalization: $\mathbb{W}(\overline{\mA\K})$ defined in \@ifstar\@eqref\@@eqref{WY}.

With the necessary preliminaries in place, we now illustrate the main result about the sketching dimension $m$ to guarantee a qualified output for a constrained program.
\begin{theorem}
\label{sketch_dimension}
Let $n_1, n_2, p, m \in\mathbb{N}^+.$ Consider the constrained least squares \@ifstar\@eqref\@@eqref{QP} ($n = n_1n_2$) with coefficient matrix $\mathbf{A}\in \mathbb{R}^{n_1n_2\times p},$ the vector $\mathbf{b}\in\mathbb{R}^{n_1n_2},$ the optimal solution $\mathbf{x}^*\in\mathbb{R}^p$ and the tangent cone $\bm{\mathcal{K}}\subset \mathbb{R}^p$ \@ifstar\@eqref\@@eqref{cone}. Fix the parameters $q\in (0,1],\varepsilon\in(0,1)$ and $\delta \in (0,\frac{1}{2}).$ If the sketching matrix $\mathbf{S}\in\mathbb{R}^{m \times n_1n_2}$ defined in \@ifstar\@eqref\@@eqref{sketch}-\@ifstar\@eqref\@@eqref{sketch ii} with density $q$ has the sketching size $m$ satisfying
\begin{equation}
\label{sketch_dimension_term}
m \geq C\, \max \left(\frac{\mathbb{W}(\overline{\mA\K})^2}{\varepsilon^2\, q^2}, \frac{|\log \delta|}{\varepsilon^2\, q}\right),
\end{equation}
for some finite constant $C>0$ \footnote{The constant $C$ in \@ifstar\@eqref\@@eqref{sketch_dimension_term} depends on the largest $\psi_2$ norm of the entries in $\bm{\phi}^{(1)}, \bm{\phi}^{(2)}$ and the maximal component magnitude $\max_{k\in[m]} \|{\bm{\xi}}_k\|_\infty,$ recalling the definitions in \@ifstar\@eqref\@@eqref{sketch} - \@ifstar\@eqref\@@eqref{sketch ii}. We refer the readers to equation \@ifstar\@eqref\@@eqref{C} in \cref{sketch_dimension proof} for the explicit choice of $C.$}, then the sketched solution $\hat{\mathbf{x}}\in\mathbb{R}^p$ of \@ifstar\@eqref\@@eqref{SQP} suffices to have
\begin{equation}
\label{eps_distort}
\|\mathbf{A}\hat{\mathbf{x}}-\mathbf{b}\|^2_2\leq (1+\varepsilon)^2\, \|\mathbf{A}\mathbf{x}^* -\mathbf{b}\|_2^2
\end{equation}
with probability exceeding $1-\delta.$
\end{theorem}
The proof of \cref{sketch_dimension} is in \cref{sketch_dimension proof}.

We emphasize that the bound of the sketching dimension $m$ \@ifstar\@eqref\@@eqref{sketch_dimension_term} is independent of the ambient dimensions $n_1$ and $n_2.$
The dependences on $\mathbb{W}(\overline{\mA\K}),$ the sketching error $\varepsilon,$ and failure probability $\delta$ are optimal and match the optimal result of the unstructured sub-Gaussian sketches given in \cite{PW15}.
Moreover, the squared Gaussian width $\mathbb{W}(\overline{\mA\K})^2$ is proportional to the statistical dimension of the cone $\mA\K.$ The statistical dimension is a robust version of the algebraic dimension for a linear subspace. (See Section 7.6 in \cite{V18} for details.)
We also see that the sketching dimension bound \@ifstar\@eqref\@@eqref{sketch_dimension_term} may change its dependence on the density $q,$ from $\frac{1}{q^2}$ to $\frac{1}{q},$ as $q$ increases. We investigate if such shift of dependence on $q$ exists by the numerical experiment shown in \cref{density}.

\subsection{Concrete case studies}
\label{applications}
\cref{sketch_dimension} lays out a unified framework to estimate the sketching dimension via the calculation of the Gaussian width $\mathbb{W}(\overline{\mA\K})$ based on the data matrix $\mathbf{A}$ and the geometry of the constraint $\bm{\mathcal{C}}.$ In this subsection, we provide the case studies for different kinds of optimization problems. In each case, we give an explicit computation for $\mathbb{W}(\overline{\mA\K})$ and hence obtain a guarantee for the sketch $\mathbf{S}$ as a direct corollary from \cref{sketch_dimension}.

\subsubsection{Unconstrained linear regression}
Unconstrained linear regression is the most common example of the convex program \@ifstar\@eqref\@@eqref{QP}, by setting $\bm{\mathcal{C}}$ to be the whole Euclidean space $\mathbb{R}^p.$ The following corollary shows that it suffices to have $\mathcal{O}(\text{rank}(\mathbf{A}))$ rows in the sketching matrix to solve an unconstrained linear regression problem. Our guarantee for the row-wise tensor sub-Gaussian sketch \@ifstar\@eqref\@@eqref{sketch} almost matches the optimal result for unstructured sub-Gaussian developed in \cite{PW15}. We additionally consider a density level $q$ for the sketching design.

\begin{corollary}
\label{unconstrained}
Let $n_1, n_2, p, m \in\mathbb{N}^+.$ Consider the linear regression problem \@ifstar\@eqref\@@eqref{QP} with data matrix $\mathbf{A}\in \mathbb{R}^{n_1n_2\times p}$ and the constraint $\bm{\mathcal{C}}=\mathbb{R}^p$. Fix the parameters $q\in (0,1],$ $\varepsilon\in(0,1)$ and $\delta \in (0,\frac{1}{2}).$ If the sketching matrix $\mathbf{S} \in \mathbb{R}^{m \times n_1n_2}$ defined in \@ifstar\@eqref\@@eqref{sketch}-\@ifstar\@eqref\@@eqref{sketch ii} with density $q$ has the sketching size
\begin{equation}
m \geq C\,\max\left(\frac{4\,\text{rank}(\mathbf{A})}{\varepsilon^2\, q^2}, \frac{|\log\delta|}{\varepsilon^2\, q} \right),
\end{equation}
then the sketched solution $\hat{\mathbf{x}}\in\mathbb{R}^{p}$ in \@ifstar\@eqref\@@eqref{SQP} satisfies the accuracy as in \@ifstar\@eqref\@@eqref{eps_distort} with probability exceeding $1-\delta.$ Above, $C>0$ is a finite constant.
\end{corollary}

\begin{proof}
The proof of \cref{unconstrained} is deduced from \@ifstar\@eqref\@@eqref{sketch_dimension_term} in \cref{sketch_dimension} by computing $\mathbb{W}(\overline{\mA\K}),$ which is the Gaussian width of the range of $\mathbf{A}$ intersected with the unit sphere $\mathcal{S}^{n_1n_2-1}.$ From Example 3.7 in \cite{V14}, the Gaussian width of a $r$-dimensional subset intersected with a unit sphere is upper bounded by $2\sqrt{r}.$ Since the dimension of $\text{Range}(\mathbf{A})$ is $\text{rank} (\mathbf{A}),$ we have
$
\mathbb{W}(\text{Range}(\mathbf{A})\cap \mathcal{S}^{n_1n_2-1}) \leq 2\, \sqrt{\text{rank} (\mathbf{A})}.
$
We complete the proof by plugging the above result in \@ifstar\@eqref\@@eqref{sketch_dimension_term}.
\end{proof}

\subsubsection{Sparse recovery via $\ell_1$-constrained optimization}
We study the noiseless sparse recovery problem, which plays a central role in compressive sensing and signal processing.
The goal of the recovery is to find a sparse $\hat{\mathbf{x}}\in\mathbb{R}^p$ to approximate an unknown signal $\overline{\mathbf{x}}\in\mathbb{R}^p$ (also sparse in practice), from a small number of random measurements $\bm{\Phi} \overline{\mathbf{x}} \in \mathbb{R}^m,$ with a short wide matrix $\bm{\Phi}\in\mathbb{R}^{m \times p} (m \ll p).$
Furthermore, we can optimize the sparse recovery by an $\ell_1$-constrained least squares problem \cite{FNW07}, also known as the Lasso approach \cite{RT96}. Such optimization has the formulation
\begin{equation}
\label{compressed}
\hat{\mathbf{x}} = \arg\min_{\|\mathbf{x}\|_1\leq R} \|\bm{\Phi}\mathbf{x}-\bm{\Phi}\overline{\mathbf{x}}\|_2^2,
\end{equation}
where $R>0$ is the radius of the $\ell_1$ ball.

We show that \cref{sketch_dimension} provides an estimation for the number of measurements $m$ to pursue an accurate recovery from the tensor sub-Gaussian sketching matrix $\mathbf{S}$ \@ifstar\@eqref\@@eqref{sketch}.
Recall the sketched program in \@ifstar\@eqref\@@eqref{SQP} and let $n_1n_2 = p,$
\begin{equation}
\label{sketch-lasso}
\mathbf{S} =\bm{\Phi} \in\mathbb{R}^{m\times p} ,\quad \mathbf{A} = \mathbf{I}_{p} \in \mathbb{R}^{p\times p}, \quad \mathbf{b} =\overline{\mathbf{x}}\in\mathbb{R}^p,
\end{equation}
the sketched least squares with an $\ell_1$ constraint,
\begin{equation}
\label{sketch l_1}
\hat{\mathbf{x}} = \arg\min_{\| \mathbf{x}\|_1\leq R} \|\mathbf{S}\mathbf{A}\mathbf{x}-\mathbf{S}\mathbf{b}\|_2^2
\end{equation}
is equivalent to the formulation \@ifstar\@eqref\@@eqref{compressed}.

Furthermore, in the case of full data acquisition, we denote the one solution
\begin{equation}
\label{uncompressed}
\mathbf{x}^* = \arg\min_{\|\mathbf{x}\|_1\leq R} \|\mathbf{x}-\overline{\mathbf{x}}\|_2^2,
\end{equation}
to have the least nonzeros among all optimizers.

\begin{corollary}
\label{sparse recovery}
Let $m, p \in \mathbb{N}^+,$ $\delta \in (0, \frac{1}{2}), \varepsilon \in (0,1)$ and $q\in(0,1]$. Consider the signal to uncover $\overline{\mathbf{x}}\in\mathbb{R}^p$. Suppose the optimal solution $\mathbf{x}^*\in\mathbb{R}^p$ of \@ifstar\@eqref\@@eqref{uncompressed} has $s$ nonzero entries. If $\bm{\Phi} \in \mathbb{R}^{m \times p}$ in \@ifstar\@eqref\@@eqref{compressed} defined as \@ifstar\@eqref\@@eqref{sketch}-\@ifstar\@eqref\@@eqref{sketch ii} ($n_1n_2=p$) has the number of measurements
\begin{equation}
\label{measurements}
m \geq C\, \max\left(\frac{\min\left(36\, s\log p, 4p \right)}{\varepsilon^2\, q^2}, \frac{|\log\delta|}{\varepsilon^2\, q}\right),
\end{equation}
then the solution $\hat{\mathbf{x}}\in \mathbb{R}^p$ of \@ifstar\@eqref\@@eqref{compressed} satisfies
\[
\|\hat{\mathbf{x}} - \overline{\mathbf{x}}\|_2^2 \leq (1+\varepsilon)^2\, \|\mathbf{x}^* - \overline{\mathbf{x}}\|_2^2,
\]
with probability exceeding $1-\delta.$ Above, $C>0$ is a finite constant.
\end{corollary}

\begin{proof}
Due to the equivalence of \@ifstar\@eqref\@@eqref{compressed} and \@ifstar\@eqref\@@eqref{sketch l_1} via the relationship \@ifstar\@eqref\@@eqref{sketch-lasso}, we apply \cref{sketch_dimension} and obtain a bound of $m$ for successful recovery, by calculating the constrained cone's Gaussian width $\mathbb{W}(\overline{\mA\K}) = \mathbb{W}(\overline{\bm{\mathcal{K}}}) ~(\mathbf{A} = \mathbf{I}_p).$
We thus focus on the tangent cone based on the constraint set $\bm{\mathcal{C}}= \{\mathbf{x}\in \mathbb{R}^p, ~\vert ~\|\mathbf{x}\|_1\leq R\}$. Recalling the definition of $\bm{\mathcal{K}}$ in \@ifstar\@eqref\@@eqref{cone}, due to $\mathbf{\Delta} = t\, (\mathbf{x}-\mathbf{x}^*),$ the cone can be written as
\begin{equation*}
\bm{\mathcal{K}}= \left\{\mathbf{\Delta}\in \mathbb{R}^p~~\vert~~\|\frac{\mathbf{\Delta}}{t}+\mathbf{x}^*\|_1 \leq R, ~\text{for}~t\geq 0\right\}.
\end{equation*}

\begin{enumerate}
\item
If the hyperparameter $R$ satisfies $R>\|\overline{\mathbf{x}}\|_1,$ then the constrained problem \@ifstar\@eqref\@@eqref{uncompressed} returns $\mathbf{x}^*$ as exactly the true signal $\overline{\mathbf{x}},$ and $\|\mathbf{x}^*\|_1 = \|\overline{\mathbf{x}}\|_1< R.$
The $\ell_1$ distance from $\mathbf{x}^*$ to the constraint boundary is strictly positive. Hence for any $\mathbf{\Delta} \in \mathbb{R}^p,$ there exists $t>0$ such that $\|\frac{\mathbf{\Delta}}{t}+\mathbf{x}^*\|_1\leq \|\frac{\mathbf{\Delta}}{t}\|_1+\|\mathbf{x}^*\|_1< R,$ meaning the tangent cone $\bm{\mathcal{K}}$ takes over the whole Euclidean space $\mathbb{R}^p$. Thus $\mathbb{W}(\overline{\bm{\mathcal{K}}})=\mathbb{W}(\mathcal{S}^p) \leq 2\,\sqrt{p}$ is obtained from the proof of \cref{unconstrained}.

\item
If $R\leq \|\overline{\mathbf{x}}\|_1,$ since both the constraint set and the least squares problem are strictly convex, the optimal $\mathbf{x}^*$ must be on the boundary of the $\ell_1$ ball constraint, implying $\|\mathbf{x}^*\|_1 =R$. Let the support of $\mathbf{x}^*$ be $S$ and $s=|S|.$ We denote $\mathbf{\Delta}_S, \mathbf{\Delta}_{S^c}$ the subvectors of $\mathbf{\Delta}$ in and outside the support $S.$ Note that
\begin{equation*}
\begin{array}{ll}
\displaystyle \|\frac{\mathbf{\Delta}}{t}+\mathbf{x}^*\|_1 &=\displaystyle \|\left(\frac{\mathbf{\Delta}}{t}+\mathbf{x}^*\right)_S\|_1+ \|\left(\frac{\mathbf{\Delta}}{t}+\mathbf{x}^*\right)_{S^c}\|_1 \\\\
&=\displaystyle \left\vert\langle \frac{\mathbf{\Delta}_S}{t}, \text{sign}(\mathbf{x}^*_S)\rangle +\|\mathbf{x}^*_S\|_1 \right\vert+\| \frac{\mathbf{\Delta}_{S^c}}{t}+\mathbf{x}^*_{S^c} \|_1 \\\\
&=\displaystyle \left\vert\langle \frac{\mathbf{\Delta}_S}{t}, \text{sign}(\mathbf{x}^*_S)\rangle +\|\mathbf{x}^*\|_1 \right\vert+\| \frac{\mathbf{\Delta}_{S^c}}{t} \|_1\leq R = \|\mathbf{x}^*\|_1.
\end{array}
\end{equation*}

This further yields that
\[
\langle\frac{\mathbf{\Delta}_S}{t}, \text{sign}(\mathbf{x}^*_S)\rangle +\| \frac{\mathbf{\Delta}_{S^c}}{t}\|_1 = \frac{\langle\mathbf{\Delta}_S, \text{sign}(\mathbf{x}^*_S)\rangle+\|\mathbf{\Delta}_{S^c}\|_1}{t} \leq 0.
\]
Since $t$ is any positive number, the condition can be expressed as
\[
\langle\mathbf{\Delta}_S, \text{sign}(\mathbf{x}^*)\rangle+\|\mathbf{\Delta}_{S^c}\|_1\leq 0,
\]
and
\begin{equation*}
\bm{\mathcal{K}}\subseteq \{\mathbf{\Delta}\in \mathbb{R}^p~~\vert~~\langle\text{sign} (\mathbf{x}^*_S), \mathbf{\Delta}_S \rangle +\|\mathbf{\Delta}_{S^c}\|_1\leq 0\}.
\end{equation*}

The Gaussian width computation follows a direct result in the proof of Corollary 3 in \cite{PW15}, that $\mathbb{W}(\overline{\bm{\mathcal{K}}}) \leq 6\, \sqrt{s\, \log p}.$
\end{enumerate}

Combining the discussions of the two cases, we can pick the free parameter $R$ to achieve the smaller $\mathbb{W}(\overline{\bm{\mathcal{K}}})$ of the two cases and conclude the proof by plugging the bound for $\mathbb{W}(\overline{\bm{\mathcal{K}}}): \min (6\, \sqrt{s\, \log p}, 2\,\sqrt{p})$ in the result \@ifstar\@eqref\@@eqref{sketch_dimension_term}.
\end{proof}

\section{Bounding the sketching size}
\label{sec: 3}
In this section, we prove the main result \cref{sketch_dimension} via \cref{sup_error} which provides an estimate of the supreme embedding error over a set from the sketching matrix $\mathbf{S}$.

\subsection{Analysis of tensor sub-Gaussian processes}
To bound the error \@ifstar\@eqref\@@eqref{eps_distort} in \cref{sketch_dimension}, it is sufficient to bound the embedding errors for the elements in the cone $\mA\K.$ In this regard, we study the behavior of the (centered) distribution $\mathbf{S}\mathbf{y}$ with the tensor-structured design $\mathbf{S}\in\mathbb{R}^{m \times n_1n_2}$ \@ifstar\@eqref\@@eqref{sketch}-\@ifstar\@eqref\@@eqref{sketch ii} and an arbitrarily chosen vector $\mathbf{y}$ from a set $\bm{\mathcal{Y}}.$ The analysis provides the knowledge to control the embedding error and the failure probability by adjusting the number of measurements $m$, and essentially yields the sketching size estimation for convex programs. In particular, we show that the embedding error is bounded by the Gaussian width of the normalized $\overline{\bm{\mathcal{Y}}}$ \@ifstar\@eqref\@@eqref{WY} over the square root of the sample size $m$, followed by a Gaussian tail.

\begin{theorem}
\label{sup_error}
There exist universal constants $c, C_1, C_2 >0$ for which the following holds. Let $n_1, n_2, m\in\mathbb{N}^+, q\in(0,1]$ and $\omega \in (0, \frac{1}{2}]$. For any fixed set $\bm{\mathcal{Y}} \subset \mathbb{R}^{n_1n_2},$ let the sketching matrix $\mathbf{S} \in \mathbb{R}^{m \times n_1n_2}$ be defined in \@ifstar\@eqref\@@eqref{sketch}-\@ifstar\@eqref\@@eqref{sketch ii} with density $q.$ If the integer $m \geq \frac{C_2\, \mathbb{W}(\overline{\bm{\mathcal{Y}}})^2}{q^2},$ then with probability at least $1-4\,{\text{exp}}(-c\, m \,  \frac{\omega^2\, q}{2}),$ one can achieve
\begin{equation}
\label{error}
\sup_{\mathbf{y} \in \overline{\bm{\mathcal{Y}}}} \left\vert\|\mathbf{S}\mathbf{y}\|_2^2 -1 \right\vert \leq  \alpha^2\, (\mathcal{M}^2 +1)\, \left(\frac{C_1\, \mathbb{W}(\overline{\bm{\mathcal{Y}}})}{\sqrt{m}\, q} +\omega\right)\,.
\end{equation}
Here, $\alpha \geq 1$\footnote{The value of $\alpha$ is greater than $1$ is because $\bm{\phi}^{(1)}, \bm{\phi}^{(2)}$ both have entries with zero-mean and standard deviation of $1$. And the standard deviation of a zero-mean r.v., which is essentially the $\ell_2$ norm, is no more than its $\psi_2$ norm. We refer the readers to Example 2.7.12 and 2.7.13 in \cite{V18} for details. \label{alpha}} is the maximal $\psi_2$ norm of entries in $\bm{\phi}^{(1)}, \bm{\phi}^{(2)}$ and $\mathcal{M}= \max_{k \in [m]} \|{\bm{\xi}}_k\|_\infty,$ recalling the definitions in \@ifstar\@eqref\@@eqref{sketch}-\@ifstar\@eqref\@@eqref{sketch ii}.
\end{theorem}
The proof of \cref{sup_error} is in \cref{sup_error proof}.

We stress that the failure probability term $4\,{\text{exp}}(-c\, m \,  \frac{\omega^2\, q}{2})$ is independent of the ambient dimensions $n_1, n_2,$ which are usually large in practice.
The error estimate \@ifstar\@eqref\@@eqref{error} implies that the density $q$ contributes to reducing the embedding error in a linear decay manner.
However, the tensor structure in the sketching design slightly weakens the theoretical result by an extra factor of $\mathcal{M}^2+1$ from the maximal coordinate of the tensor factors ${\bm{\xi}}_k$ \@ifstar\@eqref\@@eqref{sketch ii}. We explore this numerically in \cref{BN} and show that the maximal magnitude in the tensor-structured sketch indeed affects the sketching performance as the theory suggests.

\begin{remark}
\label{finite constant}
The scaling factor $\alpha^2\, (\mathcal{M}^2 +1)$ in the error bound \@ifstar\@eqref\@@eqref{error} is a finite constant. In fact, recalling \@ifstar\@eqref\@@eqref{sketch ii}, the entries of ${\bm{\xi}}_k\in\mathbb{R}^{n_2}$ are products of bounded random variables and Bernoulli variables, so they can be uniformly bounded by a finite $\mathcal{M}.$ Furthermore, the entries of $\bm{\phi}^{(1)}, \bm{\phi}^{(2)}$ are sub-Gaussian variables, thus the $\psi_2$ norms can also be uniformly bounded by a finite number $\alpha.$
\end{remark}

\subsection{Proof of \cref{sketch_dimension}}
\label{sketch_dimension proof}
\begin{proof}
The proof follows the ideas of Lemma 1, 2, 3 in \cite{PW15}. The next lemma relates the sketching error \@ifstar\@eqref\@@eqref{eps_distort} with two specific terms $D_1$ and $D_2,$ both of which can be further bounded by the supreme embedding error in \@ifstar\@eqref\@@eqref{error} over the cone $\mA\K.$

\begin{lemma}{(Lemma 1 in \cite{PW15}.)}
\label{D_2/D_1}
Let $n_1, n_2, p, m\in\mathbb{N}^+$. Consider the least squares \@ifstar\@eqref\@@eqref{QP} with data matrix $\mathbf{A} \in \mathbb{R}^{n_1n_2\times p},$ the vector $\mathbf{b}\in\mathbb{R}^{n_1n_2},$ the optimal solution $\mathbf{x}^*\in\mathbb{R}^p$ and the tangent cone $\bm{\mathcal{K}}\subset \mathbb{R}^p$ defined in \@ifstar\@eqref\@@eqref{cone}. For any fixed matrix $\mathbf{S}\in\mathbb{R}^{m\times n_1n_2}$, define the two quantities
\begin{equation*}
D_1 = \inf_{\mathbf{y}\in \overline{\mA\K}} \|\mathbf{S}\mathbf{y}\|_2^2, \quad D_2 = \sup_{\mathbf{y}\in \overline{\mA\K}} \left\vert\langle \frac{\mathbf{A}\mathbf{x}^*-\mathbf{b}}{\| \mathbf{A}\mathbf{x}^*-\mathbf{b}\|_2}, (\mathbf{S}^\top\mathbf{S}-\mathbf{I}_{n_1n_2})\,\mathbf{y} \rangle\right\vert.
\end{equation*}
Then the sketched solution $\hat{\mathbf{x}}$ of \@ifstar\@eqref\@@eqref{SQP} satisfies
\begin{equation}
\label{1+D_2/D_1}
\|\mathbf{A}\hat{\mathbf{x}}-\mathbf{b}\|_2^2 \leq \left(1+\frac{2\, D_2}{D_1}\right)\,\|\mathbf{A}\mathbf{x}^*-\mathbf{b}\|_2^2.
\end{equation}
\end{lemma}
\begin{remark}
\label{AKB=AKS}
We note that there is a difference between the set-ups of \cref{D_2/D_1} and Lemma 1 in \cite{PW15}. \cref{D_2/D_1} considers the vector $\mathbf{y}$ on the normalized set $\overline{\mA\K}$ while \cite{PW15} defines $\mathbf{y}$ to be on the intersection of $\mA\K$ and the unit sphere. In fact, it can be shown that $\overline{\mA\K} = \mA\K\cap \mathcal{S}^{n_1n_2-1},$ hence the result of \cite{PW15} is applicable for \cref{D_2/D_1}. We refer the readers to the proof of Lemma 1 in \cite{PW15} for details.
\end{remark}

It remains to bound $D_1, D_2$ to keep the accuracy of $\hat{\mathbf{x}}$ for \cref{sketch_dimension}. Respectively applying \cref{sup_error}, we obtain a lower bound for $D_1$ and an upper bound for $D_2.$
\begin{lemma}
\label{D_1 D_2}
There exist universal constants $c, C_1, C_2>0$ for which the following holds. Follow the same setting in \cref{D_2/D_1}. Fix $q \in (0,1], \omega \in (0, \frac{1}{2}]$ and the integer $m \geq \frac{C_2\,\mathbb{W}(\overline{\mA\K})^2}{q^2},$ then for a sketching matrix $\mathbf{S}\in\mathbb{R}^{m \times n_1n_2}$ defined in \@ifstar\@eqref\@@eqref{sketch}-\@ifstar\@eqref\@@eqref{sketch ii} and $\theta =\alpha^2\, (\mathcal{M}^2+1)$ given in \@ifstar\@eqref\@@eqref{error},
\begin{enumerate}
\item the event
\begin{equation}
\label{eqn: D_1}
D_1 = \inf_{\mathbf{y}\in \overline{\mA\K}} \|\mathbf{S}\mathbf{y}\|_2^2\geq 1- \theta\,\left(\frac{C_1\, \mathbb{W}(\overline{\mA\K})}{\sqrt{m}\, q} +\omega\right)
\end{equation}
achieves with probability exceeding $1-4\,{\text{exp}}(-c\, m \,  \frac{\omega^2\, q}{2});$

\item another event
\begin{equation}
\label{eqn: D_2}
D_2 = \sup_{\mathbf{y}\in \overline{\mA\K}} \left\vert\langle  \frac{\mathbf{A}\mathbf{x}^*-\mathbf{b}}{\| \mathbf{A}\mathbf{x}^*-\mathbf{b}\|_2}, (\mathbf{S}^\top\mathbf{S}-\mathbf{I}_{n_1n_2})\,\mathbf{y} \rangle\right\vert \leq \theta\, \left(\frac{5\,C_1\, \mathbb{W}(\overline{\mA\K})}{\sqrt{m}\, q} +3\,\omega\right)
\end{equation}
achieves with probability exceeding $1-24\,{\text{exp}}(-c\, m \,  \frac{\omega^2\, q}{2}).$
\end{enumerate}
\end{lemma}

We show the proof of \cref{D_1 D_2} in \cref{D_1 D_2 proof}.

We now prove \cref{sketch_dimension}. For fixed $\varepsilon\in(0,1)$ and $\delta \in (0,\frac{1}{2})$ given in \cref{sketch_dimension}, we let
\begin{equation}
\label{omega_eps}
\omega = \frac{\varepsilon}{16\, \theta+2\,\theta\, \varepsilon} < \frac{\varepsilon}{16\,\theta} < \frac{1}{2},
\end{equation}
where $\theta = \alpha^2\, (\mathcal{M}^2+1)\geq 1$ is set in  \cref{D_1 D_2}. Suppose $m\geq \frac{C_2\, \mathbb{W}(\overline{\mA\K})^2}{q^2},$ we are eligible to apply \cref{D_1 D_2}. By plugging in the bounds of $D_1, D_2$ provided in \@ifstar\@eqref\@@eqref{eqn: D_1}, \@ifstar\@eqref\@@eqref{eqn: D_2} and summing up the failure probabilities, then together with \@ifstar\@eqref\@@eqref{1+D_2/D_1} in \cref{D_2/D_1}, we have the accuracy estimation:
\begin{equation}
\label{123}
\|\mathbf{A}\hat{\mathbf{x}}-\mathbf{b}\|_2^2 \leq \left(1+ \frac{\theta\, \left(\frac{10\, C_1\, \mathbb{W}(\overline{\mA\K})}{\sqrt{m}\, q}+ 6\,\omega\right)}{1- \theta\left(\frac{C_1\, \mathbb{W}(\overline{\mA\K})}{\sqrt{m}\, q} +\omega\right)}\right)^2\, \|\mathbf{A}\mathbf{x}^*-\mathbf{b}\|_2^2
\end{equation}
achieves with probability $1-28\,{\text{exp}}(-c\, m \,  \frac{\omega^2\, q}{2}).$

Furthermore, we assume
\begin{equation}
\label{m_eps}
\begin{array}{ll}
m &\geq \displaystyle \max\left(\frac{C_1^2\, \mathbb{W}(\overline{\mA\K})^2}{\omega^2\, q^2}, ~~\frac{12\,|\log\delta|}{c\, \omega^2\, q}, ~~ \frac{C_2\, \mathbb{W}(\overline{\mA\K})^2}{q^2}\right),
\end{array}
\end{equation}
then $\frac{C_1\, \mathbb{W}(\overline{\mA\K})}{\sqrt{m}\, q} \leq\omega.$ By \@ifstar\@eqref\@@eqref{123}, one can achieve
\begin{equation*}
\begin{array}{ll}
\|\mathbf{A}\hat{\mathbf{x}}-\mathbf{b}\|_2^2 &\displaystyle \leq \left(1+ \frac{16\, \theta \, \omega}{1- 2\,\theta\, \omega}\right)^2\, \|\mathbf{A}\mathbf{x}^*-\mathbf{b}\|_2^2\\\\
&= (1+\varepsilon)^2\, \|\mathbf{A}\mathbf{x}^*-\mathbf{b}\|_2^2,
\end{array}
\end{equation*}
with probability exceeding $1-\delta.$
The relationship
$
\varepsilon = \frac{16\, \theta \, \omega}{1- 2\,\theta\, \omega}
$
is derived from \@ifstar\@eqref\@@eqref{omega_eps}.
The failure probability $\delta>28\,{\text{exp}}(-c\, m \,  \frac{\omega^2\, q}{2})$ is due to $\delta\in (0, \frac{1}{2})$ and $$\displaystyle m\geq \frac{12\,|\log\delta|}{c\, \omega^2\, q} >\frac{2|\log \frac{\delta}{28}|}{c\,\omega^2\, q}.$$

From the choice of $m$ in \@ifstar\@eqref\@@eqref{m_eps} and the relation of $\varepsilon$ and $\omega$ in \@ifstar\@eqref\@@eqref{omega_eps}, we define a finite constant
\begin{equation}
\label{C}
C = \max \left(16^2\, \theta^2\, C_1^2, ~~ \frac{16^2\cdot 12\, \theta^2}{c}, ~~C_2\right),
\end{equation}
where $\theta = \alpha^2\, (\mathcal{M}^2+1)$ given in \@ifstar\@eqref\@@eqref{error} is a finite constant, see \cref{finite constant}, and only depends on the distribution of the sketching matrix $\mathbf{S}.$ Then the assumption \@ifstar\@eqref\@@eqref{m_eps} on $m$ can be simplified via \@ifstar\@eqref\@@eqref{omega_eps} as follows
\begin{equation*}
\begin{array}{ll}
m &\displaystyle\geq C\,  \max \left(\frac{\mathbb{W}(\overline{\mA\K})^2}{\varepsilon^2\, q^2}, \frac{|\log \delta|}{\varepsilon^2\, q}\right)\\\\
&\geq \displaystyle \max\left(\frac{16^2\, \theta^2\, C_1^2\, \mathbb{W}(\overline{\mA\K})^2}{\varepsilon^2\, q^2}, ~~\frac{16^2\, \theta^2\, 12\,|\log\delta|}{c\, \varepsilon^2\, q},~~ \frac{C_2\, \mathbb{W}(\overline{\mA\K})^2}{\varepsilon^2\,q^2}\right)\\\\
&\geq \displaystyle \max\left(\frac{C_1^2\, \mathbb{W}(\overline{\mA\K})^2}{\omega^2\, q^2}, ~~\frac{12\,|\log\delta|}{c\, \omega^2\, q}, ~~ \frac{C_2\, \mathbb{W}(\overline{\mA\K})^2}{q^2}\right).
\end{array}
\end{equation*}
This ensures the sketched solution $\hat{\mathbf{x}}$ achieves \@ifstar\@eqref\@@eqref{eps_distort} with probability exceeding $1-\delta$ and concludes our proof.
\end{proof}

\section{Bounding the error with Gaussian width}
\label{sec: 4}
In this section, we give the proof of \cref{sup_error} and validate that the Gaussian width is the correct measure to bound the tensorized sub-Gaussian processes approximation error. The major tool is the standard generic chaining approach \cite{T05}.

\subsection{Proof ingredients}
We first present the crucial supporting proposition and lemmas for \cref{sup_error}. These results own proofs are incorporated in \cref{proof HW,proof 1,proof H,proof Q}.

The cornerstone of \cref{sup_error} is the sparse Hanson-Wright inequality in Theorem 1.1 of \cite{Z19} for a single sub-Gaussian process. We present an extension of the result by averaging the quadratic forms from multiple independent processes. The following lemma shows how the density level $q$ affects the sub-Gaussian tail of the quadratic form.

\begin{proposition}
\label{sparse HW}
There exists a universal constant $c>0$ for which the following holds. Let $n, m\in \mathbb{N}^+$ and $q\in (0,1].$ For any determined squared matrices $\mathbf{A}_k\in \mathbb{R}^{n \times n}$ for $k \in [m]$, draw $m$ independent copies $\mathbf{s}_k$ of $\mathbf{s}=\bm{\phi} \circ \bm{\sigma} \in\mathbb{R}^n,$ where $\bm{\phi}\in\mathbb{R}^n$ has independent zero-mean entries with bounded $\psi_2$ norms, and the vector $\bm{\sigma}\sim \text{Ber}(q)^n$. Then for every $t>0,$
\begin{equation}
\label{eqn: sparse HW}
\mathbf{Pr}\left(\left\vert \frac{1}{m}\, \sum_{k=1}^m \mathbf{s}_k^\top\, \mathbf{A}_k\, \mathbf{s}_k - \mathbb{E} \, \mathbf{s}_k^\top\, \mathbf{A}_k\, \mathbf{s}_k\right\vert >t\right) \leq 2\,{\text{exp}} \left(-c\, m\, \min \left(\frac{t}{K_1\, \beta^2}, \frac{t^2}{K_2^2\, \beta^4\, q}\right) \right),
\end{equation}
where $\beta>0$ is the maximal $\psi_2$ norm of entries in $\bm{\phi},$ $K_1= \max_{k\in[m]} \|\mathbf{A}_k\|_2$ and $K_2 = \max_{k\in[m]} \|\mathbf{A}_k\|_F.$
\end{proposition}

Without the loss of generality, we assume the fixed set $\bm{\mathcal{Y}} = \overline{\bm{\mathcal{Y}}}$ is on the unit sphere $\mathcal{S}^{n_1n_2-1}$ below and in the proof of \cref{sup_error} (\cref{sup_error proof}). For any vector $\mathbf{y}\in\bm{\mathcal{Y}},$ recalling the sketching matrix $\mathbf{S}$ defined in \@ifstar\@eqref\@@eqref{sketch}-\@ifstar\@eqref\@@eqref{sketch ii}, we focus on two related random variables
\begin{equation}
\label{HQ}
\begin{array}{l}
H(\mathbf{y}) = \displaystyle\left(\frac{1}{m} \sum_{k=1}^{m} \langle \bm{\eta}_k\otimes {\bm{\xi}}_k,\mathbf{y} \rangle^2\right)^{1/2} = \|\mathbf{S}\mathbf{y}\|_2,\\\\
Q(\mathbf{y} )=\displaystyle \frac{1}{m} \sum_{k=1}^{m} \langle \bm{\eta}_k\otimes {\bm{\xi}}_k, \mathbf{y} \rangle^2 - \|\mathbf{y}\|_2^2 = \|\mathbf{S}\mathbf{y}\|_2^2 -\|\mathbf{y}\|_2^2.
\end{array}
\end{equation}

We then introduce the basic estimates of $H, Q$ by applying \cref{sparse HW}. The concentration results will be used in the chaining procedure.
\begin{lemma}
\label{1}
There exists a universal constant $c>0$ for which the following holds. Let $n_1, n_2, m \in \mathbb{N}^+$ and $q\in(0,1].$ For every $\mathbf{y}, \mathbf{y}' \in \bm{\mathcal{Y}} \subset \mathcal{S}^{n_1n_2-1} (\mathbf{y}\neq \mathbf{y}'),$ consider $H, Q$ defined in \@ifstar\@eqref\@@eqref{HQ} with $\alpha\geq1$ and $\mathcal{M}\geq 0$ given in \cref{sup_error}.
\begin{enumerate}
\item
For every $u>0,$
\begin{equation}
\label{pr_Q}
\mathbf{Pr}\left(|Q(\mathbf{y}) | \geq u\, \alpha^2\, (\mathcal{M}^2+1)\right)\leq 4\, {\text{exp}}\left(-c\, m \, \min \left(u\, q, u^2\, q\right)\right);
\end{equation}

\item
for every $u >0,$
\begin{equation}
\label{pr_Q_diff}
\mathbf{Pr}\left(|Q(\mathbf{y}) - Q(\mathbf{y}')| \geq u\, \alpha^2\, (\mathcal{M}^2+1)\, \|\mathbf{y}-\mathbf{y}'\|_2\right)\leq 4\, {\text{exp}}\left(-c\, m \, \min \left(\frac{u\, q}{\sqrt{2}}, \frac{u^2\, q}{2}\right)\right);
\end{equation}

\item
for $u \geq \sqrt{1+\frac{1}{\alpha^2}},$
\begin{equation}
\label{pr_H}
\mathbf{Pr}\left(H(\mathbf{y}-\mathbf{y}') \geq u\, \alpha\, \sqrt{\mathcal{M}^2+1} \, \|\mathbf{y}-\mathbf{y}'\|_2 \right) \leq 4\, {\text{exp}}\left(-c\, m \, \frac{u^2\, q}{1+\frac{1}{\alpha^2}}\right).
\end{equation}
\end{enumerate}
\end{lemma}

We apply the generic chaining technique \cite{T05} and follow the detailed illustrations in \cite{MPT07}. \cref{H} proves that one can project the whole set $\bm{\mathcal{Y}}$ to a subset $\bm{\mathcal{Y}}'$ of restricted cardinality and maintain a small projection error with high probability. The result depends on a fine construction of a subset $\bm{\mathcal{Y}}'.$ \cref{Q} bounds the supremum embedding error by $\mathbf{S}$ over such subset.

\begin{lemma}
\label{H}
There exists a universal constant $C'>0$ for which the following holds. Follow the same set-up in \cref{1}. There is a subset $\bm{\mathcal{Y}}' \subset \bm{\mathcal{Y}} \subset \mathcal{S}^{n_1n_2-1}$ such that $|\bm{\mathcal{Y}}'| \leq 4^m$ and with probability at least $1-{\text{exp}} (-4\, m),$ we have
\begin{equation}
\sup_{\mathbf{y}\in \bm{\mathcal{Y}}} H(\mathbf{y} - \pi(\mathbf{y})) \leq C'\,\alpha\, \sqrt{\mathcal{M}^2+1}\, \frac{\mathbb{W}(\bm{\mathcal{Y}})}{\sqrt{m\, q}},
\end{equation}
where $\pi(\mathbf{y})$ is a nearest point to $\mathbf{y}$ in $\bm{\mathcal{Y}}'$ under $\|~~\|_2$ metric.
\end{lemma}

\begin{lemma}
\label{Q}
There exists a universal constant $C''>0$ for which the following holds. Follow the same set-up in \cref{1}. Fix any subset $\bm{\mathcal{Y}}'\subset \bm{\mathcal{Y}}\subset \mathcal{S}^{n_1n_2-1}$ such that $|\bm{\mathcal{Y}}'| \leq 4^m.$ Then for every $0<\omega< 1,$
\begin{equation}
\sup_{\mathbf{y}\in\bm{\mathcal{Y}}'} |Q(\mathbf{y})| \leq \alpha^2\, (\mathcal{M}^2+1)\, \left(C''\, \frac{\mathbb{W}(\bm{\mathcal{Y}})}{\sqrt{m}\, q} + \omega \right)
\end{equation}
holds with probability at least $1- 3\, {\text{exp}}\left(-c\, m\, \frac{\omega^2\, q}{2}\right).$
\end{lemma}

\subsection{Proof of \cref{sup_error}}
\label{sup_error proof}
\begin{proof}
Without the loss of generality, we assume $\bm{\mathcal{Y}} = \overline{\bm{\mathcal{Y}}}$ is on the unit sphere $\mathcal{S}^{n_1n_2-1}.$ Let $\bm{\mathcal{Y}}'$ be one subset of $\bm{\mathcal{Y}}$ that satisfies the condition stated in \cref{H}. We denote a projection by $\pi: \bm{\mathcal{Y}} \to \bm{\mathcal{Y}}'$, which maps the points in $\bm{\mathcal{Y}}$ to their nearest points in $\bm{\mathcal{Y}}'$ respectively.

The goal of \cref{sup_error} is to estimate $\sup_{\mathbf{y} \in \bm{\mathcal{Y}}} \left\vert\|\mathbf{S}\mathbf{y}\|_2^2 - \|\mathbf{y}\|_2^2\right\vert=\sup_{\mathbf{y}\in\bm{\mathcal{Y}}}|Q(\mathbf{y})|$. Notice that $Q(\mathbf{y})$ and $H(\mathbf{y})$ defined in \@ifstar\@eqref\@@eqref{HQ}, we rewrite $Q(\mathbf{y})$ in the following form:
\begin{equation}
\label{Q incomplete}
\begin{aligned}
Q(\mathbf{y}) &= Q(\pi(\mathbf{y})) + Q(\mathbf{y}-\pi(\mathbf{y})) + 2\langle \mathbf{S} (\mathbf{y}-\pi(\mathbf{y})),\mathbf{S}\, \pi(\mathbf{y}) \rangle - 2\langle \mathbf{y}-\pi(\mathbf{y}), \pi(\mathbf{y})\rangle 	 \\
&= Q(\pi(\mathbf{y})) + H( \mathbf{y}-\pi(\mathbf{y}))^2 - \| \mathbf{y}-\pi(\mathbf{y})\|_2^2 + 2\langle \mathbf{S} (\mathbf{y}-\pi(\mathbf{y})),\mathbf{S} \,\pi(\mathbf{y}) \rangle - 2\langle \mathbf{y}-\pi(\mathbf{y}), \pi(\mathbf{y})\rangle \\
& = Q(\pi(\mathbf{y})) + H( \mathbf{y}-\pi(\mathbf{y}))^2 + 2\langle \mathbf{S} (\mathbf{y}-\pi(\mathbf{y})),\mathbf{S} \,\pi(\mathbf{y}) \rangle  - \| \mathbf{y}-\pi(\mathbf{y})\|_2^2 - 2\langle \mathbf{y}-\pi(\mathbf{y}), \pi(\mathbf{y})\rangle\\
\end{aligned}
\end{equation}
Since both $\mathbf{y},\pi(\mathbf{y})$ are on the unit sphere,
\[
- \| \mathbf{y}-\pi(\mathbf{y})\|_2^2 - 2\langle \mathbf{y}-\pi(\mathbf{y}), \pi(\mathbf{y})\rangle = - (\| \mathbf{y} \|_2^2 - \| \pi(\mathbf{y})\|_2^2) = 0,
\]
continue from \@ifstar\@eqref\@@eqref{Q incomplete},
\[
 Q(\mathbf{y}) = Q(\pi(\mathbf{y})) + H( \mathbf{y}-\pi(\mathbf{y}))^2 + 2\langle \mathbf{S} (\mathbf{y}-\pi(\mathbf{y})),\mathbf{S} \,\pi(\mathbf{y}) \rangle.
 \]
We then apply triangles inequality and H\"older's inequality to derive the following,
\begin{equation*}
|Q(\mathbf{y})| \leq |Q(\pi(\mathbf{y}))| + H(\mathbf{y}-\pi(\mathbf{y}))^2 + 2\,H(\mathbf{y}-\pi(\mathbf{y}))\, H(\pi(\mathbf{y})).
\end{equation*}
Taking supremum over $\bm{\mathcal{Y}}$ for both sides above and notice that $\pi(\mathbf{y}) \in \bm{\mathcal{Y}}'$, we have
\begin{equation}
\label{goal bound}
\sup_{\mathbf{y}\in\bm{\mathcal{Y}}}|Q(\mathbf{y})| \leq \sup_{\mathbf{y}'\in\bm{\mathcal{Y}}'}|Q(\mathbf{y}')| + \sup_{\mathbf{y}\in\bm{\mathcal{Y}}} H(\mathbf{y}-\pi(\mathbf{y}))^2 + 2\, \left(\sup_{\mathbf{y}\in\bm{\mathcal{Y}}} H(\mathbf{y}-\pi(\mathbf{y}))\right)\, \left(\sup_{\mathbf{y}'\in\bm{\mathcal{Y}}'} H(\mathbf{y}')\right).
\end{equation}
The inequality \@ifstar\@eqref\@@eqref{goal bound} indicates that $\sup_{\mathbf{y}\in\bm{\mathcal{Y}}}|Q(\mathbf{y})|$ is bounded by three components. In the following, we handle the first term $\sup_{\mathbf{y}'\in \bm{\mathcal{Y}}'}|Q(\mathbf{y}')|$ by \cref{Q} and the second by \cref{H}. The third term is controlled simultaneously since each factor can be bounded from the first and second components respectively.

\begin{enumerate}
\item
Fix $\omega \in (0,\frac{1}{2}]$ stated in \cref{sup_error}, since $|\bm{\mathcal{Y}}'|\leq 4^m,$ we apply \cref{Q}. Then with probability at least $1- 3\, {\text{exp}}\left(-c\, m\, \frac{\omega^2\, q}{2}\right),$ one has
\begin{equation}
\label{event2}
\sup_{\mathbf{y}'\in\bm{\mathcal{Y}}'}|Q(\mathbf{y}')| \leq \alpha^2\, (\mathcal{M}^2+1) \, \left(C''\, \frac{\mathbb{W}(\bm{\mathcal{Y}})}{\sqrt{m}\, q} + \omega \right).
\end{equation}

Moreover, if we assume the integer $m \geq \frac{4C''^2\, \mathbb{W}(\bm{\mathcal{Y}})^2}{q^2},$ then
\begin{equation*}
C''\, \frac{\mathbb{W}(\bm{\mathcal{Y}})}{\sqrt{m}\,q}+\omega \leq 1.
\end{equation*}
This further implies that
\begin{equation*}
\displaystyle \sup_{\mathbf{y}'\in\bm{\mathcal{Y}}'} H(\mathbf{y}')^2  \leq  \displaystyle \sup_{\mathbf{y}'\in\bm{\mathcal{Y}}'}|Q(\mathbf{y}')| + \|\mathbf{y}'\|_2^2 \leq \displaystyle\alpha^2\,  (\mathcal{M}^2+1) + 1\leq 2\, \alpha^2\,  (\mathcal{M}^2+1).
\end{equation*}
where we use \@ifstar\@eqref\@@eqref{event2} and $\|\mathbf{y}'\|_2 =1$ in the second inequality above. The last inequality holds because $\alpha \geq1$ \footref{alpha}.
Consequently,
\begin{equation}
\label{event2then}
\sup_{\mathbf{y}'\in\bm{\mathcal{Y}}'} H(\mathbf{y}') \leq \sqrt{2}\, \alpha\,  \sqrt{\mathcal{M}^2+1}.
\end{equation}

\item
To deal with the second term in \@ifstar\@eqref\@@eqref{goal bound}, since we define the subset $\bm{\mathcal{Y}}'$ to satisfy the condition in \cref{H}, we apply \cref{H},
\begin{equation}
\label{event11}
\sup_{\mathbf{y}\in\bm{\mathcal{Y}}} H(\mathbf{y}-\pi(\mathbf{y})) \leq \alpha\,  \sqrt{\mathcal{M}^2+1}\, C'\, \frac{\mathbb{W}(\bm{\mathcal{Y}})}{\sqrt{m\,q}},
\end{equation}
or equivalently,
\begin{equation}
\label{event12half}
\sup_{\mathbf{y}\in\bm{\mathcal{Y}}} H(\mathbf{y}-\pi(\mathbf{y}))^2 \leq \alpha^2\, (\mathcal{M}^2+1)\,\left(C'\, \frac{\mathbb{W}(\bm{\mathcal{Y}})}{\sqrt{m\,q}}\right)^2,
\end{equation}
achieves with probability at least $1- {\text{exp}}(-4\, m).$

We further assume that $m \geq \frac{C'^2\, \mathbb{W}(\bm{\mathcal{Y}})^2}{q},$ then $C'\, \frac{\mathbb{W}(\bm{\mathcal{Y}})}{\sqrt{m\, q}} \leq 1.$ And \@ifstar\@eqref\@@eqref{event12half} implies that
\begin{equation}
\label{event12}
\sup_{\mathbf{y}\in\bm{\mathcal{Y}}} H(\mathbf{y}-\pi(\mathbf{y}))^2 \leq \alpha^2\, (\mathcal{M}^2+1)\, C'\, \frac{\mathbb{W}(\bm{\mathcal{Y}})}{\sqrt{m\, q}}.
\end{equation}
\end{enumerate}

We now plug all the bounds in \@ifstar\@eqref\@@eqref{event2}, \@ifstar\@eqref\@@eqref{event12}, \@ifstar\@eqref\@@eqref{event11} and \@ifstar\@eqref\@@eqref{event2then} into the partition form in \@ifstar\@eqref\@@eqref{goal bound}. Consequently, the following event
\begin{equation}
\label{C',C''}
\sup_{\mathbf{y}\in\bm{\mathcal{Y}}}|Q(\mathbf{y})| \leq \alpha^2\, (\mathcal{M}^2+1)\,\left[(\sqrt{q}\, C' + \sqrt{2\, q}\, C' + C'')\, \frac{\mathbb{W}(\bm{\mathcal{Y}})}{\sqrt{m}\,q} +\omega\right]
\end{equation}
holds with failure probability no more than ${\text{exp}}(-4\, m) + 3\, {\text{exp}}(-c\, m\, \frac{\omega^2\, q}{2}).$

Notice that $q\in (0,1]$ and the constant $c\leq \frac{1}{4e}$ as shown in \cite{Z19}, it follows that $0 < \omega \leq \frac{1}{2}\leq \sqrt{\frac{8}{c\, q}}$. This further implies that ${\text{exp}}(-4\, m) \leq {\text{exp}}(-c\, m\, \frac{\omega^2\, q}{2}),$ so the failure probability can be upper bounded by $4\, {\text{exp}}(-c\, m\,\frac{\omega^2\, q}{2}) $.

To conclude, given the assumption
\begin{equation*}
m \geq\displaystyle  \frac{C_2\, \mathbb{W}(\bm{\mathcal{Y}})^2}{q^2} \geq \max(\frac{C'^2}{q}, \frac{4\,C''^2}{q^2})\, \mathbb{W}(\bm{\mathcal{Y}})^2,
\end{equation*}
then from \@ifstar\@eqref\@@eqref{C',C''},
\begin{equation*}
\sup_{\mathbf{y} \in \bm{\mathcal{Y}}} \left\vert\|\mathbf{S}\mathbf{y}\|_2^2 - \|\mathbf{y}\|_2^2\right\vert =\sup_{\mathbf{y}\in\bm{\mathcal{Y}}}|Q(\mathbf{y})| \leq \alpha^2\, (\mathcal{M}^2+1)\, \left(C_1\, \frac{\mathbb{W}(\bm{\mathcal{Y}})}{\sqrt{m}\, q} +\omega\right)
\end{equation*}
holds with probability no less than $1-4\, {\text{exp}}(-c\, m\,  \frac{\omega^2\, q}{2}),$ where the universal constants $C_1, C_2$ are set to satisfy
\begin{equation*}
\left\{
\begin{array}{l}
\displaystyle C_1 \geq \sqrt{q}\, C' + \sqrt{2\, q} \,C' + C'' ,\\\\
\displaystyle C_2 \geq   \max(C'^2\, q, 4\,C''^2).
\end{array}
\right.
\end{equation*}
The proof of \cref{sup_error} is complete.
\end{proof}

\section{Numerical experiments}
\label{sec: numerics}
In this section, we demonstrate the numerical performance for the row-wise tensor sketching matrix $\mathbf{S}\in\mathbb{R}^{m \times n_1n_2}$ defined in \@ifstar\@eqref\@@eqref{sketch}-\@ifstar\@eqref\@@eqref{sketch ii}. Specifically, we choose a common case of $\mathbf{S}$ to be: Gaussian+Rademacher (G+R), meaning the tensor component $\bm{\phi}^{(1)}\in\mathbb{R}^{n_1}$ in \@ifstar\@eqref\@@eqref{sketch ii} follows the standard normal distribution $\mathcal{N}(\mathbf{0}, \mathbf{I}_{n_1}),$ while the other $\bm{\phi}^{(2)}\in\{-1,1\}^{n_2}$ is a Rademacher vector whose entries are i.i.d. and take the values $1$ and $-1$ with equal probability $1/2.$

We implement the sketch $\mathbf{S}$ on the unconstrained linear regression
\begin{equation}
\label{unconstrained_numerics}
\mathbf{x}^* = \arg\min_{\mathbf{x}\in\mathbb{R}^p}\|\mathbf{A}\mathbf{x}-\mathbf{b}\|_2^2.
\end{equation}
We focus on three types of linear regressions where the data matrix $\mathbf{A}\in\mathbb{R}^{n_1n_2\times p}$ has different properties:
\begin{enumerate}
\item {\bf{Well-conditioned program.}}\\
In this setting, the data matrix $\mathbf{A}$ has a relatively small condition number.

\item {\textbf{Ill-conditioned program.}}\\
For the ill-conditioned problem, we build the condition number of $\mathbf{A}$ to be large, for instance, $10^4.$

\item {\bf{Structured program.}}\\
As we mentioned in \cref{design}, in the structured program, the columns of the matrix $\mathbf{A}$ admit tensor structure:
\begin{equation}
\label{structured}
\mathbf{a}_j = \mathbf{f}_j\otimes \mathbf{g}_j, \quad\text{for~} j \in [p].
\end{equation}
\end{enumerate}

In terms of building the input system, we generate the matrix $\mathbf{A}$ based on the singular value decomposition:
\begin{equation*}
\underset{n_1n_2\times p}{\mathbf{A}}\quad = \underset{n_1n_2\times p}{\mathbf{U}}\quad\underset{p\times p}{\bm{\Sigma}}\quad\underset{p \times p}{\mathbf{V}^\top},
\end{equation*}
where the left and right factors $\mathbf{U}, \mathbf{V}$ are built from the orthogonal matrices $\mathbf{Q}$ in QR decompositions of standard Gaussian matrices, whose entries are i.i.d. and normally distributed with zero mean and unit variance.
For a well-conditioned matrix $\mathbf{A}$ with a small condition number, the singular values are drawn from normal distribution $\mathcal{N}(1, 0.04)$ and thus are centered around $1$ with high probability. For a ill-conditioned matrix $\mathbf{A},$ we set the singular values to be
$$\displaystyle \bm{\Sigma}_{j,j} = 10^{(-4)\, \frac{j-1}{p-1}}, \quad \text{for~} j \in [p].$$
The condition number of $\mathbf{A}$ is thus $\bm{\Sigma}_{1,1}/\bm{\Sigma}_{p,p} = 10^4.$
For a structured matrix $\mathbf{A},$ we instead form two factor matrices $\mathbf{F} = [\mathbf{f}_1, \dots, \mathbf{f}_p] \in \mathbb{R}^{n_1 \times p}$ and $\mathbf{G} = [\mathbf{g}_1, \dots, \mathbf{g}_p] \in \mathbb{R}^{n_2 \times p}$ respectively, following the way of generating a well-conditioned matrix and then construct $\mathbf{A}$ by the rule \@ifstar\@eqref\@@eqref{structured}.
To form the input vector $\mathbf{b}\in\mathbb{R}^{n_1n_2},$ first we generate a reference vector $\mathbf{x}_{\text{ref}}\in\mathbb{R}^p$ following the distribution $\mathcal{N}(\mathbf{1},0.25\, \mathbf{I}_p)$ and a small noise vector following $\mathcal{N}(\mathbf{0},10^{-2}\, \mathbf{I}_{n_1n_2}).$ We build the data vector $\mathbf{b}$ as
\[
\mathbf{b} = \mathbf{A}\mathbf{x}_{\text{ref}}+\text{noise}.
\]

In the numerical experiments, we perform the sketching construction $\mathbf{S}$ on the aforementioned three types of linear regressions.
We then compare the performance of $\mathbf{S}$ with that of the standard Gaussian matrix. We denote $\mathbf{x}^*\in\mathbb{R}^{p}$ as \@ifstar\@eqref\@@eqref{unconstrained_numerics}, and $\hat{\mathbf{x}}\in\mathbb{R}^{p}$ the sketched solution
$
\hat{\mathbf{x}} = \arg\min_{\mathbf{x}\in\mathbb{R}^p}\|\mathbf{S}\mathbf{A}\mathbf{x}-\mathbf{S}\mathbf{b}\|_2^2.
$
To measure the quality of a sketched solution $\hat{\mathbf{x}}$, we define
\begin{equation}
\label{error ratio}
\text{error~ratio} = \left\vert\frac{\|\mathbf{A}\hat{\mathbf{x}}-\mathbf{b}\|_2^2-\|\mathbf{A}\mathbf{x}^*-\mathbf{b}\|_2^2}{\|\mathbf{A}\mathbf{x}^*-\mathbf{b}\|_2^2}\right\vert
\end{equation}
For all numerical tests below, we calculate the error ratio as the average of $100$ independent simulations.

Notice that \@ifstar\@eqref\@@eqref{eps_distort} in \cref{sketch_dimension} implies that the above error ratio is bounded by $\varepsilon^2+2\varepsilon$ with the parameter $\varepsilon$ used in the theoretical results, indicating that
\begin{equation}
\label{ratio_eps}
\text{error~ratio} = \left\{
\begin{array}{ll}
O(\varepsilon),& \varepsilon~\text{and~error~ratio~are~relatively~small},\\
O(\varepsilon^2),&\varepsilon~\text{and~error~ratio~are~relatively~large}.
\end{array}
\right.
\end{equation}

In \cref{sketch_size,rank,density}, we plot the error ratios respectively depending on different choices of the parameters:
\begin{enumerate}
\item
{\bf the sketching dimension $m,$}
\item
{\bf the number of unknown variables $p,$ }
\item
{\bf the density level $q.$}
\end{enumerate}

The plots are all in the log-log form.

\subsection{Dependence on the sketching size $m$}
It is of interest to test how the sketching dimension $m$ affects the quality of our sketching matrix $\mathbf{S}.$ \cref{sketch_size} presents the empirical performance of $\mathbf{S}$ as the sketching dimension $m$ varies. We summarize the numerical observations as follows:
\begin{enumerate}
\item
In the small error regime (under the scale between $10^0$ and $10^{-1}$), the roughly parallel lines validate the theory that the G+R sketch has the same performance as that of the standard Gaussian sketch.
\item
In contrast, there are rapid drops in the large error ratio regime for the G+R sketching in both well and ill-conditioned programs. Such phenomena can be explained by the quantitative shift of the dependence on $\varepsilon$ shown in \@ifstar\@eqref\@@eqref{ratio_eps}. More specifically, the error ratio decays at the order $\mathcal{O}(\varepsilon^2)$ when the sketching size $m$ is small but gradually drops at a slower rate $\mathcal{O}(\varepsilon)$ as $m$ grows.
\item
Unlike its performance in the unstructured programs, the G+R sketching does not have a drastic decay in the structured program. This may suggest a better estimate of the sketching dimension for tensor-structured programs.
\end{enumerate}

\begin{figure}[!ht]
\centering
\includegraphics[width=0.55\linewidth]{sketch_size_log.eps}
\caption{We plot in solid curves the error ratios averaged over 100 trials of the G+R sketching with density $q=0.2$, respectively on the well-conditioned, ill-conditioned and structured linear regressions with varying sketching size $m$. In comparison, the performance of unstructured Gaussian sketching matrix is plotted in dash lines on the same three types of regression problems for reference. The size of the regression problems is set as $n_1n_2 = 64^2$ by $p=15$. }
\label{sketch_size}
\end{figure}

\subsection{Dependence on the number of unknowns $p$}
The matrix $\mathbf{A}\in\mathbb{R}^{n_1n_2\times p}$ is full rank with high probability due to our construction method, thus the number of unknowns $p=\text{rank}(\mathbf{A})$ considering $n_1,n_2\geq p$. We show in \cref{rank} the performance of the sketching strategy as the number of unknowns $p$ changes.
\begin{enumerate}
\item
When the error ratios are relatively small, i.e. under the scale $10^{-1},$ the G+R construction for all three types of programs is shown to have the same dependence as the standard Gaussian matrix on $p,$ or equivalently $\text{rank}(\mathbf{A}),$ given that the slopes are rather similar. This observation is important and provides numerical support for our main results \cref{sketch_dimension} and \cref{unconstrained}, that the tensor-structured sketching maintains the same optimal dependence on $\text{rank}(\mathbf{A})$ as conventional Gaussian matrix in unconstrained linear regressions.

\item
There are rises in the later part of the solid curves for well and ill-conditioned programs when the errors are getting bigger. The reason is still the quantitive change of the error ratio, see \@ifstar\@eqref\@@eqref{ratio_eps}.
\end{enumerate}

\begin{figure}[!ht]
\centering
\includegraphics[width=0.55\linewidth]{rank_log.eps}
\caption{We plot the sketching error ratios with varying number of unknown variables $p$ with fixed ambient dimension $n_1n_2=64^2$ for three different types of linear regression programs. We fix the sketching dimension $m=400$. We compare the performances of the G+R sketching constructed with density $q=0.2$ and standard Gaussian sketching.}
\label{rank}
\end{figure}

\subsection{Dependence on the density level $q$}
\cref{density} clearly shows that the sketching accuracy improves when the density level $q$ in the construction of $\mathbf{S}$ increases.
\begin{enumerate}
\item
Again, abrupt drops appear in the beginning of the plots, which can be explained by the quantitative change in the error ratio, see \@ifstar\@eqref\@@eqref{ratio_eps}.
In the later part of the plots for well and ill-conditioned programs, i.e. $q \in [0.2, 0.75]$, the two solid curves overlap and smoothly decay at the same rate. Such decay is supported by the second term in our theoretical bound \@ifstar\@eqref\@@eqref{sketch_dimension_term} when $q$ is relatively large, implying that $\varepsilon \sim \frac{1}{\sqrt{q}}$ if $m$ is fixed. In fact, a linear fitting calculation suggests that the two solid curves decay at the slope $-0.62,$ which roughly agrees with the theoretical prediction slope $-0.5.$

\item
When the density level is greater than $0.4$, the performances of the G+R sketchings are nearly the same on all three different regression problems. Moreover, the plots suggest that for density level $q\geq 0.6$, a sparse tensor sketching strategy may achieve the same performance of unstructured dense Gaussians.
\end{enumerate}

\begin{figure}[!ht]
\centering
\includegraphics[width=0.55\linewidth]{density_log.eps}
\caption{We compare the performance of the G+R sketching strategy with varying density level $q$ and  the fixed sketching dimension $m=400$ on three different regression problems. The size of the regressions is set as $n_1n_2 = 64^2$ by $p=15$. We add the standard dense Gaussian matrices on the same regression problems for reference. }
\label{density}
\end{figure}

\subsection{Bounded vs unbounded sketches}
\label{BN}
The theoretical result \@ifstar\@eqref\@@eqref{error} in \cref{sup_error} indicates that the embedding error contains a factor $(\mathcal{M}^2 +1)$, which depends on the maximal magnitude in one tensor component of the sketch. This is a consequence of our assumption in the sketching matrix \@ifstar\@eqref\@@eqref{sketch}-\@ifstar\@eqref\@@eqref{sketch ii} that one of the tensor factors is uniformly bounded. In \cref{GR}, we investigate if the bounded condition has an impact on the sketching performance. In the experiment set-up, besides for the G+R construction, we also use the Gaussian+Gaussian (G+G) and Rademacher+Rademacher (R+R) as the examples for unbounded and bounded sketches. They both fit into the row-wise tensor sketch framework.

In \cref{GR}, for the well-conditioned linear regression only, we test three sketching strategies: G+R, G+G and R+R. All three curves seem to be parallel to that of the standard Gaussian and Rademacher sketching matrices. We can clearly see the impact of the constant factor $(\mathcal{M}^2 +1)$ in these three curves: the R+R curve has smaller error ratios because it has bounded magnitude in the tensor factors of the sketching matrix. We thus conclude that the magnitude in the tensor-structured sketch does affect the sketching quality and the bounded assumption in the theory statement is necessary.
\begin{figure}[!ht]
\centering
\includegraphics[width=0.55\linewidth]{GR.eps}
\caption{We compare the sketching performances of the G+R, G+G and R+R sketchings with varying sketching size $m$ on a well-conditioned linear regression problem of size $n_1n_2 = 64^2$ by $p=15.$ All three types of random sketches are constructed with density level $q=0.2.$ In addition, we implement the standard Gaussian and Rademacher matrices on the same program for reference.}
\label{GR}
\end{figure}

\section{Conclusions and future work}
\label{sec: conclusion}
In this paper, motived by structured least squares in practical applications, we presented a row-wise tensor-structured sketching design to accelerate the solving procedure.
We have demonstrated a state-of-the-art estimate for the sketching dimension of the proposed sketch in terms of the Gaussian width. The result directly applies to common types of optimizations in different geometry landscapes.
In light of the theoretical support to the main result, we developed the analysis for the maximum embedding error of the proposed sketching matrix over a fixed set, with the help of the generic chaining technique.
We ran numerical simulations and showed that the empirical results are consistent with the theory.

Future work remains, first, extending the current result in the order-$2$ case to higher order tensor structure.
Secondly, the sketching model in this work considers only the sub-Gaussian random variables, an interesting open problem would be analyzing the sketching property for the randomized FFT-related construction with tensor structure, as such class of sketch enables fast matrix-vector multiplications.
The last open problem is inspired by the numerical observations shown in \cref{sketch_size,rank,density}, that the proposed sketching design works better on the tensor-structured programs than unstructured programs. This may suggest that a tighter sketching size estimate could be derived for tensor-structured programs.

\section*{Acknowledgements}
The authors would like to thank Rachel Ward for her insights and helpful discussions. Chen is supported by the Office of Naval Research (award N00014-18-1-2354) and by the National Science Foundation (awards 1952735 and 1620472).

\bibliographystyle{plain}

Optimization and Control (math.OC); Numerical Analysis (math.NA); Probability (math.PR)