New approximations for the cone of copositive matrices and its dual

Abstract: We provide convergent hierarchies for the cone C of copositive matrices and its dual, the cone of completely positive matrices. In both cases the corresponding hierarchy consists of nested spectrahedra and provide outer (resp. inner) approximations for C (resp. for its dual), thus complementing previous inner (resp. outer) approximations for C (for the dual). In particular, both inner and outer approximations have a very simple interpretation. Finally, extension to K-copositivity and K-complete positivity for a closed convex cone K, is straightforward.

\section{Introduction}

In recent years the convex cone $\mathcal{C}$ of {\it copositive} matrices
and its dual cone $\mathcal{C}^*$ of {\it completely positive} matrices
have attracted a lot of attention,
in part because several interesting NP-hard problems can be modelled as convex conic optimization
problems over those cones. For a survey of results and discussion on $\mathcal{C}$ and its dual, the interested reader is referred to e.g. Anstreicher and Burer \cite{anstreicher}, Bomze \cite{bomze1}, Bomze et al. \cite{bomze2},
Burer \cite{burer}, D\"ur \cite{durr} and Hiriart-Urruty and Seeger \cite{jbhu-sirev}..

As optimizing over $\mathcal{C}$ (or its dual) is in general difficult, a
typical approach is to optimize over simpler and more tractable cones. In particular,
nested hierarchies of tractable convex cones $\mathcal{C}_k$, $k\in\mathbb{N}$, that provide {\it inner} approximations of $\mathcal{C}$ have been proposed, notably by Parrilo \cite{parrilo}, deKlerk and Pasechnik \cite{de-pasech}, Bomze and deKlerk \cite{bomze}, as well
as Pe\~na et al. \cite{pena}. For example, denoting by
$\mathcal{N}$ (resp. $\mathcal{S}_+$) the convex cone of nonnegative (resp. positive semidefinite) matrices,
the first cone in the hierarchy of \cite{de-pasech} is $\mathcal{N}$, and $\mathcal{N}+\mathcal{S}_+$ in that of \cite{parrilo},
whereas the hierarchy of \cite{pena} is in sandwich between that of \cite{de-pasech} and \cite{parrilo}.
Of course, to each such hierarchy of inner approximations $(\mathcal{C}_k)$, $k\in\mathbb{N}$,
of $\mathcal{C}$, one may associate the hierarchy $(\mathcal{C}^*_k)$, $k\in\mathbb{N}$, of dual cones
which provides outer approximations of $\mathcal{C}^*$. However, quoting D\"ur in \cite{durr}: ``We are not aware of comparable approximation schemes that approximate
the completely positive cone (i.e. $\mathcal{C}^*$) from the interior."

Let us also mention Gaddum's characterization
of copositive matrices described in Gaddum \cite{gaddum} for which the set membership problem ``$\mathbf{A}\in\mathcal{C}$" (with $\mathbf{A}$ known)
reduces to solving a linear programming (LP) problem (whose size is not polynomially bounded in the input size of the matrix $\mathbf{A}$).
However, Gaddum's characterization is {\it not} convex in the coefficients of the matrix $\mathbf{A}$, and so
cannot be used to provide a hierarchy of outer approximations of $\mathcal{C}$. On the other hand,
de Klerk and Pasechnik \cite{de-pasek} have used Gaddum's characterization to help solve
quadratic optimization problems on the simplex
via solving a hierarchy of LP-relaxations of increasing size and with {\it finite} convergence of the process.

{\bf Contribution.} The contribution of this note is precisely to describe an explicit hierarchy of tractable convex cones that provide
{\it outer} approximations of $\mathcal{C}$ which converge monotonically and asymptotically to $\mathcal{C}$. And so, by duality, the corresponding hierarchy of dual cones
provides {\it inner} approximations of $\mathcal{C}^*$ converging to $\mathcal{C}^*$, answering D\"ur's question and also showing  that
$\mathcal{C}$ and $\mathcal{C}^*$ can be sandwiched between two converging hierarchies of tractable convex cones. The present outer approximations are {\it not} polyhedral and hence are different from the outer polyhedral approximations
$\mathcal{O}^n_r$ of $\mathcal{C}$ defined in Yildirim \cite{yildirim}. $\mathcal{O}^n_r$ are based on a certain discretization $\Delta(n,r)$ of the simplex $\Delta_n:=\{\mathbf{x}\in\mathbb{R}^n_+:\,{\rm e}^T \mathbf{x}=1\}$ whose size is parametrized by $r$. All matrices associated with quadratic forms nonnegative on $\Delta(n,r)$ belong to $\mathcal{O}^r_n$ for all $r$, and $\mathcal{C}=\cap_{r=1}^\infty\mathcal{O}^r_n$;
Combining with the inner approximations of de Klerk and Pasechnik
\cite{de-pasech}, the author provides tight bounds on the gap between upper and lower bounds
for quadratic optimization problems;  for more details the interested reader is
referred to Yildirim \cite{yildirim}.

In fact, our result is a specialization of a more general result of \cite{lasserre} about nonnegativity of polynomials
on a closed set $\mathbf{K}\subset\mathbb{R}^n$, in the specific context of quadratic forms and $\mathbf{K}=\mathbb{R}^n_+$.
In such a context, and identifying copositive matrices with
quadratic forms nonnegative on $\mathbf{K}=\mathbb{R}^n_+$, this specialization yields
inner approximations for $\mathcal{C}^*$  with a very simple interpretation directly related to the definition of $\mathcal{C}^*$,
which might be of interest for the community interested in $\mathcal{C}$ and $\mathcal{C}^*$ but
might be ``lost" in the general case treated in \cite{lasserre}, whence the present note.
Finally, following Burer \cite{burer},  and $\mathcal{K}\subset\mathbb{R}^n$ being a closed convex cone,
one may also consider the convex cone $\mathcal{C}_\mathcal{K}$
of $\mathcal{K}$-copositive matrices, i.e., real symmetric matrices $\mathbf{A}$ such that
$\mathbf{x}^T\mathbf{A}\mathbf{x}\geq0$ on $\mathcal{K}$, and its dual cone $\mathcal{C}^*_\mathcal{K}$ of $\mathcal{K}$-completely positive matrices.
Then the outer approximations previously defined have an immediate and straightforward analogue
(as well as the inner approximations of the dual).

\section{Main result}
\subsection{Notation and definitions}
Let $\mathbb{R}[\mathbf{x}]$ be the ring of polynomials in the variables
$\mathbf{x}=(x_1,\ldots,x_n)$.
Denote by $\mathbb{R}[\mathbf{x}]_d\subset\mathbb{R}[\mathbf{x}]$ the vector space of
polynomials of degree at most $d$, which forms a vector space of dimension $s(d)={n+d\choose d}$, with e.g.,
the usual canonical basis $(\mathbf{x}^\alpha)$ of monomials.
Also, let $\mathbb{N}^n_d:=\{\alpha\in\mathbb{N}^n\,:\,\sum_i\alpha_i\leq d\}$ and
denote by $\Sigma[\mathbf{x}]\subset\mathbb{R}[\mathbf{x}]$ (resp. $\Sigma[\mathbf{x}]_d\subset\mathbb{R}[\mathbf{x}]_{2d}$)
the space of sums of squares (s.o.s.) polynomials (resp. s.o.s. polynomials of degree at most $2d$).
If $f\in\mathbb{R}[\mathbf{x}]_d$, write
$f(\mathbf{x})=\sum_{\alpha\in\mathbb{N}^n_d}f_\alpha \mathbf{x}^\alpha$ in the canonical basis and
denote by $\mathbf{f}=(f_\alpha)\in\mathbb{R}^{s(d)}$ its vector of coefficients. Finally, let $\mathcal{S}^n$ denote the space of
$n\times n$ real symmetric matrices, with inner product $\langle \mathbf{A},\mathbf{B}\rangle ={\rm trace}\,\mathbf{A}\mathbf{B}$, and where the notation
$\mathbf{A}\succeq0$ (resp. $\mathbf{A}\succ0$) stands for $\mathbf{A}$ is positive semidefinite.

Given $\mathbf{K}\subseteq\mathbb{R}^n$, denote by ${\rm cl}\,\mathbf{K}$ (resp. ${\rm conv}\,\mathbf{K}$) the closure (resp. the convex hull) of $\mathbf{K}$.
Recall that given a convex cone $\mathbf{K}\subseteq\mathbb{R}^n$, the convex cone
$\mathbf{K}^*=\{\mathbf{y}\in\mathbb{R}^n: \langle\mathbf{y},\mathbf{x}\rangle\geq0\:\forall\mathbf{x}\in\mathbf{K}\}$ is called the {\it dual} cone of $\mathbf{K}$, and satisfies
$(\mathbf{K}^*)^*={\rm cl}\,\mathbf{K}$ . Moreover, given
two convex cones $\mathbf{K}_1,\mathbf{K}_2\subseteq\mathbb{R}^n$,
\[\begin{array}{ccccc}
\mathbf{K}_1^*\,\cap\,\mathbf{K}_2^*&=&(\mathbf{K}_1+\mathbf{K}_2)^*&=&(\mathbf{K}_1\cup\mathbf{K}_2)^*\\
(\mathbf{K}_1\cap\mathbf{K}_2)^*&=&{\rm cl}\,(\mathbf{K}_1^*+\mathbf{K}_2^*)&=&{\rm cl}\,({\rm conv}\,(\mathbf{K}_1^*\cup\mathbf{K}_2^*)\,).\end{array}\]
See for instance Rockafellar \cite[Theorem 3.8; Corollary 16.4.2]{rockafellar}.

\subsection*{Moment matrix} With a sequence $\mathbf{y}=(y_\alpha)$, $\alpha\in\mathbb{N}^n$,
let $L_\mathbf{y}:\mathbb{R}[\mathbf{x}]\to\mathbb{R}$ be the linear functional
\[h\quad (=\sum_{\alpha}h_{\alpha}\,\mathbf{x}^\alpha)\quad\mapsto\quad
L_\mathbf{y}(h)\,=\,\sum_{\alpha}h_{\alpha}\,y_{\alpha},\quad h\in\mathbb{R}[\mathbf{x}].\]
With $d\in\mathbb{N}$, let $\mathbf{M}_d(\mathbf{y})$ be the symmetric matrix with rows and columns indexed
in $\mathbb{N}^n_d$, and defined by:
\begin{equation}
\label{moment}\mathbf{M}_d(\mathbf{y})(\alpha,\beta)\,:=\,L_\mathbf{y}(\mathbf{x}^{\alpha+\beta})\,=\,y_{\alpha+\beta},\quad (\alpha,\beta)\in\mathbb{N}^n_d\times \mathbb{N}^n_d.\end{equation}

The matrix $\mathbf{M}_d(\mathbf{y})$ is called the moment matrix associated with $\mathbf{y}$, and it is straightforward to check that
\[\left[\,L_\mathbf{y}(g^2)\geq0\quad\forall g\in\mathbb{R}[\mathbf{x}]\,\right]\quad\Leftrightarrow\quad\mathbf{M}_d(\mathbf{y})\,\succeq\,0,\quad d=0,1,\ldots.\]

\subsection*{Localizing matrix} Similarly, with $\mathbf{y}=(y_{\alpha})$, $\alpha\in\mathbb{N}^n$,
and $f\in\mathbb{R}[\mathbf{x}]$ written
\[\mathbf{x}\mapsto f(\mathbf{x})\,=\,\sum_{\gamma\in\mathbb{N}^n}f_{\gamma}\,\mathbf{x}^\gamma,\]
let $\mathbf{M}_d(f\,\mathbf{y})$ be the symmetric matrix with rows and columns indexed in $\mathbb{N}^n_d$,
and defined by:
\begin{equation}
\label{localizing}
\mathbf{M}_d(f\,\mathbf{y})(\alpha,\beta)\,:=\,L_\mathbf{y}\left(f(\mathbf{x})\,\mathbf{x}^{\alpha+\beta}\right)\,=\,\sum_{\gamma}f_{\gamma}\,
y_{\alpha+\beta+\gamma},\qquad (\alpha,\beta)\in\mathbb{N}^n_d\times \mathbb{N}^n_d.\end{equation}
The matrix $\mathbf{M}_d(f\,\mathbf{y})$ is called the localizing matrix associated with $\mathbf{y}$ and $f\in\mathbb{R}[\mathbf{x}]$.
Observe that
\begin{equation}
\label{local1}
\langle \mathbf{g},\mathbf{M}_d(f\,\mathbf{y})\mathbf{g}\rangle\,=\,L_\mathbf{y}(g^2\,f),\qquad \forall g\in\mathbb{R}[\mathbf{x}]_d,
\end{equation}
and so if $\mathbf{y}$ has a representing finite Borel measure $\mu
$, i.e., if
\[y_\alpha\,=\,\int_{\mathbb{R}^n} \mathbf{x}^\alpha\,d\mu,\qquad \forall\,\alpha\in\mathbb{N}^n,\]
then (\ref{local1}) reads
\begin{equation}
\label{local2}
\langle \mathbf{g},\mathbf{M}_d(f\,\mathbf{y})\mathbf{g}\rangle\,=\,L_\mathbf{y}(g^2\,f)\,=\,\int_{\mathbb{R}^n} g(\mathbf{x})^2f(\mathbf{x})\,d\mu(\mathbf{x}),\qquad \forall g\in\mathbb{R}[\mathbf{x}]_d.
\end{equation}
Actually, the localizing matrix $\mathbf{M}_d(f\,\mathbf{y})$ is nothing less than the moment matrix associated with the
sequence $\mathbf{z}=f\,\mathbf{y}=(z_\alpha)$, $\alpha\in\mathbb{N}^n$, with $z_\alpha=\sum_\gamma f_\gamma y_{\alpha+\gamma}$.
In particular, if $f$ is nonnegative on the support ${\rm supp}\,\mu$ of $\mu$ then
the localizing matrix $\mathbf{M}_d(f\,\mathbf{y})$ is just the moment matrix associated
with the finite Borel measure $d\mu_f=fd\mu$, absolutely continuous with respect to $\mu$ (denoted $\mu_f\ll\mu$), and with density $f$.
For instance, with $d=1$ one may write
\[\mathbf{M}_1(f\,\mathbf{y})\,=\,
\displaystyle\int_{\mathbb{R}^n}\:\left[\begin{array}{ccc}1&\vert &\mathbf{x}^T\\ -& &-\\ \mathbf{x} &\vert &\mathbf{x}\x^T\end{array}\right]\,f(\mathbf{x})\,d\mu(\mathbf{x})\,=\,
\displaystyle\int_{\mathbb{R}^n}\:\left[\begin{array}{ccc}1&\vert &\mathbf{x}^T\\ -& &-\\ \mathbf{x} &\vert &\mathbf{x}\x^T\end{array}\right]\,d\mu_f(\mathbf{x}),\]
or, equivalently,
\begin{equation}
\label{second}
\mathbf{M}_1(f\,\mathbf{y})\,=\,\mbox{mass($\mu_f$)}\times\,
\displaystyle\left[\begin{array}{ccc}1&\vert &{\rm E}_{\tilde{\mu}_f}(\mathbf{x})^T\\ -& &-\\ {\rm E}_{\tilde{\mu}_f}(\mathbf{x}) &\vert &
{\rm E}_{\tilde{\mu}_f}(\mathbf{x}\x^T)\end{array}\right],\end{equation}
where ${\rm E}_{\tilde{\mu}_f}(\cdot)$ denotes the expectation operator associated with the normalization $\tilde{\mu}_f$ of $\mu_f$
(as a probability measure), and ${\rm E}_{\tilde{\mu}_f}(\mathbf{x}\x^T)$
denotes the matrix of noncentral second-order moments of $\tilde{\mu}_f$
(and the covariance matrix of $\tilde{\mu}_f$ if ${\rm E}_{\tilde{\mu}_f}(\mathbf{x})=0$).

\subsection{Main result}
In \cite[Theorem 3.2]{lasserre} the author has shown in a general context
that a polynomial $f\in\mathbb{R}[\mathbf{x}]$ is nonnegative on a closed set $\mathbf{K}\subseteq\mathbb{R}^n$ if and only if
\begin{equation}
\label{aux1}
\int_\mathbf{K} g^2\,f\,d\mu\,\geq\,0,\qquad\forall g\in\mathbb{R}[\mathbf{x}],\end{equation}
where $\mu$ is a given finite Borel measure with support ${\rm supp}\,\mu$ being exactly $\mathbf{K}$; if $\mathbf{K}$
is compact then $\mu$ is arbitrary whereas if $\mathbf{K}$ is not compact then $\mu$ has to satisfy a
certain growth condition on its moments. If $\mathbf{y}=(y_\alpha)$, $\alpha\in\mathbb{N}^n$,
is the sequence of moments of $\mu$ then  (\ref{aux1}) is in turn equivalent to
\[\mathbf{M}_d(f\,\mathbf{y})\,\succeq\,0,\qquad\forall d=0,1,\ldots,\]
where $\mathbf{M}_d(f\,\mathbf{y})$ is the localizing matrix associated with $f$ and $\mathbf{y}$, defined in (\ref{localizing}).
In this section we particularize this result to the case of copositive matrices viewed
as homogeneous forms of degree $2$, nonnegative on the closed set $\mathbf{K}=\mathbb{R}^n_+$.\\

So with $\mathbf{A}=(a_{ij})\in\mathcal{S}^n$, let denote by $f_\mathbf{A}\in\mathbb{R}[\mathbf{x}]_2$ the
quadratic form $\mathbf{x}\mapsto \mathbf{x}^T\mathbf{A}\mathbf{x}$, and let $\mu$ be the
joint probability measure associated with $n$ {\it i.i.d.} exponential variates (with mean $1$), with support ${\rm supp}\,\mu=\mathbb{R}^n_+$,
and with moments $\mathbf{y}=(y_\alpha)$, $\alpha\in\mathbb{N}^n$, given by:
\begin{equation}
\label{mom}
y_\alpha=\int_{\mathbb{R}^n_+} \mathbf{x}^\alpha\,d\mu(\mathbf{x})\,=\,\int_{\mathbb{R}^n_+} \mathbf{x}^\alpha\,\exp(-\sum_{i=1}^nx_i)\,d\mathbf{x}\,=\,\prod_{i=1}^n\alpha_i{\rm!},\qquad\forall\,\alpha\in\mathbb{N}^n.\\
\end{equation}
Recall that a matrix $\mathbf{A}\in\mathcal{S}^n$ is copositive if $f_\mathbf{A}(\mathbf{x})\geq0$ for all $\mathbf{x}\in\mathbb{R}^n_+$, and denote by $\mathcal{C}\subset\mathcal{S}^n$
the cone of copositive matrices, i.e.,
\begin{equation}
\label{defcop}
\mathcal{C}\,:=\,\{\:\mathbf{A}\in\mathcal{S}^n\::\: f_\mathbf{A}(\mathbf{x})\,\geq\,0\quad \forall\,\mathbf{x}\in\mathbb{R}^n_+\:\}.
\end{equation}
Its dual cone is the closed convex cone of
{\it completely positive}, i.e., matrices of $\mathcal{S}^n$ that can be written as
the sum of finitely many rank-one matrices $\mathbf{x}\x^T$, with $\mathbf{x}\in\mathbb{R}^n_+$, i.e.,
\begin{equation}
\label{cstar}
\mathcal{C}^*\,=\,{\rm conv}\,\{\:\mathbf{x}\x^T\::\:\mathbf{x}\in\mathbb{R}^n_+\:\}.
\end{equation}
Next, introduce the following sets $\mathcal{C}_d\subset\mathcal{S}^n$, $d=0,1,\ldots$, defined by:
\begin{equation}
\label{def-approxd}
\mathcal{C}_d\,:=\,\{\:\mathbf{A}\in\mathcal{S}^n\::\:\mathbf{M}_d(f_\mathbf{A}\,\mathbf{y})\,\succeq\,0\:\},\qquad d=0,1,\ldots
\end{equation}
where $\mathbf{M}_d(f_\mathbf{A}\,\mathbf{y})$ is the localizing matrix defined in (\ref{localizing}), associated with the
quadratic form $f_\mathbf{A}$ and the sequence $\mathbf{y}$ in (\ref{mom}).

Observe that in view of the definition (\ref{localizing}) of the localizing matrix, the entries of
the matrix $\mathbf{M}_d(f_\mathbf{A}\,\mathbf{y})$ are homogeneous and linear in $\mathbf{A}$. Therefore,
the condition $\mathbf{M}_d(f_\mathbf{A}\,\mathbf{y})\succeq0$ is a homogeneous Linear Matrix Inequality (LMI) and defines a closed
convex cone (in fact a spectrahedron) of $\mathcal{S}^n$ (or equivalently, of $\mathbb{R}^{n(n+1)/2}$).
Each $\mathcal{C}_d\subset\mathcal{S}^n$ is a convex cone defined solely in terms of the entries $(a_{ij})$ of $\mathbf{A}\in\mathcal{S}^n$,
and the hierarchy of spectrahedra  $(\mathcal{C}_d)$, $d\in\mathbb{N}$, provides a nested sequence of outer approximations of $\mathcal{C}$.

\begin{thm}
\label{thmain}
Let $\mathbf{y}$ be as in (\ref{mom}) and let $\mathcal{C}_d\subset\mathcal{S}^n$, $d=0,1,\ldots$, be the hierarchy of convex cones defined in
(\ref{def-approxd}).
 Then $\mathcal{C}_0\supset\mathcal{C}_1\cdots\supset\mathcal{C}_d\cdots\supset\mathcal{C}$ and $\mathcal{C}=\displaystyle\bigcap_{d=0}^\infty\mathcal{C}_d$.
\end{thm}
The proof is a direct consequence of \cite[Theorem 3.3]{lasserre} with $\mathbf{K}=\mathbb{R}^n_+$ and $f=f_\mathbf{A}$.
Since $f_\mathbf{A}$ is homogeneous, alternatively one may use the probability measure $\nu$ uniformly supported on the $n$-dimensional simplex $\Delta=\{\mathbf{x}\in\mathbb{R}^n_+:\sum_ix_i\leq1\}$ and invoke \cite[Theorem 3.2]{lasserre}. The moments
$\tilde{\mathbf{y}}=(\tilde{y}_\alpha)$ of $\nu$ are also quite simple to obtain and read:
\begin{equation}
\label{mom-nu}
\tilde{y}_\alpha\,=\,\int_{\Delta}\mathbf{x}^\alpha\,d\mathbf{x}\,=\,\frac{\alpha_1{\rm !}\cdots\alpha_n{\rm !}}{(n+\sum_i\alpha_i){\rm !}},\qquad \forall\alpha\in\mathbb{N}^n.\end{equation}
(See e.g. Grundmann \cite{grundmann}.)

Observe that the set membership problem ``$\mathbf{A}\in\mathcal{C}_d$", i.e., testing whether a given matrix
$\mathbf{A}\in\mathcal{S}^n$ belongs to $\mathcal{C}_d$, is an eigenvalue problem as one has to
check whether the smallest eigenvalue of $\mathbf{M}_d(f_\mathbf{A}\,\mathbf{y})$ is nonnegative. Therefore, instead of using
standard packages for Linear Matrix Inequalities, one  may use powerful specialized softwares for computing eigenvalues
of real symmetric matrices.\\

We next describe an inner approximation of the convex cone
$\mathcal{C}^*$ via the hierarchy of convex cones $(\mathcal{C}_d^*)$, $d\in\mathbb{N}$, where each $\mathcal{C}_d^*$ is the dual cone of $\mathcal{C}_d$ in Theorem \ref{thmain}.

Recall that $\Sigma[\mathbf{x}]_d$ is the space of polynomials that are sums of squares of polynomials of
degree at most $d$. A matrix $\mathbf{A}\in\mathcal{S}^n$ is also identified with a vector $\mathbf{a}\in\mathbb{R}^{n(n+1)/2}$, and conversely,
with any vector $\mathbf{a}\in\mathbb{R}^{n(n+1)/2}$ is associated a matrix $\mathbf{A}\in\mathcal{S}^n$. For instance, with $n=2$,
\begin{equation}
\label{equivalence}
\mathbf{A}\,=\,\left[\begin{array}{cc} a &b\\ b&c\end{array}\right]\quad\leftrightarrow\quad
\mathbf{a}\,=\,\left[\begin{array}{c} a\\ 2b\\ c\end{array}\right].\end{equation}
So we will not distinguish between a convex cone in $\mathbb{R}^{n(n+1)/2}$ and the corresponding cone in $\mathcal{S}^n$.

\begin{thm}
\label{thm-dual}
Let $\mathcal{C}_d\subset\mathcal{S}^n$ be the convex cone defined in (\ref{def-approxd}). Then
\begin{equation}
\label{def1-approx-dual}
\mathcal{C}_d^*={\rm cl}\,\left\{\left( \langle\,\mathbf{X},\mathbf{M}_d(x_ix_j\,\mathbf{y})\,\rangle\,\right)_{1\leq i\leq j\leq n}\::\:\mathbf{X}\in\mathcal{S}^{s(d)}_+\right\}.
\end{equation}
Equivalently:
\begin{eqnarray}
\nonumber
\mathcal{C}_d^*&=&{\rm cl}\,\left\{\int_{\mathbb{R}^n_+} \mathbf{x}\x^T\,\underbrace{\sigma(\mathbf{x})\,d\mu(\mathbf{x})}_{d\mu_\sigma(\mathbf{x})}\::\: \sigma\in\Sigma[\mathbf{x}]_d\:\right\}\\
\label{def2-approx-dual}
&=&{\rm cl}\,\left\{\:{\rm mass}(\mu_\sigma){\rm E}_{\tilde{\mu}_\sigma}(\mathbf{x}\x^T)\::\: \sigma\in\Sigma[\mathbf{x}]_d\:\right\},
\end{eqnarray}
with $\tilde{\mu}_\sigma$ and ${\rm E}_{\tilde{\mu}_\sigma}(\mathbf{x}\x^T)$ as in (\ref{second}).
\end{thm}
\begin{proof}
Let
\[\Delta_d\,:=\,\left\{\left( \langle\,\mathbf{X},\mathbf{M}_d(x_ix_j\,\mathbf{y})\,\rangle\,\right)_{1\leq i\leq j\leq n}\::\:\mathbf{X}\in\mathcal{S}^{s(d)}_+\right\},\]
so that
\begin{eqnarray*}
\Delta^*_d&=&\left\{\mathbf{a}\in\mathbb{R}^{n(n+1)/2}\::\:\sum_{1\leq i\leq j\leq n}a_{ij}\,\langle\mathbf{X},\mathbf{M}_d(x_ix_j\,\mathbf{y}\rangle\geq0\quad\forall\, \mathbf{X}\in\mathcal{S}^{s(d)}_+\:\right\},\\
&=&\left\{\mathbf{a}\in\mathbb{R}^{n(n+1)/2}\::\:\left\langle\mathbf{X},\mathbf{M}_d\left((\sum_{1\leq i\leq j\leq n}a_{ij}x_ix_j)\,\mathbf{y}\right)\right\rangle\geq0\quad\forall \,\mathbf{X}\in\mathcal{S}^{s(d)}_+\:\right\},\\
&=&\{\mathbf{A}\in\mathcal{S}^n \::\:\langle\mathbf{X},\mathbf{M}_d(f_\mathbf{A}\,\mathbf{y})\rangle\geq0\quad\forall \,\mathbf{X}\in\mathcal{S}^{s(d)}_+\:\}\quad\mbox{[with $\mathbf{A},\mathbf{a}$ as in (\ref{equivalence})]}\\
&=&\{\mathbf{A}\in\mathcal{S}^n \::\:\mathbf{M}_d(f_\mathbf{A}\,\mathbf{y})\,\succeq0\:\}\,=\,\mathcal{C}_d.\end{eqnarray*}
And so we obtain the desired result $\mathcal{C}_d^*=(\Delta_d^*)^*={\rm cl}\,(\Delta_d)$.
Next, writing the singular decomposition of $\mathbf{X}$ as $\sum_{k=0}^s\mathbf{q}_k\,\mathbf{q}_k^T$ for some $s\in\mathbb{N}$ and some
vectors $(\mathbf{q}_k)\subset\mathbb{R}^{s(d)}$,
one obtains that for every $1\leq i\leq j\leq n$,
\begin{eqnarray*}
\langle\,\mathbf{X},\mathbf{M}_d(x_ix_j\,\mathbf{y})\,\rangle &=&\sum_{k=0}^s\langle\,\mathbf{q}_k\mathbf{q}_k^T,\mathbf{M}_d(x_ix_j\,\mathbf{y})\,\rangle\,=\,
\sum_{k=0}^s\langle\,\mathbf{q}_k,\mathbf{M}_d(x_ix_j\,\mathbf{y})\mathbf{q}_k\,\rangle\\
&=&\sum_{k=0}^s\int_{\mathbb{R}^n_+} x_ix_j \,q_k(\mathbf{x})^2\,d\mu(\mathbf{x})\quad\mbox{[by (\ref{local2})]}\\
&=&\int_{\mathbb{R}^n_+} x_ix_j\,\underbrace{\sigma(\mathbf{x})\,d\mu(\mathbf{x})}_{d\mu_\sigma(\mathbf{x})},\end{eqnarray*}
where $\sigma(\mathbf{x})=\sum_{k=0}^sq_k(\mathbf{x})^2\in\Sigma[\mathbf{x}]_d$, and $\mu_\sigma(B)=\int_B\sigma\,d\mu$ for all Borel sets $B$.
\end{proof}
So Theorem \ref{thm-dual} states that $\mathcal{C}_d^*$ is the closure of the convex cone generated by
second-order moments of measures $d\mu_\sigma=\sigma d\mu$, absolutely continuous with respect to $\mu$ (hence with support on $\mathbb{R}^n_+$) and with density being a s.o.s. polynomial $\sigma$ of degree at most $2d$.
Of course we immediately have:
\begin{cor}
\label{cor2}
Let $\mathcal{C}^*_d$, $d\in\mathbb{N}$, be as in (\ref{def2-approx-dual}). Then $\mathcal{C}_d^*\subset\mathcal{C}^*_{d+1}$ for all $d\in\mathbb{N}$, and
\[\mathcal{C}^*\,=\,{\rm cl}\,\bigcup_{d=0}^\infty\mathcal{C}^*_d.\]
\end{cor}
\begin{proof}
As $\mathcal{C}^*_d\subset\mathcal{C}^*_{d+1}$ for all $d\in\mathbb{N}$, the result follows from
\[\mathcal{C}^*\,=\,\left(\,\bigcap_{d=0}^\infty\,\mathcal{C}_d\,\right)^*\,=\,{\rm cl}\,\left({\rm conv}\,\bigcup_{d=0}^\infty\,\mathcal{C}^*_d\,\right)\,=\,
{\rm cl}\,\bigcup_{d=0}^\infty\,\mathcal{C}^*_d.\]
\end{proof}
In other words, $\mathcal{C}_d^*$ approximates $\mathcal{C}^*={\rm conv}\,\{\mathbf{x}\x^T:\mathbf{x}\in\mathbb{R}^n_+\}$ (i.e., the convex hull of second-order moments of Dirac measures
with support in $\mathbb{R}^n_+$) from inside by second-order moments of measures $\mu_\sigma\ll\mu$ whose density is a s.o.s.
polynomial $\sigma$ of degree at most $2d$, and better and better approximations are obtained by letting $d$ increase.
\begin{ex}
{\rm  For instance, with $n=2$, and
$\mathbf{A}=\left[\begin{array}{cc}a & b\\mathcal{B}&c\end{array}\right]$,
it is known that $\mathbf{A}$ is copositive if and only if $a,c\geq0$ and $b+\sqrt{ac}\geq0$; see e.g. \cite{jbhu-sirev}.
Let $\mu$ be the exponential measure on $\mathbb{R}^2_+$ with moments defined in (\ref{mom}).
With $f_\mathbf{A}(\mathbf{x}):=ax_1^2+2bx_1x_2+cx_2^2$ and $d=1$,
the condition $\mathbf{M}_d(f_\mathbf{A}\,\mathbf{y})\succeq0$ which reads
\begin{equation}
\label{mat1}
2\left[\begin{array}{ccc}
a+b+c & 3a+2b+c& a+2b+3c\\
3a+2b+c & 12a+6b+2c &3a+4b+3c\\
a+2b+3c&3a+4b+3c&2a+6b+12c\end{array}\right]\,\succeq\,0,\end{equation}
defines the convex cone $\mathcal{C}_1\subset\mathbb{R}^3$. It is a connected component of the basic semi-algebraic set
$\{(a,b,c): {\rm det}\,(\mathbf{M}_1(f_\mathbf{A}\,\mathbf{y}))\geq0\}$, that is, elements $(a,b,c)$ such that:
\begin{equation}
\label{det}
3a^3+15a^2b+29a^2c+16ab^2+50abc+29ac^2+4b^3+16b^2c+15bc^2+3c^3\,\geq\,0.\end{equation}
Alternatively,  by homogeneity, instead of $\mu$ we may
take the probability measure $\nu$ uniformly supported on the simplex $\Delta$
and with moments $\tilde{\mathbf{y}}=(\tilde{y}_\alpha)$ given in (\ref{mom-nu}), in which case
the corresponding matrix $\mathbf{M}_1(f_\mathbf{A}\,\tilde{\mathbf{y}})$ now reads:
\begin{equation}
\label{mat2}
\frac{1}{360}\left[\begin{array}{ccc}
30(a+b+c) & 6(3a+2b+c)& 6(a+2b+3c)\\
6(3a+2b+c) & 12a+6b+2c &3a+4b+3c\\
6(a+2b+3c)&3a+4b+3c&2a+6b+12c\end{array}\right]\,\succeq\,0.\end{equation}
It turns out that up to a multiplicative factor both matrices (\ref{mat1}) and (\ref{mat2}) have same determinant and so define the same
convex cone $\mathcal{C}_1$ (the same connected component of (\ref{det})).
Figure 1 below displays the projection on the $(a,b)$-plane of the sets $\mathcal{C}_1$ and $\mathcal{C}$ intersected with the unit ball.
\begin{center}
\begin{figure}
\resizebox{0.5\textwidth}{!}
{\includegraphics{copos}}
\caption{$n=2$: Projection on the $(a,b)$-plane of $\mathcal{C}$ versus $\mathcal{C}_1$, both intersected with the unit ball
\end{figure}
\end{center}
}\end{ex}

\subsection{An alternative representation of the cone $\mathcal{C}^*_d$}

From its definition (\ref{def1-approx-dual})
in Theorem \ref{thm-dual}, the cone $\mathcal{C}^*_d\subset\mathcal{S}^n$ is defined through the
matrix variable $\mathbf{X}\in\mathcal{S}^{s(d)}$ which lives in a (lifted) space of dimension $s(d)(s(d)+1)/2$
and with the linear matrix inequality (LMI)  constraint $\mathbf{X}\succeq0$ of size $s(d)$. In contrast,
the convex cone $\mathcal{C}_d\subset\mathcal{S}^n$ defined in (\ref{def-approxd}) is defined solely in terms of the entries of
the matrix $\mathbf{A}\in\mathcal{S}^n$, that is, no projection from a higher dimensional space (or no lifting) is needed.

We next provide another explicit description on how $\mathcal{C}^*_d$ can be generated with no LMI constraint
and with only $s(d)$ variables, but of course this characterization is not well suited for optimization purposes.

Since every $g\in\Sigma[\mathbf{x}]_d$ can be written $\sum_\ell g_\ell^2$ for finitely many
polynomials $(g_\ell)\subset\mathbb{R}[\mathbf{x}]_d$, the convex cone $\Sigma[\mathbf{x}]_d$ of s.o.s. polynomials can be written
\[\Sigma[\mathbf{x}]_d\,=\,{\rm conv}\,\{g^2\::\:g\in\mathbb{R}[\mathbf{x}]_d\:\}.\]
Next, for $g\in\mathbb{R}[\mathbf{x}]_d$, with vector of coefficients $\mathbf{g}=(g_\alpha)\in\mathbb{R}^{s(d)}$, let
$\mathbf{g}^{(2)}=(g^{(2)}_\alpha)\in\mathbb{R}^{s(2d)}$ be the vector of coefficients of $g^2$, that is,
\[g(\mathbf{x})=\sum_{\alpha\in\mathbb{N}^n_d}g_\alpha\,\mathbf{x}^\alpha \quad\rightarrow\quad g(\mathbf{x})^2\,=\,\sum_{\alpha\in\mathbb{N}^n_{2d}}g^{(2)}_\alpha\,\mathbf{x}^\alpha.\]
Notice that for each $\alpha\in\mathbb{N}^n_{2d}$, $g^{(2)}_\alpha$ is quadratic in $\mathbf{g}$.
For instance, with $n=2$ and $d=1$, $g(\mathbf{x})=g_{00}+g_{10}x_{1}+g_{01}x_{2}$ with $\mathbf{g}=(g_{00},\,g_{10},g_{01})^T\in\mathbb{R}^{s(1)}$, and so
\[\mathbf{g}^{(2)}\,=\,(g_{00}^2,\, 2g_{00}g_{10},\,2g_{00}g_{01},\,g_{10}^2,\,2g_{10}g_{01},\,g_{02}^2)^T\in\mathbb{R}^{s(2)}.\]

Next, for $g\in\mathbb{R}[\mathbf{x}]_d$ and every $1\leq i\leq j\leq n$, let $\mathbf{G}_d=(G_{ij})\in\mathcal{S}^n$ be defined by:
\begin{equation}
\label{definal}
G_{ij}\,:=\,\int_{\mathbb{R}^n}g(\mathbf{x})^2\,x_ix_j\,d\mu(\mathbf{x})\,=\,\sum_{\alpha\in\mathbb{N}^n_{2d}}g^{(2)}_\alpha\,(\alpha_i+1){\rm !}(\alpha_j+1){\rm !}\prod_{k\neq i,j}\alpha_k{\rm !},
\end{equation}
for all $1\leq i\leq j\leq n$. We can now describe $\mathcal{C}^*_d$.
\begin{cor}
\label{cor3}
Let $\mathbf{G}_d\in\mathcal{S}^n$ be as in (\ref{definal}). Then:
\begin{equation}
\label{cor3-1}
\mathcal{C}^*_d\,=\,{\rm cl}\,\left({\rm conv}\,\{\mathbf{G}_d\::\: \mathbf{g}\in\mathbb{R}^{s(d)}\:\}\,\right).\end{equation}
\end{cor}
\begin{proof}
From Theorem \ref{thm-dual}
\[\mathcal{C}_d^*={\rm cl}\,\left\{\left( \langle\,\mathbf{X},\mathbf{M}_d(x_ix_j\,\mathbf{y})\,\rangle\,\right)_{1\leq i\leq j\leq n}\::\:\mathbf{X}\in\mathcal{S}^{s(d)}_+\right\}.\]
As in the proof of Theorem \ref{thm-dual}, writing $\mathbf{X}\succeq0$ as $\sum_{k=1}^s\mathbf{g}_k\mathbf{g}_k^T$ for some vectors
$(\mathbf{g}_k)\subset\mathbb{R}^{s(d)}$ (and associated polynomials $(g_k)\subset\mathbb{R}[\mathbf{x}]_d$),
\[\langle\,\mathbf{X},\mathbf{M}_d(x_ix_j\,\mathbf{y})\,\rangle\,=\,\sum_{k=1}^s \int_{\mathbb{R}^n_+} x_ix_jg_k(\mathbf{x})^2\,d\mu,
\,=\,\sum_{k=1}^s G^k_{ij},\]
for all $1\leq i\leq j\leq n$, where $G^k_{ij}$ is as in (\ref{definal}) (but now associated with $g_k$ instead of $g$). Hence
\[\left(\langle\,\mathbf{X},\mathbf{M}_d(x_ix_j\,\mathbf{y})\,\rangle\,\right)_{1\leq i\leq j\leq n}=\sum_{k=1}^s \mathbf{G}^k_d\,\in\,{\rm conv}\,\{\mathbf{G}_d\::\: \mathbf{g}\in\mathbb{R}^{s(d)}\:\}.\]
Hence $\mathcal{C}^*\subseteq {\rm cl}\,\left({\rm conv}\,\{\mathbf{G}_d\::\: \mathbf{g}\in\mathbb{R}^{s(d)}\:\}\right)$.

Conversely, let $h\in {\rm conv}\,\{\mathbf{G}_d\::\: \mathbf{g}\in\mathbb{R}^{s(d)}\:\}$, i.e., for some integer $s$ and polynomials $(g_k)\subset\mathbb{R}[\mathbf{x}]_d$,
and for all $1\leq i\leq j\leq n$,
\begin{eqnarray*}
h_{ij}\,=\,\sum_{k=1}^s \underbrace{\lambda_k}_{>0}\,G^k_{ij}&=&\int_{\mathbb{R}^n_+}\left(\sum_{k=1}^s\lambda_k\,g_k^2(\mathbf{x})\right)\,x_ix_j\,d\mu(\mathbf{x})\\
&=&\langle \mathbf{X},\mathbf{M}_d(x_ix_j\,\mathbf{y})\rangle
\end{eqnarray*}
where $\mathbf{X}=\sum_{k=1}^s\lambda_k\,\mathbf{g}_k\mathbf{g}_k^T\in\mathcal{S}^{s(d)}_+$. Hence
${\rm cl}\,\left({\rm conv}\,\{\mathbf{G}_d\::\: \mathbf{g}\in\mathbb{R}^{s(d)}\:\}\right)\subseteq\mathcal{C}^*$.
\end{proof}
The characterization (\ref{cor3-1}) of $\mathcal{C}^*_d$ should be compared with
the characterization ${\rm conv}\,\{\mathbf{x}\x^T\,:\:\mathbf{x}\in\mathbb{R}^n_+\}$ of $\mathcal{C}^*$.
\begin{ex}
With $n=2$ and $d=1$, $\mathbf{G}_1$ reads:
{\scriptsize
\[\left[\begin{array}{cc}
2g^2_{00}+12g_{10}(g_{00}+g_{01})+4g_{01}(g_{00}+g_{01})+24g_{10}^2&
g_{00}^2+4g_{00}(g_{10}+g_{01})+6(g_{10}^2+g_{01}^2)+8g_{10}g_{01}\\
g_{00}^2+4g_{00}(g_{10}+g_{01})+6(g_{10}^2+g_{01}^2)+8g_{10}g_{01}&
2g^2_{00} +4g_{10}(g_{00}+g_{10})+12g_{01}(g_{00}+g_{10})+24g_{01}^2\end{array}\right]\]
}
\end{ex}

\subsection{$\mathcal{K}$-copositive and $\mathcal{K}$-completely positive matrices}

Let $\mathcal{K}\subset\mathbb{R}^n$ be a closed convex cone and let
$\mathcal{C}_\mathcal{K}$ be the convex cone of $\mathcal{K}$-copositive matrices, i.e., matrices $\mathbf{A}\in\mathcal{S}^n$ such that $\mathbf{x}^T\mathbf{A}\mathbf{x}\geq0$ on $\mathcal{K}$.
Its dual cone $\mathcal{C}^*_\mathcal{K}\subset\mathcal{S}^n$ is the cone of $\mathcal{K}$-completely positive matrices.

Then one may define a hierarchy of convex cones $(\mathcal{C}_{\mathcal{K}})_d$ and $(\mathcal{C}^*_{\mathcal{K}})_d$, $d\in\mathbb{N}$, formally exactly as in Theorem \ref{thmain}
and Theorem \ref{thm-dual}, but now $\mathbf{y}$ is the moment sequence of a finite Borel measure
$\mu$ with ${\rm supp}\,\mu=\mathcal{K}$ (instead of ${\rm supp}\,\mu=\mathbb{R}^n_+$ in (\ref{mom})), i.e., $\mathbf{y}=(y_\alpha)$, $\alpha\in\mathbb{N}^n$, with:
\begin{equation}
\label{newmom}
y_\alpha\,:=\,\int_\mathcal{K}\,\mathbf{x}^\alpha\,d\mu,\qquad \forall\alpha\in\mathbb{N}^n\quad\mbox{ (and where ${\rm supp}\,\mu=\mathcal{K}$)}.
\end{equation}

And so with $\mathbf{y}$ as in (\ref{newmom}) Theorem \ref{thmain} and \ref{thm-dual}, as well as Corollary \ref{cor2}, are still valid.
(In (\ref{def2-approx-dual} replace $\int_{\mathbb{R}^n_+}$ with $\int_\mathcal{K}$). Therefore,
\[\mathcal{C}_\mathcal{K}\,=\,\bigcap_{d=0}^\infty (C_{\mathcal{K}})_d\quad\mbox{ and }\quad \mathcal{C}^*_\mathcal{K}\,=\,{\rm cl}\,\bigcup_{d=0}^\infty(\mathcal{C}_{\mathcal{K}})_d^*.\]
But of course, for practical implementation, one need to know the sequence $\mathbf{y}=(y_\alpha)$, $\alpha\in\mathbb{N}^n$, which was easy when $\mathcal{K}=\mathbb{R}^n_+$.
For instance, if $\mathcal{K}$ is a polyhedral cone, by homogeneity of $f_\mathbf{A}$, one may equivalently
consider a compact base of $\mathcal{K}$ (which is a polytope $\mathcal{K}'$), and take for $\mu$ the Lebesgue measure on $\mathcal{K}'$.
Then all  moments of $\mu$ can be calculated exactly. The same argument works for every
convex cone $\mathcal{K}$ for which one may compute all moments of a finite Borel measure $\mu$
whose support is a compact base of $\mathcal{K}$.

\section*{Acknowledgement}
The author wishes to thank two anonymous referees for their very helpful remarks and suggestions
to improve the initial version of this note.


Tighter bound of Sketched Generalized Matrix Approximation

Abstract: Generalized matrix approximation plays a fundamental role in many machine learning problems, such as CUR decomposition, kernel approximation, and matrix low rank approximation. Especially with today's applications involved in larger and larger dataset, more and more efficient generalized matrix approximation algorithems become a crucially important research issue. In this paper, we find new sketching techniques to reduce the size of the original data matrix to develop new matrix approximation algorithms. Our results derive a much tighter bound for the approximation than previous works: we obtain a $(1+\epsilon)$ approximation ratio with small sketched dimensions which implies a more efficient generalized matrix approximation.

\section{Introduction}

Matrix manipulations are the basis of modern data analysis. As the datasets becomes larger and larger, it is much more difficult to perform exact matrix multiplication, inversion, and  decomposition. Consequently, matrix approximation techniques have been extensively studied, including approximate matrix multiplication \cite{cohen1999approximating,drineas2006fast,kyrillidis2014approximate} and low-rank matrix approximation \cite{bourgain2013toward,boutsidis2014near,clarkson2013low,martinsson2011randomized}.

In this paper we are concerned with the generalized matrix approximation problem \cite{stewart1999four,friedland2007generalized,sou2012generalized}:
\[ \min_{{\bf X}}\|{\bf A}-{\bf M}{\bf X}{\bf N}\|_F,\]
where ${\bf A}\in{\RB^{m\times n}}$, ${\bf M}\in{\mathbb R}^{m\times c}$ and ${\bf N}\in{\mathbb R}^{r\times n}$. It is well known that the solution is ${\bf X}^* = {\bf M}^{\dagger}{\bf A}{\bf N}^{\dagger}$. It costs ${\mathcal O}(\mathrm{nnz}({\bf A})\cdot\min(c,r)+mc^2+nr^2)$ time to solve the generalized matrix approximation exactly to get ${\bf X}^{*}$. Generalized matrix approximation takes an important role in solving some machine learning problems such as the CUR decomposition \cite{wang2013improving,gittens2013revisiting,wang2015towards}, modified Nystr{\"o}m method \cite{wang2013improving}, and distributed PCA \cite{liang2014improved,boutsidis2015optimal}. It is also a key research topic in numerical linear algebra \cite{stewart1999four,friedland2007generalized}.

When ${\bf N}$ is the identity matrix, the generalized matrix approximation degenerates to the ordinary least squares regression. To solve the least squares regression $\tilde{{\bf X}} = \mathop{\rm argmin}_{\bf X}\|{\bf M}{\bf X} - {\bf A}\|_F$ more efficiently when $c\ll m$, recent studies suggest multiplying a sketching matrix ${\bf S}\in{\mathbb R}^{s_c\times m}$, where $s_c = {\mathcal O}(c/\epsilon)$, to get a sketched least squares regression $\hat{{\bf X}} = \mathop{\rm argmin}_{\bf X}\|{\bf S}({\bf M}{\bf X} - {\bf A})\|_F$. The studies~\cite{clarkson2013low,drineas2011faster,sarlos2006improved} also prove that the reduced least squares regression obtains a $(1+\epsilon)$ relative error bound, which is, $\|{\bf M}\hat{{\bf X}}-{\bf A}\|_F\leq(1+\epsilon)\|{\bf M}\tilde{{\bf X}}-{\bf A}\|_F$.

Since the ordinary least squares regression is a special case of the generalized matrix approximation, it is reasonable to use the sketching technique to accelerate solving the generalized matrix approximation.   Recently, \citet{wang2015towards} proposed to use leverage-score sketching matrices ${\bf S}_{\bf M}\in {\mathbb R}^{s_c\times m}$ and ${\bf S}_{\bf N}\in{\mathbb R}^{s_r\times n}$ to efficiently solve the generalized matrix approximation via
\[
\hat{{\bf X}} = \mathop{\rm argmin}_{\bf X} \|{\bf S}_{\bf M}({\bf A}-{\bf M}{\bf X}{\bf N}){\bf S}_{\bf N}^T\|_F.
\]
However, it needs $s_c = {\mathcal O}(\sqrt{\min(m,n)}c/\sqrt{\epsilon})$ and $s_r = {\mathcal O}(\sqrt{\min(m,n)}r/\sqrt{\epsilon})$ to get a $(1+\epsilon)$ relative error bound in \cite{wang2015towards}. The result of \cite{wang2015towards} can be easily extended to other sketching matrices with the same reduced dimensions, i.e., the same $s_c$ and $s_r$. \citet{boutsidis2015optimal} proposed to solve the generalized rank-constrained matrix approximation problem $\min_{\text{rank}({\bf X})\leq k}\|{\bf A}-{\bf M}{\bf X}{\bf N}\|_F$ by multiplying sketching matrices ${\bf S}_{\bf M}$ and ${\bf S}_{\bf N}$ with the affine embedding property. It needs $s_c = {\mathcal O}(c/\epsilon^2)$ and $s_r = {\mathcal O}(r/\epsilon^2)$ to achieve a $(1+\epsilon)$ relative error bound.

In this paper, we prove that  $s_c = {\mathcal O}(c/\epsilon)$ and $s_r = {\mathcal O}(r/\epsilon)$ are enough to achieve a $(1+\epsilon)$ relative error bound for the generalized matrix approximation problem. We compare our result with previous work in Table~\ref{tb:comp} in detail. As we can see, our bound is much tighter than previous work. Besides, a tigher bound leads to faster sketched generalized matrix approximation. Using leverage-score sketching, the running time in Theorem~\ref{thm:gmp_lev_score} is much less than that of \citet{boutsidis2015optimal} since, $c$ and $r$ is much less than $n$ and $m$, especially when ${\bf A}$ is dense. Comparing with \cite{wang2015towards}, though the first part of compuation cost both is ${\mathcal O}(\mathrm{nnz}({\bf M}))$, the running time is $m$ times less than \cite{wang2015towards} as to the second part of computation cost. Hence, our result is better, especially when $\mathrm{nnz}({\bf M})$ is small, because the leading cost will be the second part of computation cost.
\begin{table}[]
	\centering
	\caption{Comparisons of skecthed dimensions (Assume $m\leq n$ and $\mathrm{nnz}({\bf M})\leq \mathrm{nnz}({\bf N})$)}
	\label{tb:comp}
	\begin{tabular}{|c|c|c|l|}
		\hline
		Reference                    & $s_c$                   & $s_r$                                           & Computation Cost         \\ \hline
		\citet{boutsidis2015optimal} & ${\mathcal O}(c/\epsilon^2)$     & ${\mathcal O}(r/\epsilon^2)$                             & ${\mathcal O}(\mathrm{nnz}({\bf A}) + \mathrm{nnz}({\bf M})+poly(c,r,\epsilon^{-1}))$             \\ \hline
		\citet{wang2015towards}      & ${\mathcal O}(\sqrt{m}c/\sqrt{\epsilon})$    & ${\mathcal O}(\sqrt{m}c/\sqrt{\epsilon})$    &  ${\mathcal O}(\mathrm{nnz}({\bf M})+m\cdot poly(c,r,\epsilon^{-1})$            \\ \hline
		Theorem~\ref{thm:gmp_sps_gauss}   & ${\mathcal O}(c/\epsilon)$    & ${\mathcal O}(r/\epsilon)$                             & ${\mathcal O}(\mathrm{nnz}({\bf A})+\mathrm{nnz}({\bf M})+poly(c,r,\epsilon^{-1}))$            \\ \hline
		Theorem~\ref{thm:gmp_lev_score}   & ${\mathcal O}(c/\epsilon+c\log c)$    &${\mathcal O}(r/\epsilon+r\log r)$    &        ${\mathcal O}(\mathrm{nnz}({\bf M})+poly(c,r,\epsilon^{-1}))$      \\ \hline
	\end{tabular}
\end{table}
\section{Notation and Preliminaries}
\label{sec:notation}

In this section, we introduce some notation and  preliminaries
that will be used in this paper.

\subsection{Notation}

Let ${\bf I}_m$ be the $m{\times}m$ identity matrix.
Given a matrix ${\bf A}=[a_{ij}] \in {\mathbb R}^{m \times n}$ of rank $ \rho $,  its SVD is given as
${\bf A}={\bf U}\mbox{\boldmath$\Sigma$\unboldmath}{\bf V}^{T}={\bf U}_{k} \mbox{\boldmath$\Sigma$\unboldmath}_{k} {\bf V}_{k}^{T}+{\bf U}_{\rho\setminus k} \mbox{\boldmath$\Sigma$\unboldmath}_{\rho{\setminus} k}{\bf V}_{\rho{\setminus}k}^{T}$,
where ${\bf U}_{k}$ and ${\bf U}_{\rho{\setminus}k}$ contain the left singular vectors of ${\bf A}$,  ${\bf V}_{k}$ and ${\bf V}_{\rho{\setminus}k}$ contain the right
singular vectors of ${\bf A}$, and $\mbox{\boldmath$\Sigma$\unboldmath}=\mathrm{diag}(\sigma_1, \ldots, \sigma_{\rho})$ with $\sigma_1\geq \sigma_2 \geq \cdots \geq \sigma_{\rho}>0$ are
the nonzero singular values of ${\bf A}$. Accordingly,   $\|{\bf A}\|_{F} \triangleq (\sum_{i,j}a_{ij}^{2})^{1/2}=(\sum_{i}\sigma_{i}^{2})^{1/2}$
is the Frobenius norm of ${\bf A}$ and
$\|{\bf A}\|_{2}\triangleq \sigma_{1}$ is the spectral norm.

Additionally, ${\bf A}^{\dagger} \triangleq {\bf V}\mbox{\boldmath$\Sigma$\unboldmath}^{-1}{\bf U}^{T} \in {\mathbb R}^{n \times m}$ is
the  Moore-Penrose pseudoinverse of ${\bf A}$, which is  unique. It is easy to verify that ${\mathrm{rank}}({\bf A}^{\dagger}) ={\mathrm{rank}}({\bf A})=\rho$.
Moreover, for all $i=1, \dots, \rho$,
$\sigma_{i}({\bf A}^{\dagger})=1/\sigma_{\rho-i+1}({\bf A})$. If  ${\bf A}$ is of full row rank, then ${\bf A}\A^{\dagger}={\bf I}_{m}$.
Also, if ${\bf A}$ is of full column rank, then ${\bf A}^{\dagger}{\bf A}={\bf I}_{n}$.
When $m=n$, $\text{tr}({\bf A}) = \sum_{i=1}^{n}a_{ii}$ is the trace of ${\bf A}$.

It is well known that ${\bf A}_{k}\triangleq {\bf U}_{k}\mbox{\boldmath$\Sigma$\unboldmath}_{k}{\bf V}_{k}^{T}$ is the minimizer of both $\min \|{\bf A}-{\bf X}\|_{F}$ and $\min \|{\bf A}-{\bf X}\|_{2}$ over all matrices ${\bf X} \in {\mathbb R}^{m \times n}$ of rank at most $k\leq\rho$. Thus, ${\bf A}_k$ is called the best rank-$k$ approximation of ${\bf A}$.
Let $\mathrm{nnz}({\bf A})$ denote the number of nonzero entries of ${\bf A}$.

\subsection{Subspace Embedding}

Oblivious subspace embedding is an important sketching tool in randomized numerical linear algebra.
By oblivious subspace embedding, a matrix can be projected to a much lower dimensional subspace,
which leads to much faster matrix operations.
\begin{definition}[\cite{woodruff2014sketching}]
	Given $\varepsilon>0$ and $\delta>0$, let $\Pi$ be a distribution on $s \times m$ real matrices, where $s$ relies on $m$, $d$, $\varepsilon$ and $\delta$. Suppose that
	with probability at least $1-\delta$, for any fixed $m \times d$ matrix ${\bf A}$, a matrix ${\bf S}$ drawn from distribution $\Pi$
	is a $(1+\epsilon)$ $\ell_{2}$-subspace embedding for ${\bf A}$, that is, for all ${\bf x} \in {\mathbb R}^{d}$,
	$\|{\bf S}{\bf A}{\bf x}\|_{2}^{2}=(1\pm\epsilon)\|{\bf A}{\bf x}\|_{2}^{2}$ with probability
	$1-\delta$. Then we call $\Pi$ an $(\epsilon,\delta)$-oblivious $\ell_{2}$-subspace
	embedding.
\end{definition}
For the sake of conciseness, the $(\epsilon,\delta)$-oblivious $\ell_{2}$-subspace embedding is referred as an $\epsilon$-subspace embedding. Now we list some important subspace embedding matrices and their properties which will be used in this paper.

\begin{definition}[leverage-score sketching matrix] \label{def:lev_score}
	Let ${\bf V}\in{\mathbb R}^{n\times k}$  be column orthonormal basis for ${\bf A}\in{\mathbb R}^{n\times k}$  with $n>k$,  and ${\bf v}_{i,*}$ denote the $i$-th row of ${\bf V}$. Let $\ell_{i} = \|{\bf v}_{i,*}\|_{F}^{2}/k$ and $r$ be an integer with $1\leq r\leq n$. Then the $\ell_{i}$'s are leverage scores for ${\bf A}$. Construct a sampling matrix  $\mbox{\boldmath$\Omega$\unboldmath}\in{\mathbb R}^{n\times r}$ and a rescaling matrix ${\bf D}\in{\mathbb R}^{r \times r}$ as follows. For every $j = 1,\dots,r$, independently and with replacement, pick an index $i$ from the set $\{1,2\dots,n\}$ with probability  $\ell_{i}$ and set $\mbox{\boldmath$\Omega$\unboldmath}_{ij} = 1$ and ${\bf D}_{jj}=1/\sqrt{\ell_{i}r}$. The leverage-score sketching matrix ${\bf S}$ for ${\bf A}$ is then defined as ${\bf S} = \mbox{\boldmath$\Omega$\unboldmath}{\bf D}$.
\end{definition}

\begin{theorem}[\cite{woodruff2014sketching}]\label{thm:subspace_embedding}
	Given ${\bf A}\in{\mathbb R}^{m\times d}$ of full column rank, assume ${\bf S}\in {\mathbb R}^{s\times m}$ is an $\epsilon$-subspace embedding matrix for ${\bf A}$. If ${\bf S}$ is a sparse subspace embedding matrix, then $s = {\mathcal O}(d^2\epsilon^{-2})$ is sufficient. If ${\bf S}$ is an  $s \times m$ matrix of i.i.d.\ normal random variables with variance $1/s$, then $s=\Theta(d\epsilon^{-2})$ is sufficient. For a leverage-score sketching matrix ${\bf S}$, $s = {\mathcal O}(d\log d\epsilon^{-2})$ is needed.
\end{theorem}

\begin{theorem}[\cite{clarkson2013low,boutsidis2015optimal}]\label{thm:multi}
	For ${\bf A}\in {\RB^{m\times n}}$ and  ${\bf B}\in {\mathbb R}^{m \times k}$, there is an $s = \Theta(\epsilon^{-2})$, so that for an $s \times m$ sparse embedding matrix ${\bf S}$ or an $s \times m$ matrix ${\bf S}$ of i.i.d.\ normal random variables with variance $1/s$, or an $s\times m$ leverage-score sketching matrix for ${\bf A}$ under the condition that ${\bf A}$ has orthonormal columns, then it holds that
	\begin{equation}
		\|{\bf A}^{T}{\bf S}^{T}{\bf S}{\bf B}-{\bf A}^{T}{\bf B}\|_{F}^{2}<\epsilon^{2}\|{\bf A}\|_{F}^{2}\|{\bf B}\|_{F}^{2} \label{eq:multi}
	\end{equation}
	with probability at least $1-\delta$	for any fixed $\delta>0$.
\end{theorem}

Other types of sketching matrices like Subsampled Randomized Hadamard Transformation and detailed properties of sketching matrices and subspace embedding matrices can be
found in the survey~\cite{woodruff2014sketching}.
\section{Main Result}
\label{sec:main_result}

We first give the conditions that subspace embedding matrices should satisfy for a sketched generalized matrix approximation to achieve a $(1+\epsilon)$ error bound. The detailed conditions are depicted in Theorem~\ref{thm:gmp_main}.

\begin{theorem}\label{thm:gmp_main}
	Given that ${\bf A}\in {\mathbb R}^{m\times n}$, ${\bf M}\in{\mathbb R}^{m\times c}$ and ${\bf N}\in {\mathbb R}^{r\times n}$, assume that ${\bf S}_{\bf M}\in {\mathbb R}^{s_c\times m}$ is a subspace embedding matrix for ${\bf M}$ with error parameter $\epsilon_0 = 1/2$,  and ${\bf S}_{\bf M}$ also makes  Eqn.~\eqref{eq:multi} hold with error parameter $\sqrt{\epsilon/c}$. Similarly, ${\bf S}_{\bf N}\in{\mathbb R}^{{\bf s}_r\times n}$ is a subspace embedding matrix for ${\bf N}^T$ with  error parameter $\epsilon_0 = 1/2$ and ${\bf S}_{\bf N}$  also makes Eqn.~\eqref{eq:multi} hold with error parameter $\sqrt{\epsilon/r}$.  Let
	\begin{equation}
		{\bf X}^{*} =  {\bf M}^{\dagger}{\bf A}{\bf N}^{\dagger} = \mathop{\rm argmin}_{{\bf X}}\|{\bf A}-{\bf M}{\bf X}{\bf N}\|_{F} \label{eq:U*}
	\end{equation}
	and
	\begin{equation}
		\hat{{\bf X}} = ({\bf S}_{{\bf M}}{\bf M})^{\dagger}{\bf S}_{{\bf M}}{\bf A}{\bf S}_{{\bf N}}^T({\bf N}{\bf S}_{{\bf N}}^T)^{\dagger} \label{eq:U_hat}.
	\end{equation}
	Then we have
	\[
	\|{\bf A}-{\bf M}\hat{{\bf X}}{\bf N}\|_{F}\leq (1+\epsilon)\|{\bf A}-{\bf M}{\bf X}^{*}{\bf N}\|_{F}.
	\]
\end{theorem}

The conditions required in Theorem~\ref{thm:gmp_main} are all satisfied by oblivious embedding matrices, including sparse embedding matrices, gaussian matrices, subsampled randomized Hadamard matrices~\cite{woodruff2014sketching}, as well as their combinations. Now we present  Theorem~\ref{thm:gmp_sps_gauss}, which shows that  an $\mathrm{nnz}({\bf A})$ generalized matrix approximation solver can be achieved and the corresponding sketched dimensions are $s_c = {\mathcal O}(c/\epsilon)$ and $s_r = {\mathcal O}(r/\epsilon)$, respectively.

\begin{theorem}\label{thm:gmp_sps_gauss}
	We are given ${\bf A}\in {\mathbb R}^{m\times n}$, ${\bf M}\in{\mathbb R}^{m\times c}$ and ${\bf N}\in {\mathbb R}^{r\times n}$. Assume that $\mbox{\boldmath$\Pi$\unboldmath}_{\bf M}$ is a $t\times m$ sparse embedding matrix and ${\bf G}_{\bf M}$ is an $s_c\times t$ matrix of i.i.d.\ normal random variables with variance $1/s_c$, where $t = {\mathcal O}(c/\epsilon+c^2)$ and ${\bf s}_c = {\mathcal O}(c/\epsilon)$. Construct ${\bf S}_{\bf M} = {\bf G}_{\bf M}\mbox{\boldmath$\Pi$\unboldmath}_{\bf M}$. Similarly, $\mbox{\boldmath$\Pi$\unboldmath}_{\bf N}$ is a $t^{\prime}\times n$ sparse embedding matrix and ${\bf G}_{\bf N}$ is a $s_r\times t^{\prime}$ matrix of i.i.d.\ normal random variables with variance $1/s_r$, where $t^\prime = {\mathcal O}(r/\epsilon+r^2)$ and $s_r = {\mathcal O}(r/\epsilon)$. Then we construct ${\bf S}_{{\bf N}} = {\bf G}_{{\bf N}}\mbox{\boldmath$\Pi$\unboldmath}_{{\bf N}}$. If ${\bf X}^{*}$ and $\hat{{\bf X}}$ are defined as \eqref{eq:U*} and \eqref{eq:U_hat} respectively, then we have
	\[
	\|{\bf A}-{\bf M}\hat{{\bf X}}{\bf N}\|_{F}\leq (1+\epsilon)\|{\bf A}-{\bf M}{\bf X}^{*}{\bf N}\|_{F},
	\]
	with high probability and $\hat{{\bf X}}$ can be constructed with the computational complexity of
	\begin{align}
		{\mathcal O}(&\mathrm{nnz}({\bf A})+mc+nr+ c^2r/\epsilon^3 +cr^2/\epsilon^3 + c^2r^2/\epsilon^2 \notag+ c^3r/\epsilon^2\\&+cr^3/\epsilon^2 +\min(c^2r^3/\epsilon, c^3r^2/\epsilon))+c^4/\epsilon+r^4/\epsilon). \label{eq:time_analysis}
	\end{align}
\end{theorem}

\begin{proof}
	Let $\mbox{\boldmath$\Pi$\unboldmath}_{\bf M}$ and ${\bf G}_{\bf M}$ be $\epsilon_0$-subspace embedding matrices for $c$-subspace with $\epsilon_0=1/4$, which needs $t = {\mathcal O}(c^2)$ and $s_c = {\mathcal O}(c)$. By Lemma~\ref{lem:embed_add}, we have ${\bf G}_{\bf M}\mbox{\boldmath$\Pi$\unboldmath}_{\bf M}$ is a $1/2$-subspace embedding matrix for ${\bf M}$. By Lemma~\ref{lem:mult_add} and Theorem~\ref{thm:multi}, $t = {\mathcal O}(c/\epsilon)$  and $s_c = {\mathcal O}(c/\epsilon)$, and ${\bf S}_{\bf M} = {\bf G}_{\bf M}\mbox{\boldmath$\Pi$\unboldmath}_{\bf M}$ makes Eqn.~\eqref{eq:multi} hold with error parameter $\sqrt{\epsilon/c}$. The similar result holds for $\mbox{\boldmath$\Pi$\unboldmath}_{\bf N}$, ${\bf G}_{\bf N}$ and ${\bf S}_{\bf N}$. Thus, ${\bf S}_{\bf M}$ and ${\bf S}_{\bf N}$ satisfy all conditions in Theorem~\ref{thm:gmp_main}. By Theorem~\ref{thm:gmp_main}, we obtain the result.

	For computation complexity, it needs ${\mathcal O}(\mathrm{nnz}({\bf A})+ c^2r/\epsilon^3 +cr^2/\epsilon^3 + c^2r^2/\epsilon^2+ c^3r/\epsilon^2+cr^3/\epsilon^2 +\min(c^2r^3/\epsilon, c^3r^2/\epsilon))$ to compute ${\bf S}_{\bf M}{\bf A}{\bf S}_{\bf N}^T$. Computing $({\bf S}_{\bf M}\M)^{\dagger}$ and $({\bf N}{\bf S}_{\bf N}^T)^{\dagger}$ requires ${\mathcal O}(mc + c^3/\epsilon^2 + c^4/\epsilon)$ and ${\mathcal O}(nr+r^3/\epsilon^2+r^4/\epsilon)$ respectively. Matrix multiplications of  $({\bf S}_{{\bf M}}{\bf M})^{\dagger}{\bf S}_{{\bf M}}{\bf A}{\bf S}_{{\bf N}}^T({\bf N}{\bf S}_{{\bf N}}^T)^{\dagger}$ need ${\mathcal O}(\min(c^2r/\epsilon^2+cr^2/\epsilon, cr^2/\epsilon^2+c^2r/\epsilon))$. Hence, the total cost of constructing $\hat{{\bf X}}$ is ${\mathcal O}(\mathrm{nnz}({\bf A})+mc+nr+ c^2r/\epsilon^3 +cr^2/\epsilon^3 + c^2r^2/\epsilon^2+ c^3r/\epsilon^2+cr^3/\epsilon^2 +\min(c^2r^3/\epsilon, c^3r^2/\epsilon))+c^4/\epsilon+r^4/\epsilon)$.
\end{proof}
The lemmas mentioned in the proof of Theorem~\ref{thm:gmp_sps_gauss} are given in Appendix~\ref{app:lemma}.

Leverage-score sketching matrices are significant in randomized numerical linear algebra. Using leverage-score sketching matrices, we can achieve faster sketched generalized matrix approximation than ${\mathcal O}(\mathrm{nnz}({\bf A}))$. It just needs ${\mathcal O}(\mathrm{nnz}({\bf M})+\mathrm{nnz}({\bf N}))$ arithmetic operations comparing with ${\mathcal O}(\mathrm{nnz}({\bf A}))$ in Theorem~\ref{thm:gmp_sps_gauss}.

\begin{theorem} \label{thm:gmp_lev_score}
We are given ${\bf A}\in {\mathbb R}^{m\times n}$, ${\bf M}\in{\mathbb R}^{m\times c}$ and ${\bf N}\in {\mathbb R}^{r\times n}$. Let ${\bf S}_{\bf M}\in {\mathbb R}^{s_c\times m}$ be the leverage-score sketching matrix for ${\bf M}$  with $s_c = {\mathcal O}(c/\epsilon+c\log c)$. Similarly, ${\bf S}_{\bf N}\in{\mathbb R}^{s_r\times n}$ is the leverage-score sketching matrix for ${\bf N}^T$ with $s_r = {\mathcal O}(r/\epsilon+r\log r)$. If ${\bf X}^{*}$ and $\hat{{\bf X}}$ are defined as \eqref{eq:U*} and \eqref{eq:U_hat} respectively, then we have
	\[
	\|{\bf A}-{\bf M}\hat{{\bf X}}{\bf N}\|_{F}\leq (1+\epsilon)\|{\bf A}-{\bf M}{\bf X}^{*}{\bf N}\|_{F}
	\]
	with high probability. And $\hat{{\bf X}}$ can be constructed with  the computational complexity of
	\begin{align*}
	{\mathcal O}(&\mathrm{nnz}({\bf M})+\mathrm{nnz}({\bf N}) + (c^2r+cr^2)/\epsilon^2+(c^2r+cr^2)\log(cr)/\epsilon \\&+ (c^3+r^3)/\epsilon+(c^2r+cr^2)\log c \log r+ c^3\log c+r^3\log r)
	\end{align*}
\end{theorem}
\begin{proof}
	By Theorem~\ref{thm:subspace_embedding} and~\ref{thm:multi}, it is easy to check that ${\bf S}_{\bf M}$ and ${\bf S}_{\bf N}$ satisfy the conditions in Theorem~\ref{thm:gmp_main}. Hence, the result holds by Theorem~\ref{thm:gmp_main}.

	As for compuational cost, it costs ${\mathcal O}(\mathrm{nnz}({\bf M}))$ and ${\mathcal O}(\mathrm{nnz}({\bf N}))$ time to compute leverage scores of ${\bf M}$ and ${\bf N}^T$ respectively~\cite{li2013iterative}. And it takes ${\mathcal O}(cr/\epsilon^2+cr\log(cr)/\epsilon + \log c\log r)$ to compute ${\bf S}_{\bf M}{\bf A}{\bf S}_{\bf N}^T$. And it requires ${\mathcal O}(c^3/\epsilon+r^3/\epsilon + c^3\log c+r^3\log r)$ time to compute $({\bf S}_{\bf M}\M)^{\dagger}$ and $({\bf S}_{\bf N}\N^T)^{\dagger}$. It costs ${\mathcal O}((c^2r+cr^2)/\epsilon^2 + (c^2r+cr^2)\log(cr)/\epsilon+(c^2r+cr^2)\log c \log r)$ arithmetic operations to achieve the multiplication of $({\bf S}_{{\bf M}}{\bf M})^{\dagger}{\bf S}_{{\bf M}}{\bf A}{\bf S}_{{\bf N}}^T({\bf N}{\bf S}_{{\bf N}}^T)^{\dagger}$. Hence, the total cost of constructing $\hat{X}$ is
	\begin{align*}
	{\mathcal O}(&\mathrm{nnz}({\bf M})+\mathrm{nnz}({\bf N}) + (c^2r+cr^2)/\epsilon^2+(c^2r+cr^2)\log(cr)/\epsilon \\&+ (c^3+r^3)/\epsilon+(c^2r+cr^2)\log c \log r+ c^3\log c+r^3\log r)
	\end{align*}
\end{proof}

Now we consider the symmetric case where ${\bf A}$ is symmetric and ${\bf N} = {\bf M}^T$,  ${\bf X}^*$ constructed as Eqn.~\eqref{eq:U*} is a symmetric matrix. Note that $\hat{{\bf X}}$ is asymmetric in most cases since ${\bf S}_{\bf M}$ and ${\bf S}_{\bf N}$ are chosen independently. However, we can construct $\tilde{{\bf X}} = (\hat{{\bf X}}+\hat{{\bf X}}^T)/2$ which is symmetric and can still keep relative error bound.
\begin{corollary}
	Let ${\bf A}\in{\mathbb R}^{m\times m}$ be a symmetric matrix. ${\bf M}$ is an $m\times c$ matrix. ${\bf S}_1$ and ${\bf S}_2$ are subspace embedding matrices for ${\bf M}$ with error parameter $\epsilon_0 = 1/2$,  and they also satisfy Eqn.~\eqref{eq:multi}  with error parameter $\sqrt{\epsilon/c}$.  Let
	\begin{equation*}
		{\bf X}^{*} =  {\bf M}^{\dagger}{\bf A}({\bf M}^T)^{\dagger} = \mathop{\rm argmin}_{{\bf X}}\|{\bf A}-{\bf M}{\bf X}{\bf M}^T\|_{F}
	\end{equation*}
	and
	\begin{equation*}
		\hat{{\bf X}} = ({\bf S}_{1}{\bf M})^{\dagger}{\bf S}_{1}{\bf A}{\bf S}_{2}^T({\bf M}^T{\bf S}_{2}^T)^{\dagger}.
	\end{equation*}
	Construct a symmetric matrix $\tilde{{\bf X}}$ as
	\[
	\tilde{{\bf X}} =(\hat{{\bf X}}+\hat{{\bf X}}^T)/2.
	\]
	Then we have
	\[
	\|{\bf A}-{\bf M}\tilde{{\bf X}}{\bf M}^T\|_{F}\leq (1+\epsilon)\|{\bf A}-{\bf M}{\bf X}^{*}{\bf M}^T\|_{F}.
	\]
\end{corollary}

\section{Proof of Main Theorem}\label{sec:main_proof}
In this section, we give the detailed proof of our main theorem, i.e., Theorem~\ref{thm:gmp_main}.
\begin{proof} {\bf{of Theorem~\ref{thm:gmp_main}}}
	We define
	\[
	{\bf A}^\vdash \triangleq {\bf A}-{\bf M}{\bf X}^*{\bf N}.
	\]
	We let the condensed SVDs of ${\bf M}$ and ${\bf N}$ be respectively
	\[
	{\bf M} = {\bf X}_{\bf M}\mbox{\boldmath$\Sigma$\unboldmath}_{{\bf M}}{\bf V}_{\bf M}^T \qquad \text{and } \qquad {\bf N} = {\bf X}_{\bf N}\mbox{\boldmath$\Sigma$\unboldmath}_{\bf N}{\bf V}_{\bf N}^T.
	\]

	We have
	\begin{align}
	{\bf M}\hat{{\bf X}}{\bf N} = & {\bf M}({\bf S}_{{\bf M}}{\bf M})^{\dagger}{\bf S}_{{\bf M}}{\bf A}{\bf S}_{{\bf N}}^T({\bf N}{\bf S}_{{\bf N}}^T)^{\dagger}{\bf N} \notag\\
	=& {\bf U}_{\bf M}\mbox{\boldmath$\Sigma$\unboldmath}_{{\bf M}}{\bf V}_{\bf M}^T({\bf S}_{\bf M}{\bf U}_{\bf M}\mbox{\boldmath$\Sigma$\unboldmath}_{{\bf M}}{\bf V}_{\bf M}^T)^\dagger{\bf S}_{{\bf M}}{\bf A}{\bf S}_{{\bf N}}^T({\bf U}_{\bf N}\mbox{\boldmath$\Sigma$\unboldmath}_{\bf N}{\bf V}_{\bf N}^T{\bf S}_{\bf N}^T)^\dagger{\bf U}_{\bf N}\mbox{\boldmath$\Sigma$\unboldmath}_{\bf N}{\bf V}_{\bf N}^T \notag\\
	=&{\bf U}_{\bf M}\mbox{\boldmath$\Sigma$\unboldmath}_{{\bf M}}{\bf V}_{\bf M}^T(\mbox{\boldmath$\Sigma$\unboldmath}_{{\bf M}}{\bf V}_{\bf M}^T)^\dagger({\bf S}_{\bf M}{\bf U}_{\bf M})^\dagger{\bf S}_{{\bf M}}{\bf A}{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^T{\bf S}_{\bf N}^T)^\dagger({\bf U}_{\bf N}\mbox{\boldmath$\Sigma$\unboldmath}_{\bf N})^{\dagger}{\bf U}_{\bf N}\mbox{\boldmath$\Sigma$\unboldmath}_{\bf N}{\bf V}_{\bf N}^T \label{eq:pinv}\\
	=& {\bf U}_{\bf M}({\bf S}_{\bf M}{\bf U}_{\bf M})^\dagger{\bf S}_{{\bf M}}{\bf A}{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^T{\bf S}_{\bf N}^T)^\dagger{\bf V}_{\bf N}^{T}, \notag
	\end{align}
	where \eqref{eq:pinv} is because ${\bf S}_{\bf M}{\bf U}_{\bf M}$ is of full column rank and $\mbox{\boldmath$\Sigma$\unboldmath}_{\bf M}{\bf V}_{\bf M}^T$ is of full row rank.

	We define ${\bf Z}^*$ by
	\[
	{\bf M}{\bf X}^*{\bf N} = {\bf U}_{\bf M}(\mbox{\boldmath$\Sigma$\unboldmath}_{{\bf M}}{\bf V}_{\bf M}^T{\bf X}^*{\bf U}_{\bf N}\mbox{\boldmath$\Sigma$\unboldmath}_{\bf N}){\bf V}_{\bf N}^T \equiv {\bf U}_{\bf M}{\bf Z}^*{\bf V}_{\bf N}^T.
	\]
	Similarly, we define $\hat{{\bf Z}}$ by
	\[
	{\bf M}\hat{{\bf X}}{\bf N} \equiv {\bf U}_{\bf M}\hat{{\bf Z}}{\bf V}_{\bf N}^T.
	\]
	Then, we have that
	\begin{align*}
	&({\bf S}_{{\bf M}}{\bf U}_{\bf M})^T({\bf S}_{{\bf M}} {\bf U}_{\bf M})\hat{{\bf Z}}({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T \\
	&=({\bf S}_{{\bf M}}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf A}{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T\\
	&=({\bf S}_{{\bf M}}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}({\bf A}^\vdash+{\bf U}_{\bf M}{\bf Z}^*{\bf V}_{\bf N}^T){\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T.
	\end{align*}
	Hence, we have
	\begin{align*}
	({\bf S}_{{\bf M}}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf A}^\vdash{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T = & ({\bf S}_{{\bf M}}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}(\hat{{\bf Z}}-{\bf Z}^*){\bf V}_{\bf N}^T{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T \\
	=&  ({\bf S}_{{\bf M}}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}{\bf Z}{\bf V}_{\bf N}^T{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T,
	\end{align*}
	where we define ${\bf Z} = \hat{{\bf Z}}-{\bf Z}^*$.

	Since ${\bf S}_{\bf M}{\bf U}_{\bf M}$ is of full column rank and $s_c\geq c\geq \rho_c$, where $\rho_c$ is the rank of ${\bf M}$, we have that $({\bf S}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}$ is nonsingular, Similarly, ${\bf V}_{\bf N}^T{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T$ is nonsingular. We obtain
	\begin{align*}
	{\bf Z} = [({\bf S}_{\bf M}\M)^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}]^{-1}[({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf A}^\vdash{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T] [{\bf V}_{\bf N}^T{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T]^{-1}
	\end{align*}
	and thus
	\begin{align*}
	\|{\bf Z}\|_F \leq& \|[({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}]^{-1}\|_2\|[{\bf V}_{\bf N}^T{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T]^{-1}\|_2\|[({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf A}^\vdash{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T]\|_F\\
	=& \sigma_{\min}^{-2}({\bf S}_{\bf M}{\bf U}_{\bf M})\sigma_{\min}^{-2}({\bf S}_{\bf N}{\bf V}_{\bf N})\cdot\|[({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf A}^\vdash{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T]\|_F.
	\end{align*}

	We can expand $\|{\bf A}-{\bf M}\hat{{\bf U}}{\bf N}\|_F^{2}$ as follows:
	\begin{align}
	&\|{\bf A}-{\bf M}\hat{{\bf X}}{\bf N}\|_F^{2}  \notag\\
	=& \|{\bf A}-{\bf M}{\bf X}^*{\bf N}+{\bf M}{\bf X}^*{\bf N}-{\bf M}\hat{{\bf X}}{\bf N}\|_F^2 \notag\\
	=& \|{\bf A}-{\bf M}{\bf X}^*{\bf N}\|_F^{2}+\|{\bf M}{\bf X}^*{\bf N}-{\bf M}\hat{{\bf X}}{\bf N}\|_F^{2} + 2\mathrm{tr}[({\bf A}-{\bf M}{\bf X}^*{\bf N})^T({\bf M}{\bf X}^*{\bf N}-{\bf M}\hat{{\bf X}}{\bf N})]  \label{eq:orth}\\
	=& \|{\bf A}-{\bf M}{\bf X}^*{\bf N}\|_F^{2} + \|{\bf U}_{\bf M}{\bf Z}{\bf V}_{\bf N}^{T}\|_F^{2}, \notag
	\end{align}
	where \eqref{eq:orth} is because
	\begin{align*}
	&\mathrm{tr}[({\bf A}-{\bf M}{\bf X}^*{\bf N})^T({\bf M}{\bf X}^*{\bf N}-{\bf M}\hat{{\bf X}}{\bf N})] \\
	=& \mathrm{tr}[(({\bf I}_m-{\bf M}\M^{\dagger}){\bf A}+({\bf M}\M^{\dagger}){\bf A}({\bf I}_n-{\bf N}^\dagger{\bf N}))^T{\bf M}({\bf X}^*-\hat{{\bf X}}){\bf N}]\\
	=& \mathrm{tr}[{\bf A}^{T}({\bf I}_m-{\bf M}\M^{\dagger}){\bf M}({\bf X}^*-\hat{{\bf X}}){\bf N}] + \mathrm{tr}[{\bf N}({\bf I}_n-{\bf N}^{\dagger}{\bf N}){\bf A}^T{\bf M}\M^{\dagger}{\bf M}({\bf X}^*-\hat{{\bf X}})] \\
	=& 0.
	\end{align*}
	Now we need to bound $\|{\bf Z}\|_F$. First, we express ${\bf A}^{\vdash}$ as follow
	\begin{align*}
	{\bf A}^{\vdash} = {\bf A} - {\bf M}{\bf X}^*{\bf N} = {\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A} + {\bf U}_{\bf M}{\bf U}_{\bf M}^T{\bf A}{\bf V}_{\bf N}^{\perp}({\bf V}_{{\bf N}}^{\perp})^T.
	\end{align*}
	Then, we have
	\begin{align*}
	&\|[({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf A}^\vdash{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T]\|_F \\
	\leq& \|({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T\|_F + \|({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}{\bf U}_{\bf M}^T{\bf A}{\bf V}_{\bf N}^{\perp}({\bf V}_{{\bf N}}^{\perp})^T{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T\|_F.
	\end{align*}

	Since ${\bf U}_{\bf M}^T{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T = {\bf 0}$, and	by Theorem~\ref{thm:multi}, we have
	\begin{align}
	&\|({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T\|_F \notag\\
	= & \|{\bf U}_{\bf M}^T{\bf S}_{\bf M}^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T - {\bf U}_{\bf M}^T{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T\|_F\notag\\
	\leq&\frac{\sqrt{\epsilon}}{\sqrt{c}}\|{\bf U}_{\bf M}^T\|_F\|{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T\|_F\label{eq:app_mult}\\
	=& \sqrt{\epsilon}\|{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}{\bf S}_{{\bf N}}^T{\bf S}_{\bf N}{\bf V}_{\bf N}\|_F \notag
	\end{align}
	where \eqref{eq:app_mult} follows from the condition that \eqref{eq:multi} holds with error parameter $\sqrt{\epsilon/c}$.
	Also, by \eqref{eq:multi} holds with error parameter $\sqrt{\epsilon/r}$, we have
	\begin{align*}
		\|{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}{\bf S}_{{\bf N}}^T{\bf S}_{\bf N}{\bf V}_{\bf N} - {\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}{\bf V}_{\bf N}\|_F \leq \frac{\sqrt{\epsilon}}{\sqrt{r}}\|{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}\|_F\|{\bf V}_{\bf N}\|_F = \sqrt{\epsilon}\|{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}\|_F
	\end{align*}
	Therefore, we obtain
	\begin{equation}
	\|{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}{\bf S}_{{\bf N}}^T{\bf S}_{\bf N}{\bf V}_{\bf N}\|_F \leq (1+\sqrt{\epsilon})\|{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}\|_F.
	\end{equation}
	Now, we get
	\begin{align*}
	\|({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T\|_F \leq 2\sqrt{\epsilon}\|{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}\|_F.
	\end{align*}

	For $\|({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}{\bf U}_{\bf M}^T{\bf A}{\bf V}_{\bf N}^{\perp}({\bf V}_{{\bf N}}^{\perp})^T{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T\|_F$, we have,
	\begin{align}
	&\|({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}{\bf U}_{\bf M}^T{\bf A}{\bf V}_{\bf N}^{\perp}({\bf V}_{{\bf N}}^{\perp})^T{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T\|_F\notag\\
	\leq&\|({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf U}_{\bf M}{\bf U}_{\bf M}^T\|_2\|{\bf A}{\bf V}_{\bf N}^{\perp}({\bf V}_{{\bf N}}^{\perp})^T{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T\|_F\notag\\
	\leq &(1+0.5)\|{\bf A}{\bf V}_{\bf N}^{\perp}({\bf V}_{{\bf N}}^{\perp})^T{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T\|_F\label{eq:uc_embed}\\
	\leq&2\frac{\sqrt{\epsilon}}{\sqrt{r}}\|{\bf V}_{\bf N}\|_F\|{\bf A}{\bf V}_{\bf N}^{\perp}({\bf V}_{{\bf N}}^{\perp})^T\|_F \label{eq:r_fnorm_kp}\\
	\leq&2\sqrt{\epsilon}\|{\bf A}{\bf V}_{\bf N}^{\perp}({\bf V}_{{\bf N}}^{\perp})^T\|_F, \notag
	\end{align}
	where \eqref{eq:uc_embed} follows from the property of subspace embedding property with error parameter $0.5$ and \eqref{eq:r_fnorm_kp} is because the condition that Equation~\eqref{eq:multi} holds with error parameter $\sqrt{\epsilon/r}$.

	Thus, we have
	\begin{align*}
	&\|[({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf A}^\vdash{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T]\|_F \\
	\leq& 2\sqrt{\epsilon}\|{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}\|_F +2\sqrt{\epsilon}\|{\bf A}{\bf V}_{\bf N}^{\perp}({\bf V}_{{\bf N}}^{\perp})^T\|_F\\
	\leq&4\sqrt{\epsilon}\|{\bf A}-{\bf M}{\bf X}^{*}{\bf N}\|_F,
	\end{align*}
	where last inequality follow from the fact that $\|{\bf U}_{\bf M}^{\perp}({\bf U}_{\bf M}^{\perp})^T{\bf A}\|_F \leq \|{\bf A}-{\bf M}{\bf X}^{*}{\bf N}\|_F$ and $\|{\bf A}{\bf V}_{\bf N}^{\perp}({\bf V}_{{\bf N}}^{\perp})^T\|_F\leq \|{\bf A}-{\bf M}{\bf X}^{*}{\bf N}\|_F$.
	Thus, we have,
	\begin{align*}
	{\bf Z} \leq & \sigma_{\min}^{-2}({\bf S}_{\bf M}{\bf U}_{\bf M})\sigma_{\min}^{-2}({\bf S}_{\bf N}{\bf V}_{\bf N})\cdot\|[({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf A}^\vdash{\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T]\|_F \\
	\leq & 4\sqrt{\epsilon}\|[({\bf S}_{\bf M}{\bf U}_{\bf M})^T{\bf S}_{{\bf M}}{\bf A}^\vdash {\bf S}_{{\bf N}}^T({\bf V}_{\bf N}^{T}{\bf S}_{\bf N}^T)^T]\|_F \\
	\leq &16\sqrt{\epsilon}\|{\bf A}-{\bf M}{\bf X}^{*}{\bf N}\|_F,
	\end{align*}
	where the second inequality is because subspace embedding properyt of ${\bf S}_{\bf M}$ and ${\bf S}_{\bf N}$. It holds that
	\[
	\sigma^{-2}_{\min} ({\bf S}_{\bf M}{\bf U}_{\bf M}) \leq \frac{1}{(1-0.5) \sigma_{\min}^2({\bf U}_{\bf M})} = 2,
	\]
	and
	\[
	\sigma^{-2}_{\min} ({\bf S}_{\bf N}{\bf V}_{\bf N}) \leq \frac{1}{(1-0.5) \sigma_{\min}^2({\bf V}_{\bf N})} = 2.
	\]
	Finally, we reach that
	\begin{align*}
	\|{\bf A}-{\bf M}\hat{{\bf X}}{\bf N}\|_F^{2}  = & \|{\bf A}-{\bf M}{\bf X}^*{\bf N}\|_F^{2}+\|{\bf M}{\bf X}^*{\bf N}-{\bf M}\hat{{\bf X}}{\bf N}\|_F^{2} \\
	=& \|{\bf A}-{\bf M}{\bf X}^*{\bf N}\|_F^{2} + \|{\bf U}_{\bf M}{\bf Z}{\bf V}_{\bf N}^{T}\|_F^{2} \\
	\leq& \|{\bf A}-{\bf M}{\bf X}^*{\bf N}\|_F^{2} + 256\epsilon\|{\bf A}-{\bf M}{\bf X}^*{\bf N}\|_F^{2}\\
	=&(1+256\epsilon)\|{\bf A}-{\bf M}{\bf X}^*{\bf N}\|_F^{2}.
	\end{align*}
	By rescaling $\epsilon$, we get the result.
\end{proof}

\section{Conclusion}

In this paper we have studied fast generalized matrix approximation using sketching techniques. We have given a tighter bound of reduced dimensions $s_c$ and $s_r$ to reach a $(1+\epsilon)$ error bound and obtained an ${\mathcal O}(\mathrm{nnz}({\bf M})+\mathrm{nnz}({\bf N}))$ generalized matrix approximation.

\bibliographystyle{plainnat}


Random Covering Sets in Metric Space with Exponentially Mixing Property

Abstract: Let $\{B(\xi_n,r_n)\}_{n\ge1}$ be a sequence of random balls whose centers $\{\xi_n\}_{n\ge1}$ is a stationary process, and $\{r_n\}_{n\ge1}$ is a sequence of positive numbers decreasing to 0. Our object is the random covering set $E=\limsup\limits_{n\to\infty}B(\xi_n,r_n)$, that is, the points covered by $B(\xi_n,r_n)$ infinitely often. The sizes of $E$ are investigated from the viewpoint of measure, dimension and topology.

\section{Introduction}

Let $(X,d)$ be a complete metric space. Given a sequence of points $\{x_n\}_{n\ge1}$ in $X$, let $\{r_n\}_{n\ge1}$ be a sequence of positive numbers decreasing to 0. A general covering problem concerns the sets
\[\limsup\limits_{n\to\infty}B(x_n,r_n)=\bigcap_{k=1}^{\infty}\bigcup_{n=k}^{\infty}B(x_n,r_n),\]
where $B(x_n,r_n)$ denotes the ball centered at $x_n$ of radius $r_n$. Many authors have investigated the size and structure of these limsup sets.

One of classical models is to let $\{x_n\}_{n\ge1}$ be a sequence of random variables on $(\Omega,\mathcal{B},\mathbb{P})$. The limsup set $E=\bigcap_{k=1}^{\infty}\bigcup_{n=k}^{\infty}B(x_n,r_n)$ is usually called random covering set, since it consists of the points covered by random balls $\{B(x_n,r_n)\}$ infinitely many times. The study of random covering sets has a long and convoluted history~\cite{Durand10,Eks19,EJS18,Fan02,LSX13,LS14,Per15}.

In 1956, Dvoretzky \cite{Dvor56} called the attention on the study of such random covering sets in the circle $\mathbb{T}$ with $\{x_n\}_{n\ge1}$ being independent and uniformly distributed. He asked the question when $E=\mathbb{T}$ a.s. or not. There was a series of contributions.  In 1971, L. Shepp~\cite{Shepp72} gave a sufficient and necessary condition: $E=\mathbb{T}$ a.s. if and only if $\sum_{n=1}^{\infty}(1/n^2)\exp(r_1+\dots+r_n)=\infty$. Kahane \cite{Kahane85} proved that $E$ is a.s. dense on $\mathbb{T}$ and moreover of second category. The applications of the Borel-Cantelli lemma and Fubini's theorem give that the Lebesgue measure of $E$ is 0 or 1 a.s. according to the convergence or divergence of the series $\sum_{n=1}^{\infty}r_n$. Many authors have studied the Hausdorff dimension and other fractal properties of random covering set $E$. Fan and Wu \cite{FanW04} considered the special case $r_n=a/{n^{\alpha}}$ with $a>0$ and $\alpha>1$. They proved that $\dim_{\rm{H}} (E)=1/{\alpha}$ a.s., where $\dim_{\rm{H}}$ denotes Hausdorff dimension. Durand \cite{Durand10} considered a general sequence $\{r_n\}_{n\ge1}$ and proved $\dim_{\rm{H}}E=\inf\{s>0\colon \sum_{n=1}^{\infty}r_n^s<\infty\}$ and $\dim_{\rm{P}} E=1$ a.s., where $\dim_{\rm{P}}$ denotes packing dimension.

Many variations of the random covering problems have been addressed by many mathematicians. For example, J\"arvenp\"a\"a, $et~al.$ \cite{JKLS14} covered the torus by self-affine sets instead of balls. Feng, $et~al.$ \cite{FJV18} extended it to any open sets. The covering model in Ahlfors regular metric space has been studied in \cite{JKLSX17}.

Instead of a sequence of random variables, many authors considered the case that $\{x_n\}_{n\ge1}$ is the orbit of a dynamical system $(X,T)$. Define the dynamical covering set $E(x,r_n)=\bigcap_{k=1}^{\infty}\bigcup_{n=k}^{\infty}B(T^{n-1}x,r_n).$
Fan, Schmeling and Troubetzkoy~\cite{FanScT13} computed the Hausdorff dimension of $E(x,r_n)$ when $X$ is the unit interval and $T\colon x\mapsto 2x$ $(\bmod\thinspace1)$. In 2013, Liao and Seuret~\cite{LiaSe13} successfully enlarged the setting to finite expanding Markov map with Gibbs measure $m$. They proved that if $r_n=n^{-\alpha}$, then $\dim_{\rm{H}} E(x,r_n)=1/\alpha$ a.e. provided that $1/\alpha$ is not larger than the dimension of the measure $m$. Pesson and Rams \cite{PR17} considered more general piecewise expanding maps than Markov maps. In 2017, Wang, Wu and Xu~\cite{WanWuX17} considered the dynamical covering problem on the triadic cantor set.

In this paper, we consider the covering set with $\{\xi_n\}_{n\ge1}$ which is a sequence of points in a compact metric space $(X,d)$, chosen randomly. And independence of $\{\xi_n\}_{n\ge1}$ is not necessary. The purpose of this article is to study some properties of random covering set in general probabilistic setting, including measure, density, fractal dimensions and so on. Next we will state the main results and provide some discussions. Section 2 is devoted to the proof of the results. In the last section 3, we will give an application to the dynamical covering problem.

Let $\{\xi_n\}_{n\ge1}$ be a stationary process on a probability space $(\Omega,\mathcal{B},\mathbb{P})$ and take values in a compact metric space $(X,d)$. Let $\mu$ be the probability measure defined by
\begin{equation}\label{eqmu}
\mu(A)=\mathbb{P}(\xi_1\in A)
\end{equation}
for any Borel set $A \subset X$. Assume that $X$ is the support of $\mu$.

We say that $\{\xi_n\}_{n\ge1}$ is $exponentially~ mixing$ if for any $n\ge1$, there exist two constants $c>0$ and $0<\gamma<1$ such that
\[\big|\mathbb{P}(\xi_1\in A|D)-\mathbb{P}(\xi_1\in A)\big|\le c\gamma^{n}\]
holds for any ball $A\subset X$ and $D\in\mathcal{B}^{n+1}$, where $\mathcal{B}^{n+1}$ is the sub-$\sigma$-field generated by $\{\xi_{n+i}\}_{i\ge1}$.

Recall that a Borel measure $\mu$ is $\textit{Ahlfors}$ $s$-$\textit{regular}$ $(0< s < \infty)$ if there exists a constant $0 < C <\infty$ such that
\begin{equation}\label{eqah}
C^{-1}r^s\le\mu\big(B(x,r)\big)\le Cr^s
\end{equation}
holds for all $x\in X$ and $0<r\le\diam X$, where $\diam X$ is the diameter of X.

A metric space $X$ is said to be $\textit{Ahlfors}$ $s$-$\textit{regular}$ if there exists a Borel measure on $X$ satisfying formula \eqref{eqah}.

Let $\{r_n\}_{n\ge1}$ be a sequence of positive real numbers decreasing to zero. For every $n\ge 1$, denote $B_n:=B(\xi_n,r_n)$. Define
\[E:=\limsup_{n\to\infty}B_n=\{y\in X\colon y\in B_n ~\text{for~ infinitely ~many}~n\ge1\}.\]
The set $E$ is a $random ~covering~ set$ and consists of the points which are covered by $\{B_n\}_{n\ge1}$ infinitely often (i.o. for short).

\begin{thm}\label{thm1}
	Let $\{\xi_n\}_{n\ge1}$ be exponentially mixing and the probability measure $\mu$ defined in~\eqref{eqmu} be Ahlfors s-regular. Then we have
	\[\mu\bigl(E\bigr)=
	\begin{cases}
	0&  \text{if $\sum_{n=1}^{\infty}r_n^s<\infty$}\\
	1&  \text{if $\sum_{n=1}^{\infty}r_n^s=\infty$}
	\end{cases} \quad a.s.\]
\end{thm}

A dimension function $f\colon \mathbb{R}^{+}\to\mathbb{R}^{+}$ is a continuous and non-decreasing function such that $f(r)\to0$ as $r\to0$. If there exists a constant $\eta>1$ such that for $r>0$, $f(2r)\le\eta f(r)$, then we say that function $f$ is $doubling$.

\begin{thm}\label{thm2}
	Let $\{\xi_n\}_{n\ge1}$ be exponentially mixing and the probability measure $\mu$ defined in~\eqref{eqmu} be Ahlfors s-regular. Suppose that $f$ is a doubling dimension function with $f(r)/r^s$ being nondecreasing as $r\to0$. Then, with probability one, for any ball $B$ of $X$,
\[\Hm^f(E\cap B)=
\begin{cases}
	0&  \text{if $\sum_{n=1}^{\infty}f(r_n)<\infty$}\\
	\Hm^f(B)&  \text{if $\sum_{n=1}^{\infty}f(r_n)=\infty$}
	\end{cases} \quad. \]
	Futhermore,
	\[\dim_{\rm{H}} E=\alpha \quad a.s.,\]
	where $\alpha=\inf\{t\le s\colon \sum_{n=1}^{\infty}r_n^t<\infty\}$.
\end{thm}

\begin{thm}\label{thm3}
	Let $\{\xi_n\}_{n\ge1}$ be exponentially mixing and the probability measure $\mu$ be Ahlfors~ s-regular. Then random covering set $E$ is dense in X almost surely.
\end{thm}

\begin{cor}\label{cor1}
	Assume that the conditions of Theorem~\ref{thm2} hold. We have $\dim_{\rm{B}} E=s$ almost surely, where $\dim_{\rm{B}}$ denotes box dimension.
\end{cor}

Recall that a set is called $residual$ if the complement of the set is a first category set.

\begin{thm}\label{thm4}
	Let $\{\xi_n\}_{n\ge1}$ be exponentially mixing and the probability measure $\mu$ be Ahlfors~ s-regular. Then random covering set $E$ is a residual set almost surely. And $E$ is also a set of second category almost surely. In particular $\dim_{\rm{P}} E=s$ almost surely.
\end{thm}

\begin{rem}\label{rem1}
         ~
	\begin{enumerate}[(i)]
        \item J. Heinonen~\cite{Hein01} proved that, if $X$ is a metric space admitting a Borel measure $\mu$ which is $\textit{Ahlfors}$ $s$-$\textit{regular}$ $(0< s < \infty)$, then $X$ has Hausdorff dimension precisely $s$.
        \item From the proof of Corollary~\ref{cor1}, we see that the box dimension of the space we considered is $s$.
        \item By Cutler~\cite[Theorem 3.16]{Cutler95}, we derive that the packing dimension of the space we considered is $s$.
        \end{enumerate}
\end{rem}

\section{Proofs of the main results}

In this section we proved Theorems~\ref{thm1}~--~\ref{thm4} and Corollary~\ref{cor1}.

\begin{lem}\label{lem3}
	Suppose that $\{\xi_n\}_{n\ge1}$ is an exponentially mixing stationary process. Let $\{h_n\}_{n\ge1}$ be a decreasing sequence. For any point $y\in X$, if the series $\sum_{n=1}^{\infty}h_n^s$ diverges, we have $\mathbb{P}(\xi_n\in B(y,h_n)~i.o.)=1$.

\end{lem}

\begin{proof}
         Let $y\in X$ and denote $\widetilde{J}_n=\{\omega\in\Omega\colon\xi_n(\omega)\in B(y,h_n)\}$. Let $N\ge1$ and $S_N=\sum_{n=1}^{N}\chi_{\widetilde{J}_n}$, where $\chi$ is the indicator function. Then
	\[\mathbb{E}(S_N)=\sum_{n=1}^N\mathbb{P}(\widetilde{J}_n)=\sum_{n=1}^N\mu\bigl(B(y,h_n)\bigr).\]
	Since $\mu$ is Ahlfors $s$-regular, by (\ref{eqah}), we have
	\[\mathbb{E}(S_N)\ge C^{-1}\sum_{n=1}^N h_n^s\to\infty,\quad \text{as} ~N\to\infty.\]
	 Thus $\sum_{n=1}^{\infty}\mathbb{P}(\widetilde{J}_n)=\lim\limits_{N\to\infty}\mathbb{E}(S_N)=\infty$.

	 By the Paley-Zygmund inequality, for all $0<\lambda<1$, we have
	\begin{equation}\label{eq21}
	\begin{split}
	\mathbb{P}\left(S_N\ge\lambda\mathbb{E}(S_N)\right)&\ge(1-\lambda)^2\frac{{\mathbb{E}}^2(S_N)}{\mathbb{E}(S_N^2)}\\
	&=(1-\lambda)^2\frac{\left(\sum_{n=1}^N{\mathbb{P}(\widetilde{J}_n)}\right)^2}{\mathbb{E}(S_N^2)}.
	\end{split}
	\end{equation}
	Now we estimate $\mathbb{E}(S_N^2)$,
	\begin{equation}\label{eq22}
	\begin{split}
	\mathbb{E}(S_N^2)&=\mathbb{E} \left(\sum_{n=1}^N \chi_{\widetilde{J}_n}\right)^2=\mathbb{E}\bigl(\sum_{n=1}^N \chi_{\widetilde{J}_n}+\sum_{n=1}^N\sum_{\substack{m=1\\ m\ne n}}^N \chi_{\widetilde{J}_n\cap\widetilde{J}_m}\bigr)\\
	&=\sum_{n=1}^N \mathbb{P}(\widetilde{J}_n)+2\sum_{n=1}^N\sum_{m=1}^{n-1}\mathbb{P}(\widetilde{J}_n\cap \widetilde{J}_m).
	\end{split}
	\end{equation}
	Using the stationarity and the exponentially mixing property of the process $\{\xi_n\}$, we get
	\[\begin{split}
	\mathbb{P}(\widetilde{J}_n\cap\widetilde{J}_m)&=\mathbb{P}\{\xi_n\in B(y,h_n),\xi_m\in B(y,h_m)\}=\mathbb{P}\{\xi_1\in B(y,h_m),\xi_{n-m+1}\in B(y,h_n)\}\\
	&\le\mathbb{P}\bigl(\xi_1\in B(y,h_m)\bigr)\mathbb{P}\bigl(\xi_{n-m+1}\in B(y,h_n)\bigr)+c{\gamma}^{n-m}\mathbb{P}\bigl(\xi_{n-m+1}\in B(y,h_n)\bigr)\\
	&=\mathbb{P}(\widetilde{J}_n)\mathbb{P}(\widetilde{J}_m)+c{\gamma}^{n-m}\mathbb{P}(\widetilde{J}_n).
	\end{split}\]
	Hence the equality \eqref{eq22} reads as follows
	\begin{equation}\label{eq23}
	\begin{split}
	\mathbb{E}(S_N^2)&\le\sum_{n=1}^N \mathbb{P}(\widetilde{J}_n)+2\sum_{n=1}^N\sum_{m=1}^{n-1}\left(\mathbb{P}(\widetilde{J}_n)\mathbb{P}(\widetilde{J}_m)+c{\gamma}^{n-m}\mathbb{P}(\widetilde{J}_n)\right)\\
	&\le\sum_{n=1}^N \mathbb{P}(\widetilde{J}_n)+\left(\sum_{n=1}^N\mathbb{P}(\widetilde{J}_n)\right)^2+\frac{2c\gamma}{1-\gamma}\sum_{n=1}^N \mathbb{P}(\widetilde{J}_n)\\
	&\le c'\sum_{n=1}^N\mathbb{P}(\widetilde{J}_n)+\left(\sum_{n=1}^N\mathbb{P}(\widetilde{J}_n)\right)^2,
	\end{split}
	\end{equation}
	where $c'$ is a constant. Combining~~\eqref{eq21} and~~\eqref{eq23}, we derive that
	\begin{equation}\label{eq24}
	         \mathbb{P}\left(S_N\ge\lambda\mathbb{E}(S_N)\right)\ge(1-\lambda)^2\frac{\left(\sum_{n=1}^N\mathbb{P}(\widetilde{J}_n)\right)^2}{c'\sum_{n=1}^N\mathbb{P}(\widetilde{J}_n)+\left(\sum_{n=1}^N\mathbb{P}(\widetilde{J}_n)\right)^2}\to1,
	 \end{equation}
	as $N\to\infty$ and $\lambda\to0$ due to the divergence of the series $\sum_{n=1}^{\infty}\mathbb{P}(\widetilde{J}_n)$.
	We notice that
	\[\{\omega\in\widetilde{J}_n~i.o.\}=\{\lim\limits_{N\to\infty}S_N=\infty\}\supset\{S_N\ge\lambda\mathbb{E}(S_N)\}.\]
	By \eqref{eq24} we have
	\[\mathbb{P}(\widetilde{J}~i.o.)\ge\lim\limits_{N\to\infty}\mathbb{P}\bigl(S_N\ge\lambda\mathbb{E}(S_N)\bigr)=1,\]
	which derives $\mathbb{P}(\widetilde{J}~i.o.)=1$.
\end{proof}

\begin{proof}[Proof of Theorem~\ref{thm1}]
	First we show that $\mu\bigl(E\bigr)=0$ for any $\omega\in\Omega$ if $\sum_{n=1}^{\infty}r_n^s<\infty$.

	Since $\mu$ is Ahlfors $s$-regular, by~\eqref{eqah}, we have $\sum_{n=1}^{\infty}\mu(B_n)\le C\sum_{n=1}^{\infty}r_n^s<{\infty}$. By the Borel-Cantelli lemma, $\mu(\limsup\limits_{n\to\infty} B_n)=0$, that is $\mu\bigl(E\bigr)=0$.

	Now we consider the divergence case. Let $y\in X$ and
	\[F(y)=\{\omega\in \Omega\colon \xi_n(\omega)\in B(y,r_n)~i.o.\}.\]
	By lemma~\ref{lem3}, we have $\mathbb{P}\bigl(F(y)\bigr)=1$. Since $y\in E\Leftrightarrow \omega\in F(y)$, applying Fubini's theorem gives
	\[
	\begin{split}
	\mathbb{P}\left(\mu(E)\right)&=\int\int1_{E}(y)d\mu(y)d\mathbb{P}(\omega)=\int\int1_{F(y)}(\omega)d\mathbb{P}(\omega)d\mu(y)\\
	&=\int\mathbb{P}\bigl(F(y)\bigr)d\mu(y)=1.
	\end{split}
	\]
	Hence, for $\mathbb{P}$-almost all $\omega$, $\mu\bigl(E\bigr)=1$.
\end{proof}

	We recall a general case of the mass transference principle~\cite[Theorem 3]{BerVel06}, suitable for the proof of our Theorem~\ref{thm2}.

	Let $(Y,\rho)$ be a locally compact metric space. Let $g$ be a doubling dimension function and suppose
	there exist  constants $0<c_1<1<c_2<\infty$ and $r_0>0$ such that
         \[c_1g\bigl(r(B)\bigr)\le\Hm^g(B)\le c_2g\bigl(r(B)\bigr),\]
         for any ball $B=B(x,r)$ with $x\in Y$ and $r\le r_0$. Next, given a dimension function $f$ and a ball $B=B(x,r)$ we define $B^f:=B\bigl(x,g^{-1}f(r)\bigr)$.

         \begin{thm}[Beresnevich--Velani]\label{thm21}
	Let $(Y,\rho)$ and $g$ be as above and let $\{B_i\}_{i\in\mathbb{N}}$ be a sequence of balls in $Y$ with $r(B_i)\to0$ as $i\to\infty$. Let $f$ be a dimension function such that $f(r)/g(r)$ is monotonic and suppose that for any
	ball $B$ in $Y$
         \[\Hm^g(B\cap \limsup\limits_{i\to \infty}B_i^f)=\Hm^g(B).\]
         Then, for any ball $B$ in $Y$
         \[\Hm^f(B\cap \limsup\limits_{i\to \infty}B_i^g)=\Hm^f(B).\]
         \end{thm}

\begin{proof}[Proof of Theorem~\ref{thm2}]
	For any $\delta>0$, there is an integer $n_0\ge1$ such that $0<2r_n<\delta$ for any $n\ge n_0$. Since $f(r)/r^s$ is increasing as $r\to0$ and $E\subset\bigcup_{n=n_0}^{\infty}B_n$, we have
\[\Hm_{\delta}^f(E)\le \sum_{n=n_0}^{\infty}f(2r_n)\le2^s\sum_{n=n_0}^{\infty}f(r_n).\]
If the series $\sum_{n=1}^{\infty}f(r_n)$ converges, then $\sum_{n=n_0}^{\infty}f(r_n)\to0$ as $\delta\to0$ since it deduces that $n_0\to\infty$. Therefore $\Hm^f(E)=\lim\limits_{\delta\to0}\Hm_{\delta}^f(E)=0$ for any $\omega\in\Omega$.

For the divergence part, denote $B_n^f=B\bigl(\xi_n,\left(f(r_n)\right)^{1/s}\bigr)$. By Theorem \ref{thm1}, we obtain
\[\mu\bigl(\limsup_ {n\to\infty}B_n^f\bigr)=1 ~\text{a.s.},\]
since
\[\sum_{n=1}^{\infty}\left(\left(f(r_n)\right)^{1/s}\right)^s=\sum_{n=1}^{\infty}f(r_n)=\infty.\]
By Theorem \ref{thm21}, we have
\[\Hm^f(E\cap B)= \Hm^f(B)~\text{a.s.}.\]
\end{proof}

For proving Theorem~\ref{thm3}, \ref{thm4} and Corollary~\ref{cor1} we will use the following elementary lemma, whose proof can be found in \cite{JRM00}.

\begin{lem}\label{lem1}
If the metric space $(X,d)$ is compact, then $X$ is bounded, separated and complete.
\end{lem}

\begin{proof}[Proof of Theorem~\ref{thm3}]
	The compact metric space $(X,d)$ is separated, hence there exists a countable dense subset denoted by $A$. Letting $\mathcal{A}=\{B(a,1/k)|a\in A,k\ge1\}$, now we prove that $\mathcal{A}$ is a countable base of $(X,d)$.

    For any open subset $U$ in $X$ and $x\in U$, we can find a $\epsilon_x>0$ such that $B(x,\epsilon_x)\subset U$. There exists $k_x\ge1$ with $2/{k_x}<\epsilon_x$. Since $A$ is dense in $X$, we can choose $a_x\in A$ satisfying $d(x,a_x)<1/{k_x}$. Hence $x\in B(a_x,1/{k_x})\subset B(x, \epsilon_x)\subset U $. It yields that $U=\bigcup\limits_{x\in U}\{x\}\subset\bigcup\limits_{x\in U}B(a_x,1/{k_x})\subset U$, that is $U=\bigcup\limits_{x\in U}B(a_x,1/{k_x})$, where $B(a_x,1/{k_x})\in \mathcal{A}$. Therefore $\mathcal{A}$ is a countable base of $(X,d)$, since $\{B(a_x,1/{k_x})|x\in U\}$ is at most countable.

	For any $B=B(a,r)\in\mathcal{A}$, we will show $\mathbb{P}(E\cap B\ne\varnothing)=1$.

	Denote $\widetilde{A}_n=\{\omega\in \Omega\colon B_n(\omega)\cap B\ne\varnothing\}$. We notice that $\widetilde{A}_n=\{\xi_n\in B(a,r+r_n)\}$.~Then we have
	\[\sum_{n=1}^N (r+r_n)^s\ge \sum_{n=1}^N r^s\to\infty, \quad\text{as}~ N\to\infty.\]
	 Thus the series $\sum_{n=1}^{\infty}(r+r_n)^s$ diverges. Hence $\mathbb{P}(E\cap B\ne\varnothing)=\mathbb{P}(\widetilde{A}_n~i.o.)=1$ due to lemma~\ref{lem3}.

	For convenience, we denote $\mathcal{A}=\{B^i\}_{i\ge1}$. For any $B^i\in\mathcal{A}$, let $A_i=\{\omega| B^i \cap E =\varnothing\}$, then $\mathbb{P}(A_i)=0$. Thus
	\[\mathbb{P}\{\omega|E\cap B^i=\varnothing \text{~for~some}~i\ge1\}=\mathbb{P}\bigl(\bigcup_{i\ge1}A_i\bigr)\le\sum_{i\ge1}\mathbb{P}(A_i)=0.\]
	So there exists a null probability event outside which for all balls $B^i$ in $\mathcal{A}$, we have $E\cap B^i\ne \varnothing$. For any $x\in X$ and open set $U\ni x$, there exist $\{B^{i_n}\}_{n\ge1}\subset \mathcal{A}$ with $U=\bigcup_{n\ge1}B^{i_n}$. Hence \[
	\mathbb{P}\{E\cap U\ne\varnothing\}\ge\mathbb{P}\{\omega|E\cap B^i\ne\varnothing \text{~for~any}~i\ge1\}=1.\]
	 Therefore $E$ is dense in $X$ a.s..
\end{proof}

\begin{proof}[Proof of Corollary~\ref{cor1}]
	Since $X$ is bounded, let $\{y_k\}_{k=1}^n$ be a maximal $r$-separated set of $X$ $(r>0)$. Then the balls $B(k)=B(y_k,r/3)$ are disjoint, that is $B(k)\cap B(j)=\varnothing$ for any $ k\ne j$. Thus, by~\eqref{eqah}
	\[C^{-1}3^{-s}nr^s\le \mu\bigl(B(1)\bigr)+\dots+\mu\bigl(B(n)\bigr)\le1.\]
	On the other hand, the balls $\{3B(k)\}^n_{k=1}$ cover $X$ where $3B(k)=B(y_k,r)$, so
	\[1\le\mu\bigl(3B(1)\bigr)+\dots+\mu\bigl(3B(n)\bigr)\le Cnr^s.\]
	Combining these estimates, it follows that $\dim_{\rm{B}} X=s$. From Theorem~\ref{thm2}, we have $\dim_{\rm{B}} E=\dim_{\rm{B}}X=s~a.s.$.
\end{proof}

\begin{proof}[Proof of Theorem~\ref{thm4}]
	First we show that $E$ is a residual set a.s..

	Now we check if $E^{\mathrm{C}}=\bigcup_{N=1}^{\infty}\bigcap_{n=N}^{\infty} B_n^{\mathrm{C}}$ is a first category set almost surely. Denote $E_N^{\mathrm{C}}=\bigcap_{n=N}^{\infty} B_n^{\mathrm{C}}$. We suppose that there exists $N_1\ge1$ such that $E_{N_1}^{\mathrm{C}}$ is not sparse. That means there is a ball $B(y,r)$ with $r>0$ satisfying
	\[\overline{B}(y,r)\subset \overline{E_{N_1}^{\mathrm{C}}}=E_{N_1}^{\mathrm{C}}=X\backslash{\bigcup_{n=N_1}^{\infty}B_n}.\]
	 Hence $B(y,r)\cap{\bigcup_{n=N_1}^{\infty}B_n}=\varnothing$. However $ \bigcup_{n=N_1}^{\infty}B_n$ is dense in $X$ a.s., since $E\subset\bigcup_{n=N_1}^{\infty}B_n$ is dense in $X$ a.s., which implies $B(y,r)\cap \bigcup_{n=N_1}^{\infty}B_n\ne\varnothing$. Contradiction. Hence $E^{\mathrm{C}}$ is a set of first category almost surely. It derives that $E$ is a residual set a.s..

	From Lemma \ref{lem1}, we know the space $X$ is complete. Then by Baire's category theorem, $X$ is a set of second category. We have shown that $E^{\mathrm{C}}$ is a first category set a.s.. Thus $E$ is second category a.s.. It follows that $\dim_{\rm{P}} E=\dim_{\rm{P}} X=s$ a.s..
\end{proof}

\section{dynamical covering problem}

Our results can be applied to the dynamical covering problem as follows.

We say that a metric measure preserving system (m.m.p.s. for short) $(X,\mathcal{B},\mu,T,d)$ is $\textit{exponentially mixing}$ if there exist two constants $c>0$ and $0<\gamma<1$ such that
\[|\mu(E|T^{-n}F)-\mu(E)|\le c\gamma^n~~~~~(n\ge1)\]
holds for any ball $E$ and any measurable set $F\in\mathcal{B}$ with $\mu(F)>0$. Here $\mu(A|B)$ denotes $\frac{\mu(A\cap B)}{\mu(B)}$. Sometimes we say $\mu$ is exponentially mixing.

We assume that $(X,d)$ is compact, endowed with a Borel probability measure $\mu$ which is Ahlfors $s$-regular $(0<s<\infty)$. Define the $\textit{dynamical covering set}$ as
\[E(x,l_n)=\bigl\{y\in X\colon d(T^n x,y)<l_n~\text{for~infinitely ~many}~n\ge0\bigr\},\]
where $x\in X$ and $\{l_n\}_{n\ge0}$ is a sequence of positive real numbers which is decreasing to zero. The dynamical covering set $E(x,l_n)=\bigcap_{k=0}^{\infty}\bigcup_{n=k}^{\infty}B(T^nx,l_n)$. We study the size of $E(x,l_n)$ from the viewpoint of measure, dimension and topology.

\begin{thm}\label{thmm}
Let $(X,\mathcal{B},\mu,T,d)$ be an exponentially mixing m.m.p.s. and the measure $\mu$ be Ahlfors s-regular.
	For $\mu$-almost all $x\in X$, we have
	\[\mu\bigl(E(x,l_n)\bigr)=
	\begin{cases}
	0&  \text{if $\sum_{n=0}^{\infty}l_n^s<\infty$}\\
	1&  \text{if $\sum_{n=0}^{\infty}l_n^s=\infty$}
	\end{cases}.\]
\end{thm}

\begin{thm}\label{thmm}
Let $(X,\mathcal{B},\mu,T,d)$ be an exponentially mixing m.m.p.s. and the measure $\mu$ be Ahlfors s-regular. Assume that $f$ is a doubling dimension function with $f(r)/r^s$ being nondecreasing as $r\to0$. Then for any ball $B$ of $X$,
\[\Hm^f(E(x,l_n)\cap B)=
\begin{cases}
	0&  \text{if $\sum_{n=0}^{\infty}f(l_n)<\infty$}\\
	\Hm^f(B)&  \text{if $\sum_{n=0}^{\infty}f(l_n)=\infty$}
	\end{cases} \quad \]
holds for $\mu$-almost all $x\in X$. Furthermore
	\[\dim_{\rm{H}} E(x,l_n)=\alpha ~a.e.,\]
	where $\alpha=\inf\{t\le s\colon \sum_{n=0}^{\infty}l_n^t<\infty\}$.
\end{thm}

\begin{thm}\label{thmd}
	Let $(X,\mathcal{B},\mu,T,d)$ be an exponentially mixing m.m.p.s. and $\mu$ be Ahlfors s-regular. Then the dynamical covering set $E(x,l_n)$ is dense in X a.e..
\end{thm}

\begin{cor}\label{cor5}
	Let $(X,\mathcal{B},\mu,T,d)$ be an exponentially mixing m.m.p.s. and $\mu$ be Ahlfors s-regular. For $\mu$-almost all $x\in X$, we have $\dim_B E(x,l_n)=s$.
\end{cor}

\begin{thm}\label{thmt}
	Let $(X,\mathcal{B},\mu,T,d)$ be an exponentially mixing m.m.p.s. and $\mu$ be Ahlfors s-regular. Therefore $E(x,l_n)$ is a residual set and moreover of second category for $\mu$-almost all $x\in X$. In particular, $\dim_P E(x,l_n)=s$ a.e..
\end{thm}

\begin{rem}\label{rem1}
         Now we give some systems which are exponentially mixing.
         \begin{enumerate}[(i)]
        \item For the doubling map $Tx=2x$ $(\bmod\thinspace1)$ on the interval $[0,1)$, Gibbs measure $\mu$ associated to H\"older potentials is exponentially mixing.
        \item For the $\beta$-shift $T_{\beta}x=\beta x$ $(\bmod\thinspace1)$ on the interval $[0,1)$, the Parry measure is exponentially mixing.
        \item  For the Gauss map $Sx=\{\frac{1}{x}\}$ $(\bmod\thinspace1)$ on the interval $[0,1)$, the Gauss measure is exponentially mixing.
        \end{enumerate}

\end{rem}

\subsection*{Acknowledgements}
This work is  supported by  NSFC 11671151 and Guangdong Natural Science Foundation 2018B0303110005.