On a Faster $R$-Linear Convergence Rate of the Barzilai-Borwein Method

Abstract: The Barzilai-Borwein (BB) method has demonstrated great empirical success in nonlinear optimization. However, the convergence speed of BB method is not well understood, as the known convergence rate of BB method for quadratic problems is much worse than the steepest descent (SD) method. Therefore, there is a large discrepancy between theory and practice. To shrink this gap, we prove that the BB method converges $R$-linearly at a rate of $1-1/\kappa$, where $\kappa$ is the condition number, for strongly convex quadratic problems. In addition, an example with the theoretical rate of convergence is constructed, indicating the tightness of our bound.

\section{Introduction}\label{sec:intro}

The Barzilai-Borwein (BB) method is a simple and efficient gradient algorithm. It takes the same search direction as the classic steepest descent (SD) method \cite{cauchy1847methode}, with a somewhat delicate stepsize rule.
Since its proposal by Barzilai and Borwein \cite{barzilai1988two}, the BB method has been shown to give competitive performances in global optimization \cite{raydan1997barzilai}, and its extensions and variants have been utilized in large numbers of applications including compressed sensing \cite{figueiredo2007gradient}, image restoration \cite{wang2007projected}, sparse reconstruction \cite{wen2010fast}, signal processing \cite{liu2010coordinated}, matrix factorization \cite{huang2015quadratic}, machine learning \cite{tan2016barzilai} and distributed optimization \cite{gao2019geometric}. \\

Despite the great empirical success of the BB method, the non-monotone behaviors of the algorithm make it difficult to perform theoretical analysis.
Most theoretical results of the BB method are derived for strictly convex quadratic objective functions.
The earliest result was presented by Barzilai and Borwein \cite{barzilai1988two}, who provided an $R$-superlinear convergence proof in the $2$-dimensional case. In the general $n$-dimensional case, global convergence was proved by Raydan \cite{raydan1993barzilai}, and an $R$-linear convergence result was given by Dai and Liao \cite{dai2002r}, the proof technique of which was also utilized to show the $R$-linear convergence of other alternative stepsize methods \cite{dai2003alternate, dai2019family}. Dai and Fletcher \cite{dai2005asymptotic} also made an interesting asymptotic analysis, a corollary of which was $R$-superlinear convergence of BB method in the $3$-dimensional case. \\

Although these results demonstrated some good properties
of BB method, our understanding of the convergence rate of BB method
is still very limited.
In fact, the known convergence rate of BB method \cite{raydan1993barzilai,dai2002r}, which is roughly $\left(1-\frac{1}{\kappa}\right)^{1/n}$,
is much worse than the rate of SD method ($ 1 - \frac{2 }{\kappa+1}$, where $\kappa$ is the condition number).
On one hand, this is understandable since in earlier days, researchers are very much interested in the qualitative behavior (linear convergence v.s. superlinear convergence), rather than the specific  convergence rate.
On the other hand, there has been a recent trend of analyzing the rate of convergence in large-scale optimization (see, e.g., \cite{nestrov12,johnson2013accelerating}), since it is believed
that the specific rate can help us better understand these algorithms.
In this context, the current gap between the rate of \cite{dai2002r}
and SD method is not desirable:
considering the excellent practical performance of BB method,
there is a huge discrepancy between theory and practice.
How far can we push the boundary of the theory?

We conjectured that BB method has at least the same rate of convergence as SD methods. Although the conjecture seems not surprising, there
are plenty of examples in optimization area that
a seemingly fast method has very slow worst-case convergence rate.
A classical example is the simplex method
which was shown to have exponential time worst-case complexity \cite{zadeh2009worst} \footnote{More rigorously speaking,
for most common pivoting rules the worst-case
time is exponential, and there is no known (deterministic) variant
that has polynomial time.}.
A more recent example is the cyclic coordinate descent method which was found to have a rate of convergence that can be $O(n)$ times worse than gradient descent methods in the worst case \cite{sun2019worst}.
Similarly, it was unclear a priori whether BB method
has a bad worst-case convergence rate.

In this paper, we prove that
the convergence rate of the BB method is at least as
good as the SD method in the sense of $R$-linear convergence, for quadratic problems. In particular, we prove that for $n$-dimensional strongly convex quadratic functions, the BB method converges $R$-linearly with a rate of $1-\frac{1}{\kappa}$. This rate is indeed comparable to that of the SD method. Moreover, we show that the lower bound of the convergence rate, if we select special (degenerate) initial points, is exactly $1-\frac{1}{\kappa}$ by giving an example in the $n$-dimensional case. This finding indicates that our convergence rate cannot be further improved without additional assumptions.

The outline of this paper is as follows. We first present the notations and introduce the BB method in Section \ref{sec:setting}. In Section \ref{sec:motivation} we provide some intuition about how the BB method proceeds during iterations. This process motivates the proof idea of categorizing modes in our main theorem, which is presented in Section \ref{sec:theorem}. The lower bound example, as well as some discussions, are provided in Section \ref{sec:lower-bound}. Finally, the conclusion is made in Section \ref{sec:conclusion}.

\section{The BB Method for the Quadratic Case}\label{sec:setting}

In this paper, we focus on the BB method for the $n$-dimensional quadratic case. Consider the following quadratic optimization problem:
\begin{equation}\label{eq:problem}
\underset{x\in\mathbb{R}^n}{\min}~f(x)=\frac{1}{2}x^\top Ax-c^\top x
\end{equation}
where $A\in \mathcal{S}^n_{++}$. The classic steepest descent (SD) method takes the negative gradient as the search direction and chooses the stepsize as the minimizer along the search direction. Denote $g_k=\nabla f(x_k)$, then the SD method can be represented as
\begin{equation*}
    x_{k+1}=x_k-\alpha_k^{SD}g_k~~ \text{where}~~ \alpha_k^{SD}=\arg \underset{\alpha>0}{\min}~f(x_k-\alpha g_k).
\end{equation*}

The BB method takes the same search direction as the SD method. Nevertheless, it gives a two-point stepsize rule to automatically determine the stepsize. In particular, the stepsize $\alpha_k$ is selected such that $\alpha_k^{-1}I$ best approximates the Hessian $f$. Denote $s_{k-1}=x_k-x_{k-1}$ and $y_{k-1}=g_k-g_{k-1}$. Through an approach similar to the quasi-Newton method, the BB method solves the following problem
\begin{equation*}
    \arg \underset{\alpha>0}{\min}~\|\alpha^{-1}s_{k-1}-y_{k-1}\|_2,
\end{equation*}
which yields
\begin{equation*}
\alpha_k^{BB}=\frac{s_{k-1}^\top s_{k-1}}{s_{k-1}^\top y_{k-1}}.
\end{equation*}

In the following, we derive some concrete iterative relations for the quadratic case. Without losing generality, we assume $c=0$ since the BB method is invariant under translations. In this case, $g_k=Ax_k$. Therefore, we have
\begin{equation*}
    s_{k-1}=x_k-x_{k-1}=\alpha_{k-1}g_{k-1},~~ y_{k-1}=g_k-g_{k-1}=A(x_k-x_{k-1})=\alpha_{k-1}A g_{k-1}.
\end{equation*}
Thus, the BB stepsizes $\alpha_k$ can be represented by $g_{k-1}$ and $A$:
\begin{equation*}
\alpha_k^{BB}=\frac{g_{k-1}^\top g_{k-1}}{g_{k-1}^\top Ag_{k-1}}.
\end{equation*}
We can further obtain the relation among the gradients $g_{k+1}, g_k$ and $g_{k-1}$ by:
\begin{equation}\label{eq:gradient}
g_{k+1}=Ax_{k+1}=Ax_k-\alpha_k^{BB}Ag_k=g_k-\frac{g_{k-1}^\top g_{k-1}}{g_{k-1}^\top Ag_{k-1}}Ag_k.
\end{equation}

We now compute the eigenvalue decomposition of relation \eqref{eq:gradient}. Without losing generality, assume that $A$ has eigenvalues $0<\lambda_1\leq\cdots\leq\lambda_n$ with the corresponding eigenvectors $v_1, \cdots, v_n$. Note that $v_1, \cdots, v_n$ form an orthogonal unit basis in $\mathbb{R}^n$, so $g_k$ is uniquely decomposed into $g_k=\sum_{i=1}^nd^i_kv_i$ where $d_k^1, \cdots, d_k^n$ are the coefficients. Thus we can also decompose \eqref{eq:gradient} into
\begin{equation}\label{eq:decomposition}
    \begin{aligned}
        \sum_{i=1}^nd^i_{k+1}v_i&=\sum_{i=1}^nd^i_kv_i-\frac{(\sum_{j=1}^nd^j_{k-1}v_j)^\top (\sum_{j=1}^nd^j_{k-1}v_j)}{(\sum_{j=1}^nd^j_{k-1}v_j)^\top A(\sum_{j=1}^nd^j_{k-1}v_j)}\cdot A\sum_{i=1}^nd^i_kv_i\\
        &=\sum_{i=1}^nd^i_kv_i-\frac{(\sum_{j=1}^nd^j_{k-1}v_j)^\top (\sum_{j=1}^nd^j_{k-1}v_j)}{(\sum_{j=1}^nd^j_{k-1}v_j)^\top (\sum_{j=1}^nd^j_{k-1}\lambda_jv_j)}\cdot \sum_{i=1}^nd^i_k\lambda_iv_i\\
        &=\sum_{i=1}^nd^i_kv_i-\frac{\sum_{j=1}^n(d^j_{k-1})^2}{\sum_{j=1}^n\lambda_j(d^j_{k-1})^2}\cdot \sum_{i=1}^nd^i_k\lambda_iv_i.
    \end{aligned}
\end{equation}
Comparing the coefficients before each $v_i$ in (\ref{eq:decomposition}), we obtain
\begin{equation}\label{eq:coefficient}
d^i_{k+1}=d^i_k\cdot\left(\frac{\sum_{j=1}^n(\lambda_j-\lambda_i)(d^j_{k-1})^2} {\sum_{j=1}^n\lambda_j(d^j_{k-1})^2}\right), ~~i=1, \cdots, n.
\end{equation}
Specifically, in the first iteration we perform a step of the gradient descent method, yielding
\begin{equation}\label{eq:first-iter}
d^i_1=d^i_0\cdot\left(\frac{\sum_{j=1}^n(\lambda_j-\lambda_i)(d^j_0)^2} {\sum_{j=1}^n\lambda_j(d^j_0)^2}\right), ~~i=1, \cdots, n.
\end{equation}
Relation \eqref{eq:coefficient} gives the dynamics of the coefficients of each eigenvector direction as the BB method proceeds. From now on we will focus on analyzing this dynamics.

\section{Dynamics of the Coefficients}\label{sec:motivation}

In this section, by working out a simple example, we make some observations to provide an intuition about how the dynamics of the coefficients proceeds during the BB iteration.

Consider a $4$-dimensional quadratic minimization problem $\min_{x\in\mathbb{R}^4}\frac{1}{2}x^\top Ax$, where $A$ has eigenvalues $(\lambda_1, \lambda_2, \lambda_3, \lambda_4) = (0.001, 0.01, 0.1, 1)$, and the initial point is chosen subject to a uniform distribution between $[0, 1]$. The following figure is an example of the trajectories of the absolute value of the coefficients $|d_k^j|$ for $j=1, 2, 3, 4$ in the BB iteration\footnote{Although the trajectories can be different for some specific cases, this figure is a typical characterization of the trajectories in regard to the random initial point selection.}.

\begin{figure}[H]\label{fig:coef}
	\centering
    \includegraphics[width=6cm]{BB_coef_4.png}
    \caption{The trajectories of the absolute value of the coefficients $|d_k^j|$ in a $4$-dimensional BB iteration.}
\end{figure}

The purple, orange, red and blue lines in Figure \ref{fig:coef} correspond to the trajectories of the absolute value of the coefficients $|d_k^j|$ for $j=1, 2, 3, 4$ (i.e., from the minimal eigenvalue to the maximal one) separately. We have the following observations from Figure \ref{fig:coef}:
\begin{enumerate}
	\item The trajectory corresponding to the minimal eigenvalue (the purple line) always descends, whereas the others fluctuate near it instead of always staying above or below.
	\item If one trajectory is significantly higher than the others at some iteration, it will drop dramatically within two iterations.
\end{enumerate}

These observations provide an intuition about how the coefficients change during the BB iteration. Except the one corresponding to the minimal eigenvalue, all coefficients have two modes: the shrinking mode (when the absolute value of the coefficient decreases) and the fluctuation mode (when the absolute value of the coefficient increases). The switch between the two modes causes the BB method to have non-monotone behaviors. However, if one coefficient has a significantly larger absolute value than the others at some iteration, it must switch from the fluctuation mode to the shrinking mode in the next. This implies that the coefficients
alternate between fluctuation and trending in a downward pattern, which motivates our proof idea.

\section{Main Results}\label{sec:theorem}
In this section, we formally define the ``shrinking mode'' and the ``fluctuation mode'' following the motivation given in the previous section. Then with this definition, we make an surprisingly simple analysis to prove that the BB method converges linearly with a rate comparable to the classic SD method
\footnote{Our first version of the proof takes
at least 6 pages, based on detailed analysis of multiple cases.
We later greatly simplified the proof to just 2.5 pages.}

We first provide a formal definition of the ``shrinking mode'' and the ``fluctuation mode'':

\begin{definition}[Shrinking mode and fluctuation mode]
    ~
    \begin{enumerate}
        \item For a given index $i$, the sequence $\{d_k^i\}$ is in the shrinking mode at iteration $k$ if
        \begin{equation}\label{eq:def-shrinking}
            \sum_{j=1}^n(\lambda_j-\lambda_i)(d_k^j)^2\geqslant0;
        \end{equation}
        \item For a given index $i$, the sequence $\{d_k^i\}$ is in the fluctuation mode at iteration $k$ if
        \begin{equation}\label{eq:def-fluctuation}
        \sum_{j=1}^n(\lambda_j-\lambda_i)(d_k^j)^2<0.
        \end{equation}
    \end{enumerate}
    \end{definition}

According to relation \eqref{eq:coefficient}, the mode of $d_{k-1}^i$ decides whether $d_{k+1}^i$ and $d_k^i$ share the same sign, which is seemingly not related to ``shrinking'' or ``fluctuation''. Nevertheless, the mode actually determines whether the absolute value of $d_{k+1}^i$ can be greater than that of $d_k^i$. In fact, if $d_{k-1}^i$ is in the shrinking mode, $\left|d_{k+1}^i\right|$ is always smaller than $|d_k^i|$. On the other hand, if $d_{k-1}^i$ is in the fluctuation mode, $\left|d_{k+1}^i\right|$ can increase. However, as long as $\left|d_{k-1}^i\right|$ is large enough, we still have $\left|d_{k+1}^i|<|d_k^i\right|$. This finding is illustrated by the following proposition:

\begin{proposition}\label{prop:shrink}
    Consider the BB method for solving the quadratic minimization problem (\ref{eq:problem}) and let $\left\{d_k^i\right\}$ be the coefficient sequences. Then, it holds that
    \begin{equation}
    \label{eq::iterbound_general}
        \left|d_{k+1}^i\right|\leq\left|d_k^i\right|\cdot\left(\max\left\{\frac{\lambda_i}{\lambda_1}-1, 1-\frac{\lambda_i}{\lambda_n}\right\}\right), \forall k\geq 1, 1\leq i\leq n.
    \end{equation}
    Moreover, if at some iteration, one of the following conditions holds:
    \begin{enumerate}
        \item  $d_{k-1}^i$ is in the shrinking mode;
        \item  $d_{k-1}^i$ is in the fluctuation mode, and $(d_{k-1}^i)^2\geq\sum_{j=1}^{i-1}(d_{k-1}^j)^2$;
    \end{enumerate}
    then $d_{k+1}^i$ and $d_k^i$ satisfy
    \begin{equation}
    \label{eq::iterbound_shrink}

        \left|d_{k+1}^i\right|\leq \left(1-\frac{1}{\kappa}\right)\left|d_k^i\right|,
    \end{equation}
    where $\kappa=\frac{\lambda_n}{\lambda_1}$ is the condition number of $A$.
    \end{proposition}

\begin{proof}
Note that if $d_k^i=0$, then $d_k^{j}=0$ holds for all $j\geq i$. Thus, without losing generality, assume that $d_k^i\neq0$.

We first prove \eqref{eq::iterbound_general}. According to \eqref{eq:coefficient},
{\footnotesize
\begin{equation*}
\begin{aligned}
    \left|\frac{d_k^i}{d_{k-1}^i}\right|&=\left|\frac{\sum_{j=1}^n(\lambda_j-\lambda_i)(d^j_{k-1})^2} {\sum_{j=1}^n\lambda_j(d^j_{k-1})^2}\right|\\
    &\leq\max\left\{\frac{\sum_{j=1}^{i-1}(\lambda_i-\lambda_j)(d_{k-1}^j)^2}{\sum_{j=1}^n\lambda_j(d_{k-1}^j)^2}, \frac{\sum_{j=i+1}^n(\lambda_j-\lambda_i)(d_{k-1}^j)^2}{\sum_{j=1}^n\lambda_j(d_{k-1}^j)^2}\right\}\\
    &=\max\left\{\sum_{j=1}^{i-1}\left(\frac{\lambda_i-1}{\lambda_j}\right)\cdot\frac{\lambda_j(d_{k-1}^j)^2}{\sum_{j=1}^n\lambda_j(d_{k-1}^j)^2}, \sum_{j=i+1}^n\left(1-\frac{\lambda_i}{\lambda_j}\right)\cdot\frac{\lambda_j(d_{k-1}^j)^2}{\sum_{j=1}^n\lambda_j(d_{k-1}^j)^2}\right\}\\
    &\leq\max\left\{\left(\frac{\lambda_i}{\lambda_1}-1\right)\cdot\sum_{j=1}^{i-1} \frac{\lambda_j(d_{k-1}^j)^2}{\sum_{j=1}^n\lambda_j(d_{k-1}^j)^2}, \left(1-\frac{\lambda_i}{\lambda_n}\right)\cdot\sum_{j=i+1}^n \frac{\lambda_j(d_{k-1}^j)^2}{\sum_{j=1}^n\lambda_j(d_{k-1}^j)^2}\right\}\\
    &\leq\max\left\{\frac{\lambda_i}{\lambda_1}-1, 1-\frac{\lambda_i}{\lambda_n}\right\}.
\end{aligned}
\end{equation*}
}

Next, to prove \eqref{eq::iterbound_shrink}, we discuss under the two conditions separately.

If $d_{k-1}^i$ satisfies the first condition, we have $\sum_{j=1}^{i-1}(\lambda_i-\lambda_j)(d_k^j)^2\leq\sum_{j=i+1}^n(\lambda_j-\lambda_i)(d_k^j)^2$ according to \eqref{eq:def-shrinking}. Therefore,
\begin{equation*}
    \begin{aligned}
   \left|\frac{d_{k+1}^i}{d_k^i}\right| &=
   \frac{\sum_{j=1}^n(\lambda_j-\lambda_i)(d^j_{k-1})^2} {\sum_{j=1}^n\lambda_j(d^j_{k-1})^2}
        \leq \frac{\sum_{j=i+1}^n(\lambda_j-\lambda_i)(d_{k-1}^j)^2}{\sum_{j=1}^n\lambda_j(d_{k-1}^j)^2}  \\
        &= \sum_{j=i+1}^n
        \left(1-\frac{\lambda_i}{\lambda_j}\right)
        \cdot\frac{\lambda_j(d_{k-1}^j)^2}{\sum_{j=1}^n\lambda_j(d_{k-1}^j)^2}

        \leq  \left(1-\frac{\lambda_i}{\lambda_n}\right)\cdot\sum_{j=i+1}^n \frac{\lambda_j(d_{k-1}^j)^2}{\sum_{j=1}^n\lambda_j(d_{k-1}^j)^2}\\
        &\leq 1-\frac{\lambda_i}{\lambda_n}
        \leq 1-\frac{1}{\kappa}.
    \end{aligned}
\end{equation*}

If $d_{k-1}^i$ satisfies the second condition, we have $\sum_{j=1}^{i-1}(\lambda_i-\lambda_j)(d_k^j)^2>\sum_{j=i+1}^n(\lambda_j-\lambda_i)(d_k^j)^2$ according to \eqref{eq:def-fluctuation}. Therefore,
\begin{equation*}
    \begin{aligned}
        \left|\frac{d_{k+1}^i}{d_k^i}\right|&=\frac{\sum_{j=1}^n(\lambda_i-\lambda_j)(d^j_{k-1})^2} {\sum_{j=1}^n\lambda_j(d^j_{k-1})^2}
        \leq \frac{\sum_{j=1}^{i-1}(\lambda_i-\lambda_j)(d_{k-1}^j)^2}{\sum_{j=1}^n\lambda_j(d_{k-1}^j)^2}\\
        &\leq \frac{\sum_{j=1}^{i-1}(\lambda_i-\lambda_j)(d_{k-1}^j)^2}{\lambda_i(d_{k-1}^i)^2}
        \leq  \left(1-\frac{\lambda_1}{\lambda_i}\right)\cdot\sum_{j=1}^{i-1} \frac{(d_{k-1}^j)^2}{(d_{k-1}^i)^2}\\
        &\leq 1-\frac{\lambda_1}{\lambda_i}
        \leq 1-\frac{1}{\kappa}.
    \end{aligned}
\end{equation*}

Combining the two cases completes the proof.
\end{proof}

With Proposition \ref{prop:shrink}, we present our main theorem of linear convergence with a rather simple proof:
\begin{theorem}\label{thm:convergence}
	Consider the BB method for solving the quadratic minimization problem (\ref{eq:problem}). Let $\{d_k^i\}$ be the coefficient sequences and $\theta=1-\frac{1}{\kappa}$, where $\kappa=\frac{\lambda_n}{\lambda_1}$ is the condition number of $A$. Then $\{d_k^i\}$ converges at least linearly with a rate of $\theta$. In particular, for any $k\geq1$ and $i=1, \cdots, n$, $|d_k^i|\leq F_i\theta^k$, where $F_i$ is a constant defined by the following recursive sequence:
	\begin{equation}\label{eq:constant}
	\left\{\begin{aligned}
	F_1&=d_0^1\\
	F_i&=\max\left\{d_0^i, \frac{d_1^i}{\theta}, \theta^{-2} C^2 \sqrt{\sum_{j=1}^{i-1}F_j^2}\right\}
	\end{aligned}\right.
	\end{equation}
	where $C=\max\left\{\frac{\lambda_i}{\lambda_1}-1, 1-\frac{\lambda_i}{\lambda_n}\right\}.$
\end{theorem}

\begin{proof}
	We prove the result by induction on $i=1, \cdots, n$. For $i=1$, note that \eqref{eq:def-shrinking} always holds, so $d_k^1$ is always in the shrinking mode. According to Proposition \ref{prop:shrink}, we have $\left|\frac{d_{k+1}^i}{d_k^i}\right| \leq 1-\frac{\lambda_1}{\lambda_n}$ for any $k$. This implies that $\left|d_k^1\right|\leq \left|d_0^1\right|\theta^k$.

	Suppose that the result holds for all $d_k^j$ when $j=1, \cdots, i-1$. We prove by contrapositive that it also holds for all $d_k^i$. The result trivially holds for $k=0, 1$. Assume, in contrast, that the result does not hold for some $k\geq 2$. We find the minimal $k\geq 2$ such that $|d_k^i|>F_i\theta^k$. Note that by Proposition \ref{prop:shrink}, $\left|\frac{d_k^i}{d_{k-1}^i}\right|\leq C$ for any $k\geq 1$. Therefore,
	\begin{equation*}
		\left|d_{k-2}^i\right|\geq\frac{\left|d_k^i\right|}{C^2} >\frac{F_i\theta^k}{C^2} \geqslant \sqrt{\sum_{j=1}^{i-1}F_j^2}\theta^{k-2} \geq\sqrt{\sum_{j=1}^{i-1}(d_{k-2}^j)^2}.
	\end{equation*}
	Using the result of Proposition \ref{prop:shrink}, we immediately obtain that $|d_{k-1}^i|\geq |d_k^i|/\theta>F_i\theta^{k-1}$, which contradicts our assumption that $k$ is the minimal index such that the result fails to hold. Hence we have shown that the result holds for all $d_k^i$, and by induction the proof is complete.
\end{proof}

\textbf{Remark 1}: The linear rate we achieve in Theorem \ref{thm:convergence} is $1-\frac{1}{\kappa}$, which is much faster than the previous results \cite{dai2002r}. Furthermore, this rate is comparable to the convergence rate of the SD method ($1-\frac{2}{\kappa+1}$), so our result indicates that the BB method is at least comparable to the SD method in terms of the convergence
rate.
It is worth mentioning that the constant term $F_i$ in the theorem is highly related to the eigenvalues of $A$, and a trivial upper bound for $F_i$ is $\mathcal{O}((1+\kappa^2)^{i-1})$.

\textbf{Remark 2}: We briefly discuss the differences
of our proof with earlier proofs of convergence in \cite{raydan1993barzilai,dai2002r}.

\section{Example with Exact Linear Rate Convergence}\label{sec:lower-bound}

One may wonder whether the BB method achieves an even faster convergence rate since in practice it performs much better than the SD method. The answer we provide to this question is: not always. In particular, we present an example with a specifically chosen initialization to show that the BB method may converge at the same speed as the SD method.

\begin{proposition}\label{prop:lower-bound}
	Consider the BB method for solving the quadratic minimization problem (\ref{eq:problem}). For any $A\in \mathcal{S}^n_{++}$, there exists $x_0\in \mathbb{R}^n$ such that the sequence $\{x_k\}$ generated by the BB method satisfies
	\begin{equation}\label{eq:lower-bound}
		|g_k|\geq\left(\frac{\kappa-1}{\kappa+1}\right)^k|g_0|,
    \end{equation}
    where $g_k=Ax_k-c$ is the gradient at each iteration.
\end{proposition}
\begin{proof}
    Assume that $A$ has eigenvalues $0<\lambda_1\leq\cdots\leq\lambda_n$ with the corresponding eigenvectors $v_1, \cdots, v_n$. Let $x_0=A^{-1}(c+v_1+v_n)$, and thus $g_0=Ax_0-c=v_1+v_n$. Decomposing $g_0$ according to the basis of $v_1, \cdots, v_n$ yields
    \begin{equation*}
        d_0^1=d_0^n=1,~~ d_0^j=0,~~ j=2, \cdots, n-1.
    \end{equation*}

    By relation \eqref{eq:coefficient}, it is apparent to see that $d_k^j=0$ for all $j=2, \cdots, n-1$. Furthermore, for all $k\geq 1$,
    \begin{equation}\label{eq:example}
        \begin{aligned}
            d_{k+1}^1&=d_k^1\cdot \left(\frac{\sum_{j=1}^n(\lambda_j-\lambda_1)(d^j_{k-1})^2} {\sum_{j=1}^n\lambda_j(d^j_{k-1})^2}\right)=d_k^1\cdot\frac{(\lambda_n-\lambda_1)(d^n_{k-1})^2}{\lambda_1(d^1_{k-1})^2+\lambda_n(d^n_{k-1})^2},\\
            d_{k+1}^n&=d_k^1\cdot \left(\frac{\sum_{j=1}^n(\lambda_j-\lambda_n)(d^j_{k-1})^2} {\sum_{j=1}^n\lambda_j(d^j_{k-1})^2}\right)=d_k^n\cdot\frac{(\lambda_1-\lambda_n)(d^1_{k-1})^2}{\lambda_1(d^1_{k-1})^2+\lambda_n(d^n_{k-1})^2}.\\
        \end{aligned}
	\end{equation}
    Note that $d_0^1=d_0^n=1$ and $d_1^1=\frac{\lambda_n-\lambda_1}{\lambda_n+\lambda_1}, d_1^n=-\frac{\lambda_n-\lambda_1}{\lambda_n+\lambda_1}$ due to \eqref{eq:first-iter}, so $(d_0^1)^2=(d_0^n)^2, (d_1^1)^2=(d_1^n)^2$. According to relation \eqref{eq:example}, we can show by induction that $(d_k^1)^2=(d_k^n)^2$ for every $k\geq0$. As a result, \eqref{eq:example} is simplified as
    \begin{equation*}
            d_{k+1}^1=d_k^1\cdot \frac{\lambda_n-\lambda_1}{\lambda_n+\lambda_1},~~
            d_{k+1}^n=d_k^n\cdot \frac{\lambda_1-\lambda_n}{\lambda_n+\lambda_1},
    \end{equation*}
    which implies that
    \begin{equation*}
        d_1^k=\left(\frac{\lambda_n-\lambda_1}{\lambda_n+\lambda_1}\right)^k=\left(\frac{\kappa-1}{\kappa+1}\right)^k,~~ d_n^k=\left(\frac{\lambda_1-\lambda_n}{\lambda_n+\lambda_1}\right)^k=(-1)^k\left(\frac{\kappa-1}{\kappa+1}\right)^k.
    \end{equation*}
    Therefore inequality \eqref{eq:lower-bound} holds for the example we construct.
\end{proof}

A $2$-dimensional version of the example is mentioned in \cite{dai2013new}. Specifically, \cite{dai2013new} proved that the BB method has a superlinear convergence rate for probability one measure of initial points while converges linearly for zero measure of initial points. Proposition \ref{prop:lower-bound} demonstrates that for any-dimensional quadratic problems, the BB method converges no faster than the SD method when selecting some specific initial points. It also implies that there is no superlinear rate convergence for the BB method if we allow arbitrary initialization, which indicates the tightness of Theorem \ref{thm:convergence}. Nevertheless, we believe that the set of such initial points is zero measure. As for generic initialization, just as the case when $n=2$, the BB method can still be much faster for generic initialization;
we leave the investigation of this matter to future work.

\section{Conclusion}\label{sec:conclusion}

In this paper, we consider the BB method for solving  quadratic problems.
The existing rate of convergence is much worse
than the steepest descent method (or gradient descent with constant stepsize).
We prove that the convergence rate of BB methods
is at least $1-\frac{1}{\kappa}$, which is stronger than the
existing results and is comparable to the SD method.
Moreover, an example is constructed to show that for some initial points, the BB method converges at the same rate as the SD method.
An interesting research direction is to study whether a faster convergence rate exists when additional assumptions are imposed on the initial points.

Optimization and Control (math.OC); Machine Learning (cs.LG)


Stationary phase lemmas for general functions

Abstract: We give some details about the stationary phase lemma. We first prove a special case where the high order terms are derived explicitly. Based on that, we prove a more general case by using Morse lemma.

\section{Lemma Statements}

\begin{lem} \label{lem:2-staphlema}
	Let $n \in \mathbb{N}^+$ be the dimension.
	We consider the integral $I(\lambda)$:
	\[
	I(\lambda) = \int_{{\mathR^n}} e^{i\lambda \agl[Q(x - x_0), x - x_0]/2} a(x;\lambda) \dif{x},
	\]
	where $\agl[Q x, y]$ signifies $(Qx)^T \cdot y$ as matrix multiplication.
	For arbitrary integers $M$, $N \in \mathbb N$, assume
	\begin{itemize}

	\item for each $\lambda$, $a(\cdot;\lambda) \in C^{n+2N+3}({\mathR^n};\mathbb C)$;

	\item for each $\lambda$, $a(\cdot;\lambda) \in C^{M}({\mathR^n};\mathbb C)$, and $\forall \alpha : |\alpha| \leq M$, there exists $\lambda$-dependent constants $C_{M,\alpha}(\lambda) > 0$ such that $\forall x \in {\mathR^n}$ there holds $|\partial_x^\alpha a(x - x_0; \lambda)| < C_{M,\alpha}(\lambda) \agl[x]^{2M-n-1-|\alpha|}$;

	\item $Q$ is a non-degenerate, symmetric, real-valued matrix.

	\end{itemize}
	Then the integral $I(\lambda)$ is well-defined in the oscillatory integral sense, and as $\lambda \to +\infty$ we have
	\begin{subequations} \label{eq:IQ-staphlema}
	\begin{empheq}[box=\widefbox]{align}
	I(\lambda)
	& = \left( \frac{2\pi}{\lambda} \right)^{n/2} \frac{e^{i\frac{\pi}{4}\mathop{\rm sgn} Q}} {|\det Q|^{1/2}} \sum_{0 \leq j \leq N} \frac {\lambda^{-j}} {j!} \left( \frac {\agl[Q^{-1}D, D]} {2i} \right)^j a(x_0; \lambda) \nonumber \\
	& \qquad + \mathcal O \big( \lambda^{-\frac{n}{2}-N-1} \times \sum_{|\alpha| \leq n+2N+3} \sup_{B(x_0,1)} |\partial^\alpha a(\cdot; \lambda)| \big) \tag{\ref{eq:IQ-staphlema}} \\
	& \qquad + \mathcal O \big( \lambda^{-M} \times \sum_{|\alpha| \leq M} \sup_{{\mathR^n}} \frac {|\partial^\alpha a(x - x_0; \lambda)|} {\agl[x]^{2M-n-1-|\alpha|}} \big), \nonumber
	\end{empheq}
	\end{subequations}
	where $B(x_0,1)$ stands for the ball centered at $x_0$ with radius 1, and $\mathop{\rm sgn} Q$ stands for the difference between the number of positive eigenvalues and the number of negative eigenvalues of the matrix $Q$.
	Especially, when $M$ is chosen to be within the interval $[\max\{ \lceil n/2 \rceil + N + 1, n+1\} , n + 2N + 3]$, \eqref{eq:IQ-staphlema} can be simplified to
	\begin{equation} \label{eq:IQs-staphlema}
	\boxed{I(\lambda)
	= \left( \frac{2\pi}{\lambda} \right)^{n/2} \frac{e^{i\frac{\pi}{4}\mathop{\rm sgn} Q}} {|\det Q|^{1/2}} \sum_{0 \leq j \leq N} \frac {\lambda^{-j}} {j!} \left( \frac {\agl[Q^{-1}D, D]} {2i} \right)^j a(x_0; \lambda) + \mathcal O \big( \lambda^{-\frac{n}{2}-N-1} \sum_{|\alpha| \leq n+2N+3} \sup_{{\mathR^n}} |\partial^\alpha a(\cdot; \lambda)| \big).}
	\end{equation}
\end{lem}

\begin{rem} \label{rem:1-staphlema}
	The first thing to note is that, in contrast to many other versions of the stationary phase lemma, here we don't require $a$ to be compactly supported.
	Instead, some boundedness conditions are required, which generates the compact-support condition.
	Second, the integers $M$ and $N$ in Lemma \ref{lem:2-staphlema} shall be chosen properly to serve for your own purposes.
	For example, if one cares more about the decaying behavior w.r.t.~$\lambda$, then the $M$ can be set to $\lceil n/2 \rceil +N+1$.
	However, if one is dealing with these functions $a$ which doesn't have good decaying behavior at the infinity, then one could set $M$ to be large enough such that $\agl[x]^{2M-n-1-|\alpha|}$ can dominate $\partial^\alpha a$, with the cost that we should demand more smoothness of $a(x)$.
	Third, the unit ball $B(x_0,1)$ involved in the term $\sup_{B(0,1)} |\partial^\alpha a|$ can be changed to other bounded domain containing the origin.
	But one should be careful that when the domain you want to use has very small radius, the underlying coefficients of the $\mathcal O(\cdot)$ notation will be very large accordingly.
	Fourth, the function $a$ can depend on $\lambda$, hence the expression \eqref{eq:IQ-staphlema} is an asymptotics only when $\partial_x^\alpha a(x;\lambda)$ don't change significantly when $\lambda \to +\infty$.
\end{rem}

If chosen $M = n + 2N + 3$, Lemma \ref{lem:2-staphlema} can be simplified as follows.

\begin{prop} \label{prop:2-staphlema}
	Assume $I(\lambda)$ and $Q$ are as stated in Lemma \ref{lem:2-staphlema}. If $a \in C^{n+2N+3}({\mathR^n};\mathbb C)$, and for $\forall \alpha : |\alpha| \leq n+2N+3$ there holds $\sup_{\mathR^n} |\partial^\alpha a(x)| < +\infty$, then \eqref{eq:IQs-staphlema} holds.
\end{prop}
Proposition \ref{prop:2-staphlema} can be extend to a more general case where the phase function is not quadratic.

\begin{lem}[Stationary phase lemma \cite{eskin2011lectures}] \label{lem:1-staphlema}
    We consider the integral $I(\lambda)$:
    \[
    I(\lambda) = \int_{{\mathR^n}} a(x) e^{i\lambda \varphi(x)} \dif{x}.
    \]
    For an arbitrary integer $N \in \mathbb N$, assume
    \begin{itemize}
   	\item $a \in C^{n+2N+3}({\mathR^n};\mathbb C)$ with $\sum_{|\alpha| \leq n + 2N+3} \sup_{{\mathR^n}} |\partial^\alpha a| < +\infty$;

   	\item $\varphi \in C^{n+2N+6}({\mathR^n};\mathbb{R}} \newcommand{\mathR}{\mathbb{R})$ with $\sum_{|\alpha| \leq n+2N+6} \sup_{{\mathR^n}} |\partial^\alpha \varphi| < +\infty$;

   	\item $x_0$ is the only critical point of $\varphi(x)$ on $\mathop{\rm supp} a(x)$, i.e., $\varphi(x_0) = \nabla \varphi(x_0) = 0$, $\varphi_x(x) \neq 0$ for $x \neq x_0$;

   	\item the Hessian $\varphi_{xx}(x_0) := [\frac{\partial^2\varphi}{\partial x_j \partial x_k}(x_0)]_{j,k=1}^n$ satisfies $\det \varphi_{xx}(x_0) \neq 0$.
    \end{itemize}
Then the integral $I(\lambda)$ is well-defined in the oscillatory integral sense, and as $\lambda \to +\infty$ we have
    \begin{subequations} \label{eq:I-staphlema}
    \begin{empheq}[box=\widefbox]{align}
    I(\lambda)
    & = \left( \frac{2\pi}{\lambda} \right)^{n/2} \frac{e^{i\lambda\varphi(x_0) + i\frac{\pi}{4}\mathop{\rm sgn} \varphi_{xx}(x_0)}}{|\det \varphi_{xx}(x_0)|^{1/2}} \Big( a(x_0) + \sum_{j=1}^{N} a_j(x_0)\lambda^{-j} \Big) \nonumber \\
    & \quad + \mathcal O \big( \lambda^{-\frac{n}{2}-N-1} \times \sum_{|\alpha| \leq n+2N+3} \sup_{{\mathR^n}} |\partial^\alpha a| \times \sum_{|\alpha| \leq n+2N+6} \sup_{{\mathR^n}} |\partial^\alpha \varphi| \big), \tag{\ref{eq:I-staphlema}}
    \end{empheq}
    \end{subequations}
    for some functions $a_j, ~1 \leq j \leq N$.

\end{lem}

\begin{rem}
	Proposition \ref{prop:2-staphlema} is a special case of Lemma \ref{lem:1-staphlema} where $\varphi(x) = \agl[Q(x - x_0), x - x_0]/2$, which guarantees $\varphi_{xx}(x_0) = Q$.
	Lemma \ref{lem:1-staphlema} is not a generalization of Lemma \ref{lem:2-staphlema} because unlike the quadratic phase function in Lemma \ref{lem:2-staphlema}, the phase function $\varphi$ in Lemma \ref{lem:1-staphlema} is not assumed to possess the property that $|\nabla \varphi(x)| \simeq |x|$ as $|x|$ is large.
	However if $\varphi$ is a homeomorphism of ${\mathR^n}$, it is possible to generalize Lemma \ref{lem:2-staphlema}.
\end{rem}

We can compute these $a_j(x_0)$ in a more precise way (cf.~\cite[(3.4.11)]{zw2012semi}), and the details are given in \cite[Second proof of Theorem 3.11]{zw2012semi}.
In \cite[Lemma 19.3]{eskin2011lectures} there is also another routine to prove the stationary phase lemma.
\cite[Chapter 5]{dim1999spe} by Dimassi and Sj\"ostrand is also a good reference.

\section{Preliminaries}

We need the Taylor's expansion.
Suppose $f \in C^{N+1}({\mathR^n}; \mathbb C)$, then we have that
\begin{equation} \label{eq:TaylorInt-staphlema}
f(x) = \sum_{|\delta| \leq N} \frac{1}{\delta!} \big( \partial^{\delta}f \big)(x_0) \cdot (x - x_0)^{\delta} + (N+1) \sum_{|\delta| = N+1} \frac{(x - x_0)^{\delta}}{\delta!} \int_0^1 (1-t)^N \big( \partial^{\delta}f \big)(x_0 + t(x-x_0)) \dif{t}.
\end{equation}
We omit the proof.

\smallskip

For a measurable function $u$, there exists a constant $C$ such that $\hat u$ exists and
\begin{equation} \label{eq:Fuu-staphlema}
	\nrm[L^1({\mathR^n})]{\hat u} \leq C \sum_{|\alpha| \leq n + 1} \nrm[L^1({\mathR^n})]{\partial^\alpha u},
\end{equation}
as long as the right-hand-side of \eqref{eq:Fuu-staphlema} is defined.

\begin{proof}
	We have
	\begin{align*}
		\int |\hat u(\xi)| \dif \xi
		& \leq \int \agl[\xi]^{-n-1} |\agl[\xi]^{n+1} \hat u(\xi)| \dif \xi
		\leq C \sup_{{\mathR^n}} |\agl[\xi]^{n+1} \hat u(\xi)| \\
		& \leq C \sum_{|\alpha| \leq n + 1} C_\alpha \sup_{{\mathR^n}} |\xi^\alpha \hat u(\xi)|
		\leq \sum_{|\alpha| \leq n + 1} C_\alpha \sup_{{\mathR^n}} |(\partial^\alpha u)^\wedge(\xi)| \\
		& \leq \sum_{|\alpha| \leq n + 1} C_\alpha \nrm[L^1({\mathR^n})]{\partial^\alpha u}.
	\end{align*}
	We arrive at the conclusion.
\end{proof}

\smallskip

We also need the following transformation.
For a fixed non-degenerate, symmetric, real-valued square matrix $Q$, we have
\begin{equation} \label{eq:FrGauiQ-staphlema}
\mathcal{F} \{ e^{\pm i\,\agl[Q \cdot, \cdot]/2} \}(\xi) = \frac {e^{\pm i \frac \pi 4 {\rm sgn}\, Q}} {|\det Q|^{1/2}} e^{\mp i \agl[Q^{-1} \xi,\xi]/2}.
\end{equation}

\section{Proof of the lemma} \label{sec:pof-staphlema}

We first prove the quadratic case.

\begin{proof}[Proof of Lemma \ref{lem:2-staphlema}]

	We omit notationally the dependence of $a$ on $\lambda$ until related clarifications are needed.
	Without loss of generality we assume $x_0 = 0$, and $a(0) = 1$.
	For readers' convenient we rewrite the expression of $I$ here: $I(\lambda) = \int_{{\mathR^n}} e^{i\lambda \agl[Qx, x]/2} a(x) \dif{x}$.

	According to the assumption on $Q$, we know there exists a decomposition $Q = P \Lambda P^T$ where $P$ is an orthogonal matrix and $\Lambda := (\alpha_j)_{j=1,\cdots,n}$ is a diagonal matrix.
	Make the change of variable $y = P^T x$, we can have
	\begin{align}
	I
	& = \int_{\mathR^n} a(Py) e^{i\lambda \sum_{j=1}^{n} \alpha_j y_j^2/2} \dif y \nonumber \\
	& = \int_{\mathR^n} e^{i\lambda \sum_{j=1}^{n} \alpha_j y_j^2/2} (1-\chi(y)) f(y) \dif y + \int_{\mathR^n} e^{i\lambda \sum_{j=1}^{n} \alpha_j y_j^2/2} \chi(y) f(y) \dif y \qquad (\det P = 1) \nonumber \\
	& =: J_1 + J_2, \label{eq:IQJ12-staphlema}
	\end{align}
	where $f(y) := a(Py)$ and $\chi \in C_c^\infty({\mathR^n})$ is a cutoff function satisfying $\chi \equiv 1$ in a neighborhood of the origin.

	Noticing that a neighborhood of the origin is not included in the support of the integrand in $J_1$, we can estimate $J_1$ by using integration by parts (in the oscillatory integral sense).
	For any integer $M \in \mathbb{N}$ we have
	\begin{align}
	|J_1|
	& = |\int_{\mathR^n} \big( \frac {\sum_j \alpha_j^{-1} y_j \partial_j} {i\lambda |y|^2} \big)^M (e^{i\lambda \sum_{j=1}^{n}\alpha_j y_j^2/2} ) \big[ (1-\chi(y)) f(y) \big] \dif y| \nonumber \\
	& \lesssim \lambda^{-M} \int_{\mathR^n} |\big( \sum_j \partial_j \circ (y_j |y|^{-2}) \big)^M \big[ (1-\chi(y)) f(y) \big] | \dif y \nonumber \\
	& \lesssim \lambda^{-M} \int_{\mathop{\rm supp} (1-\chi)} \sum_{|\alpha| \leq M} C_{M; \alpha} |y|^{|\alpha|-2M} |\partial^\alpha ((1-\chi) f(y))| \dif y \label{eq:J1Ind-staphlema} \\
	& \lesssim \lambda^{-M} \big( C \sum_{|\alpha| \leq M} \sup_{\{0 < \chi < 1\}} |\partial^\alpha f| + \sum_{|\alpha| \leq M} C_{M; \alpha} \int_{\{\chi = 0\}} |y|^{|\alpha|-2M} |\partial^\alpha f(y)| \dif y \big) \label{eq:J1Ine-staphlema} \\
	& \lesssim \lambda^{-M} \sum_{|\alpha| \leq M} \big( \sup_{\{0 < \chi < 1\}} |\partial^\alpha a| + \int_{\{\chi = 0\}} |y|^{-n-1} \agl[y]^{|\alpha|-2M+n+1} |\partial^\alpha a(y)| \dif y \big) \nonumber \\
	& \lesssim \lambda^{-M} \sum_{|\alpha| \leq M} \big( \sup_{\{0 < \chi < 1\}} |\partial^\alpha a| + \sup_{{\mathR^n}} \frac {|\partial^\alpha a(y)|} {\agl[y]^{2M-n-1-|\alpha|} } \big) \nonumber \\
	& \lesssim \lambda^{-M} \sum_{|\alpha| \leq M} \sup_{{\mathR^n}} \frac {|\partial^\alpha a(y)|} {\agl[y]^{2M-n-1-|\alpha|}}. \label{eq:J1f-staphlema}
	\end{align}
	The inequality \eqref{eq:J1Ind-staphlema} is due to the fact that
	\[
	\big( \sum_j \partial_j \circ (y_j |y|^{-2}) \big)^M \varphi = \sum_{|\alpha| \leq M} C_{M; \alpha} |y|^{|\alpha|-2M} \partial^\alpha \varphi
	\]
	which can be derived by induction and we omit the details.
	Inequality \eqref{eq:J1Ine-staphlema} is due to the fact that $\partial^\alpha ((1-\chi) f) = \partial^\alpha f$ in $\{\chi = 0\}$.

	Then we turn to $J_2$.
	Keep in mind that $f(y) = a(Py)$ and $\chi f$ is compactly support and in $C_c^{n+2N+3}({\mathR^n})$.
	Here we analyze $I$ by borrowing idea from \cite[First proof of Theorem 3.11]{zw2012semi}.
	First we use Plancherel theorem (which states $(f,g) = (\hat f, \hat g)$),
	\begin{align}
	J_2
	& = \int_{\mathR^n} \overline{ e^{-i\lambda \sum_{j=1}^{n} \alpha_j y_j^2/2} } \chi f(y) \dif y
	= \int_{\mathR^n} \overline{ \mathcal{F} \{ e^{-i\lambda \sum_{j=1}^{n} \alpha_j (\cdot)^2/2} \}(\xi) } \cdot \widehat{\chi f}(\xi) \dif \xi \nonumber \\
	& = \int_{\mathR^n} \overline{ (\lambda)^{-n/2} \frac {e^{-\frac {i\pi} 4 {\rm sgn}\, Q}} {|\det Q|^{1/2}} e^{\frac i {2\lambda} \alpha_j^{-1} \xi_j^2} } \cdot\widehat{\chi f}(\xi) \dif \xi \qquad (\text{by~} \eqref{eq:FrGauiQ-staphlema})\nonumber \\
	& =: \big( \frac {2\pi} \lambda \big)^{n/2} \frac {e^{\frac {i\pi} 4 {\rm sgn}\, Q}} {|\det Q|^{1/2}} J(1/\lambda, 1/\lambda, \chi f), \label{eq:I1J-staphlema}
	\end{align}
	where we ignored the summation notation over $j$ and the function $J$ is defined by
	\begin{equation} \label{eq:Jh-staphlema}
	J(h_1, h_2, \chi f) := (2\pi)^{-n/2} \int_{\mathR^n} e^{\frac {\xi_j^2 h_1} {i2\alpha_j}} \cdot \widehat{\chi f}(\xi; 1/h_2) \dif \xi.
	\end{equation}
	Note that in \eqref{eq:Jh-staphlema} we put emphasize on the dependence of $f$ on $h_2$ (i.e.~dependence of $a$ on $\lambda$).
	The smoothness of $J$ w.r.t.~$h_1$ is guaranteed by the $L^1$ of derivatives of $f$, namely, we have the following claim whose justification will be clear in \eqref{eq:JhR-staphlema},
	\[
	\forall m \in \mathbb N,\, \max_{|\alpha| \leq n + 2m + 1} \nrm[L^1({\mathR^n})]{\partial^\alpha f} < +\infty \ \Rightarrow \ J(\cdot, h_2, f) \in C^m(\mathbb{R}} \newcommand{\mathR}{\mathbb{R}).
	\]

	We abbreviate $\partial_{h_1} J$ as $\partial_1 J$.
	Note that
	\begin{align*}
	\partial_1^k J(0, h_2, \chi f)
	& = (2\pi)^{-n/2} \int_{\mathR^n} \partial_1^{k} (e^{\frac {\xi_j^2 h_1} {i2\alpha_j}}) |_{h_1 = 0} \cdot \widehat{\chi f}(\xi) \dif \xi
	= (2\pi)^{-n/2} \int_{\mathR^n} (\sum_j \frac {\xi_j^2} {i2\alpha_j})^k \cdot \widehat{\chi f}(\xi) \dif \xi \\
	& = (2\pi)^{-n/2} \int_{\mathR^n} \widehat{T^k \chi f}(\xi) \dif \xi
	= T^k f(0)
	= T^k \big( a(Py) \big) |_{y = 0} \\
	& = (\frac i 2 P^{lj} P^{kj} \alpha_j^{-1} \partial_{kl})^k a(0)
	= A^k a(0; 1/h_2),
	\end{align*}
	where $T = \frac i 2 \sum_j \frac {\partial_j^2} {\alpha_j}$ and $A = \frac i 2 (Q^{-1})^{jl} \partial_{jl} = \frac 1 {2i} \agl[Q^{-1}D, D]$ (recall that $D = \frac 1 i \nabla$ is vertical).
	We expand $J$ using Taylor series (i.e.~\eqref{eq:TaylorInt-staphlema}),
	\begin{align}
	J(h, h_2, \chi f)
	& = \sum_{k\leq N} \frac{h^k}{k!} \partial_h^{k}J(0, h_2, \chi f) + \frac{h^{N+1}} {N!} \int_0^1 (1-t)^{N} \cdot \partial_1^{N+1}J(th, h_2, \chi f) \dif{t} \nonumber \\
	& = \sum_{0 \leq k \leq N} \frac{(hA)^k}{k!} a(0; h_2) + \frac{h^{N+1}} {N!} \int_0^1 (1-t)^{N} \cdot \partial_1^{N+1}J(th, h_2, \chi f) \dif{t}. \label{eq:JhExp-staphlema}
	\end{align}
	By invoking \eqref{eq:Jh-staphlema}, the remainder term in \eqref{eq:JhExp-staphlema} can be estimated as
	\begin{align*}
	& \ |\frac{h^{N+1}}{N!} \int_0^1 (1-t)^{N} \cdot \partial_1^{N+1}J(th, h_2, \chi f) \dif{t}| \\
	\leq & \ C_N h^{N+1} \int_{\mathR^n} |(\frac {-i} {4\alpha_j} \xi_j^2)^{N+1} \cdot \widehat{\chi f}(\xi; 1/h_2)| \dif \xi
	\leq C_N h^{N+1} \sum_{|\beta| \leq 2N+2} C_\beta \nrm[L^1({\mathR^n})]{(\partial_x^\beta (\chi f(\cdot; 1/h_2)))^\wedge}.
	\end{align*}
	By using \eqref{eq:Jh-staphlema} and \eqref{eq:Fuu-staphlema}, we can continue
	\begin{align}
	& \ |\frac{h^{N+1}}{N!} \int_0^1 (1-t)^{N} \cdot \partial_1^{N+1}J(th, h_2, \chi f) \dif{t}| \nonumber \\
	\leq & \ C_N h^{N+1} \sum_{\substack{|\alpha| \leq n+1 \\ |\beta| \leq 2N + 2}} \nrm[L^1({\mathR^n})]{\partial_x^{\alpha + \beta} (\chi f(\cdot; 1/h_2))}
	%
	\leq C_N h^{N+1} \sum_{|\alpha| \leq n+ 2N + 3} \nrm[L^1({\mathR^n})]{\partial_x^{\alpha} (\chi f(\cdot; 1/h_2))} \nonumber \\
	\leq & \ C_N h^{N+1} \sum_{|\alpha| \leq n+2N+3} \sup_{\mathop{\rm supp} \chi} |\partial^\alpha a(\cdot; 1/h_2)|. \label{eq:JhR-staphlema}
	\end{align}
	Letting $h = h_2 = 1/\lambda$ and combining \eqref{eq:I1J-staphlema}, \eqref{eq:Jh-staphlema}, \eqref{eq:JhExp-staphlema} and \eqref{eq:JhR-staphlema}, we obtain
	\begin{align}
	J_2
	& =  \left( \frac {2\pi} \lambda \right)^{n/2} \frac {e^{\frac {i\pi} 4 {\rm sgn}\, Q}} {|\det Q|^{1/2}} \sum_{j \leq N} \frac {\lambda^{-j}} {j!} \left( \frac {\agl[Q^{-1}D, D]} {2i} \right)^j a(0; \lambda) + C_N \lambda^{-\frac n 2-N-1} \sum_{|\alpha| \leq n+2N+3} \sup_{\mathop{\rm supp} \chi} |\partial^\alpha a(\cdot; \lambda)|. \label{eq:J2t-staphlema}
	\end{align}

	Combining \eqref{eq:J2t-staphlema} with \eqref{eq:IQJ12-staphlema}, \eqref{eq:J1f-staphlema}, we have
	\begin{align*}
	I
	& = \left( \frac {2\pi} \lambda \right)^{n/2} \frac {e^{\frac {i\pi} 4 {\rm sgn}\, Q}} {|\det Q|^{1/2}} \sum_{j \leq N} \frac {\lambda^{-j}} {j!} \left( \frac {\agl[Q^{-1}D, D]} {2i} \right)^j a(0; \lambda) \\
	& \qquad + \mathcal O \big( \lambda^{-\frac n 2-N-1} \sum_{|\alpha| \leq n+2N+3} \sup_{\mathop{\rm supp} \chi} |\partial^\alpha a(\cdot; \lambda)| \big) + \mathcal O \big( \lambda^{-M} \sum_{|\alpha| \leq M} \sup_{{\mathR^n}} \frac {|\partial^\alpha a(y; \lambda)|} {\agl[y]^{2M-n-1-|\alpha|}} \big),
	\end{align*}
	which is \eqref{eq:IQ-staphlema}.

	By setting $M$ to be $\max\{ \lceil n/2 \rceil + N + 1, n+1\} \leq M \leq n + 2N + 3$, we can have $-M \leq -n/2 - N - 1$, $M \leq n + 2N + 3$ and $2M - n - 1 - |\alpha| \geq 0$ provided $|\alpha| \leq M$, so we can obtain \eqref{eq:IQs-staphlema} from \eqref{eq:IQ-staphlema}.

	The proof is complete.
\end{proof}

Now we prove the more general case.

\begin{proof}[Proof of Lemma \ref{lem:1-staphlema}]
	Without loss of generality we assume $x_0 = 0$, $\varphi(0) = 0$ and $a(0) = 1$.
	Hence by Taylor's expansion \eqref{eq:TaylorInt-staphlema} we have
	\[
	\varphi(x)
	= \sum_{j,k \leq n} x_j x_k \int_0^1 (1-t)\, \partial_{jk} \varphi(tx) \dif t
	= x^T \cdot\int_0^1 (1-t) \varphi_{xx}(tx) \dif t \cdot x.
	\]
	Note that $|\varphi_{xx}(0)| \neq 0$ and $|\varphi_{xx}(x)|$ is continuous on $x$ (\underline{$\varphi \in C^2$}), thus there exists a positive constant \boxed{r} such that $|\varphi_{xx}(x)| > |\varphi_{xx}(0)|/2 > 0$ for all $x \in B(0,r)$.
	Fix a cutoff function $\chi \in C_c^\infty({\mathR^n})$ such that $\mathop{\rm supp} \chi \subset B(0,r)$ and $\chi \equiv 1$ in $B(0,r/2)$.
	Hence on $B(0,r)$ the matrix $\varphi_{xx}$ is non-degenerate, and on $\mathop{\rm supp} a \backslash B(0,r)$ the norm of gradient is uniformly bounded from below, i.e.~there exists a constant $C > 0$ such that $|\nabla \varphi(x)| \geq C$ for $\forall x \in \mathop{\rm supp} a \backslash B(0,r)$.
	We divide $I$ into two parts
	\begin{equation} \label{eq:ISplit-staphlema}
	I(\lambda)
	= \int_{{\mathR^n}} (1 - \chi(x)) a(x) e^{i\lambda \varphi(x)} \dif x + \int_{{\mathR^n}} \chi(x) a(x) e^{i\lambda \varphi(x)} \dif x
	:= I_1 + I_2,
	\end{equation}
	and we will show that $I_1$ is rapidly decreasing w.r.t.~$\lambda$ while $I_2$ can be analyzed by using Lemma \eqref{lem:2-staphlema}.

	For $I_1$, denote
	\(
	L = \sum_{j=1}^n \frac {\varphi_{x_j}} {|\nabla \varphi|^2} \partial x_j,
	\)
	where $\varphi_{x_j}$ is short for $\partial_{x_j} \varphi$.
	Then $\frac 1 {i\lambda} L e^{i\lambda \varphi} = e^{i\lambda\varphi} \text{ and } {}^t L f = \sum_{j=1}^n \partial_{x_j} \big( \frac {\varphi_{x_j} f} {|\nabla \varphi|^2} \big)$.
	For any $N \in \mathbb{N}^+$, $I_2$ can be easily estimated as follows (which requires
	$a \in C^{n+N+1}({\mathR^n})$ and $\varphi \in C^{n+N+2}({\mathR^n})$)
	\begin{align}
	I_1
	& = \int_{\mathR^n} (1-\chi) a \cdot ((i\lambda)^{-n-N-1} L^{N+1} e^{i\lambda \varphi(x)}) \dif{x}
	= (i\lambda)^{-n-N-1} \int_{\mathR^n} ({}^t L)^{n+N+1} ((1-\chi)a) \cdot e^{i\lambda \varphi(x)} \dif{x} \nonumber\\
	& = \mathcal{O} (\lambda^{-n-N-1} \sum_{|\alpha| \leq n+N+1} \nrm[L^1({\mathR^n})]{\partial^\alpha a}), \quad \lambda \to \infty \label{eq:I2-staphlema}.
	\end{align}
	As mentioned before, due to the presence of $1-\chi$, the denominator $|\nabla \varphi|^2$ in $L$ keeps a positive distance away from 0, guaranteeing that $({}^t L)^N ((1-\chi)a)$ is bounded and compactly supported.

	Now we turn to $I_2$.
	Because $\varphi \in C^2({\mathR^n})$, $\varphi_{xx}(x)$ is symmetric and thus there exist orthogonal matrix $P(x)$ and diagonal matrix $\Lambda(x) = (\alpha_j(x))_{j=1,\cdots,n}$ such that
	\begin{equation*}
	2 \int_0^1 (1-t) \varphi_{xx}(tx) \dif t = P(x) \Lambda(x) P^T(x).
	\end{equation*}
	Especially we have
	\(
	P(0) \Lambda(0) P^T(0) = \varphi_{xx}(0).
	\)
	Denote $\alpha_j = \alpha_j(0)$ and $n \times n$ matrix $\Lambda := (\alpha_j)_{j=1,\cdots,n}$ for short. Thus
	\begin{equation*}
	\Lambda(x) = (\sqrt{\frac {\alpha_j(x)} {\alpha_j}})_{j=1,\cdots,n} \cdot \Lambda \cdot (\sqrt{\frac {\alpha_j(x)} {\alpha_j}})_{j=1,\cdots,n}.
	\end{equation*}
	Make the change of variable:
	\begin{equation} \label{eq:xToy-staphlema}
	y = \Phi(x) := \left( \sqrt{\frac {\alpha_j(x)} {\alpha_j}} \right)_{j=1,\cdots,n} \cdot P^T(x) \cdot x.
	\end{equation}
	Note that
	\begin{equation} \label{eq:sm1-staphlema}
	\varphi \in C^{n+2N+6} \Rightarrow \Phi \in C^{n+2N+4}.
	\end{equation}
	We have
	\begin{align*}
	\varphi(x)
	& = \frac 1 2 x^T \cdot \big[ 2 \int_0^1 (1-t) \varphi_{xx}(tx) \dif t \big] \cdot x
	= \frac 1 2 x^T \cdot \big[ P(x) \Lambda(x) P^T(x) \big] \cdot x \\
	& = \frac 1 2 [P^T(x) \cdot x]^T \cdot (\sqrt{\frac {\alpha_j(x)} {\alpha_j}})_{j=1,\cdots,n} \cdot \Lambda \cdot (\sqrt{\frac {\alpha_j(x)} {\alpha_j}})_{j=1,\cdots,n} \cdot [P^T(x) \cdot x] \\
	& = \frac 1 2 \big[ \big( \sqrt{\frac {\alpha_j(x)} {\alpha_j}} \big)_{j=1,\cdots,n} \cdot P^T(x) \cdot x \big]^T \cdot \Lambda \cdot \big[ \big( \sqrt{\frac {\alpha_j(x)} {\alpha_j}} \big)_{j=1,\cdots,n} \cdot P^T(x) \cdot x \big] \\
	& = \frac 1 2 \agl[\Lambda y, y].
	\end{align*}

	We have $\Phi(0) = 0$. It is easy to check that $\frac {\partial \Phi} {\partial x} (0) = P^T(0).$
	From \eqref{eq:xToy-staphlema} it is clear that there exists inverse of $\Phi$, i.e.~$\phi = \Phi^{-1}$
	Note that $x = \phi(\Phi(x))$ and
	\begin{equation} \label{eq:sm2-staphlema}
	\Phi \in C^{n+2N+4} \Rightarrow \phi \in C^{n+2N+4}.
	\end{equation}
	We have
	\begin{align*}
	I_1
	& = \int_{\mathR^n} \chi(\phi(y)) a(\phi(y)) \cdot e^{i\lambda \agl[\Lambda y, y]/2} \dif{\phi(y)} \nonumber\\
	& = \int_{\mathR^n} \chi(\phi(y)) a(\phi(y)) |\det \nabla_y \phi (y)| \cdot e^{i\lambda \agl[\Lambda y, y]/2} \dif{y} \nonumber\\
	& = \int_{\mathR^n} f(y) e^{i\lambda \agl[\Lambda y, y]/2} \dif{y}
	\end{align*}
	where
	\begin{equation*}
	f(y) = \chi(\phi(y)) \cdot a(\phi(y)) \cdot |\det \nabla_y \phi (y)|,
	\end{equation*}
	Note that
	\begin{equation} \label{eq:sm3-staphlema}
	\phi \in C^{n+2N+4},\, a \in C^{n+2N+2} \Rightarrow f \in C^{n+2N+2}.
	\end{equation}
	Now we can conclude from \eqref{eq:sm3-staphlema}, \eqref{eq:sm3-staphlema} and \eqref{eq:sm3-staphlema} that
	\begin{equation} \label{eq:sm4-staphlema}
	\varphi \in C^{n+2N+6},\, a \in C^{n+2N+3} \Rightarrow f \in C^{n+2N+3}.
	\end{equation}
	By using Lemma \ref{lem:2-staphlema}, we can obtain
	\begin{align}
	I_1(\lambda)
	& = \left( \frac{2\pi}{\lambda} \right)^{n/2} \frac{e^{i\frac{\pi}{4}\mathop{\rm sgn} \Lambda}} {|\det \Lambda|^{1/2}} \sum_{0 \leq j \leq N} \frac {\lambda^{-j}} {j!} \left( \frac {\agl[\Lambda^{-1}D, D]} {2i} \right)^j f(0) \nonumber \\
	& \qquad + \mathcal O(\lambda^{-\frac{n}{2}-N-1} \times \sum_{|\alpha| \leq n+2N+3} \sup_{{\mathR^n}} |\partial^\alpha f|) \nonumber \\
	& = \left( \frac{2\pi}{\lambda} \right)^{n/2} \frac{e^{i\frac{\pi}{4}\mathop{\rm sgn} \Lambda}} {|\det \Lambda|^{1/2}} \sum_{0 \leq j \leq N} \frac {\lambda^{-j}} {j!} \left( \frac {\agl[\Lambda^{-1}D, D]} {2i} \right)^j f(0) \nonumber \\
	& \qquad + \mathcal O(\lambda^{-\frac{n}{2}-N-1} \times \sum_{|\alpha| \leq n+2N+3} \sup_{{\mathR^n}} |\partial^\alpha a| \times \sum_{|\alpha| \leq n+2N+6} \sup_{{\mathR^n}} |\partial^\alpha \varphi|). \label{eq:I1Tem-staphlema}
	\end{align}

	It can be checked that $\mathop{\rm sgn} \Lambda = \mathop{\rm sgn} \varphi_{xx}(0)$ and $\det \Lambda = \det \varphi_{xx}(0)$.

	We are now almost arrive at \eqref{eq:I-staphlema} except for the explicit computation of the leading term in \eqref{eq:I-staphlema} and \eqref{eq:I1Tem-staphlema}.
	From equality $x = \phi(\Phi(x))$ we know $I = \nabla_y \phi(\Phi(x)) \cdot \nabla_x \Phi(x)$.
	Formula \eqref{eq:xToy-staphlema} implies $\Phi(0) = 0$ and $\nabla_x \Phi(0) = P^T(0)$,
	hence $\det \nabla_y \phi(0) = \det \nabla_y \phi(\Phi(0)) = \big( \det \nabla_x \Phi(0) \big)^{-1} = \big( \det P^T(0) \big)^{-1} = 1$.
	Therefore,
	\begin{equation} \label{eq:fToa-staphlema}
	f(0) = \chi(\phi(0)) \cdot a(\phi(0)) \cdot |\det \nabla_y \phi (0)|
	= \chi(0) \cdot a(0) = a(0).
	\end{equation}
	Combining \eqref{eq:ISplit-staphlema}, \eqref{eq:I2-staphlema}, \eqref{eq:I1Tem-staphlema} and \eqref{eq:fToa-staphlema}, we arrive at the conclusion.
\end{proof}

\begin{bibdiv}
\begin{biblist}

\bib{dim1999spe}{book}{
      author={Dimassi, Mouez},
      author={Sj\"{o}strand, Johannes},
       title={Spectral asymptotics in the semi-classical limit},
      series={London Mathematical Society Lecture Note Series},
   publisher={Cambridge University Press, Cambridge},
        date={1999},
      volume={268},
        ISBN={0-521-66544-2},
         url={https://doi.org/10.1017/CBO9780511662195},
      review={\MR{1735654}},
}

\bib{eskin2011lectures}{book}{
      author={Eskin, Gregory},
       title={Lectures on linear partial differential equations},
   publisher={American Mathematical Society},
        date={2011},
      volume={123},
}

\bib{zw2012semi}{book}{
      author={Zworski, Maciej},
       title={Semiclassical analysis},
      series={Graduate Studies in Mathematics},
   publisher={American Mathematical Society, Providence, RI},
        date={2012},
      volume={138},
        ISBN={978-0-8218-8320-4},
         url={https://doi.org/10.1090/gsm/138},
      review={\MR{2952218}},
}

\end{biblist}
\end{bibdiv}

\end{document}


Compressive Image Classification using Deterministic Sensing Matrices

Abstract: We look at the use of deterministic sensing matrices for compressed sensing and provide worst-case bounds on the classification accuracy of SVMs on compressively sensed data.

\section*{Part 1 - Deterministic Sensing Matrices}
We have studied compressive sensing, where the goal is to capture majority of the \textit{information} in the signal using very few measurements and obtain accuracte reconstructions. One of the most important aspect of compressive sensing was the design of sensing matrices $\Phi$ and we understood that if the sensing matrices obey the Restricted Isometry Property, and obtained worst case performances on the reconstructed signals.

Let the signal be $s$ sparse, let the dimension of the signal be $N$ and let the number of measurements be $m$. In particular, we looked at random sensing matrices that satisfy the RIP property with overwhelming probability with the overwhelming property given that:

\[
    m \geq \mathcal{O}(s\log(N/s))
\]

In our course, we have looked at random sensing matrices, and how they satisfy the RIP property that helps us in obtaining the \textbf{worst} case error bounds. This is analogous to Shannon's coding theory, that also provides a worst case error for a transmission channel and thus the random sensing matrices are associated with certain drawbacks which are described below:

\begin{enumerate}
    \item \textbf{Time Complexity during Reconstruction}: We have looked at a few algorithms to reconstruct signals from the measurements and some of them include the Basis Pursuit algorithm and the Orthogonal Matching Pursuit algorithm. The time complexity of these signals are given in Table \ref{tab:my_label}. In contrast, we can construct efficient reconstruction algorithms with lower time complexities.
    \item \textbf{Storing the Random Sensing Matrix}: This requires significant space, especially if the dimension of signal is large. In contrast, entries of the deterministic sensing matrix can be computed on the fly, and thus no need of storage is required.
    \item \textbf{Verifying the RIP property}: There is no efficient algorithm for verifying whether the given sensing matrix obeys the RIP property. In the course we looked at an (inefficient) algorithm based on subset ennumeration with $\mathcal{O}(N^s)$ time complexity.
\end{enumerate}

\begin{table}[H]
    \centering
    \begin{tabular}{c|c}
        Algorithm & Time Complexity  \\
        \hline
         Basis Pursuit & $\mathcal{O}(N^3)$ \\
         Orthogonal Matching Pursuit & $\mathcal{O}(s^2\log^{\alpha}(N))$ \\
    \end{tabular}
    \caption{Time Complexities of Common Reconstruction Algorithms}
    \label{tab:my_label}
\end{table}

An important thing to consider here is analogous to coding theory, instead of the worst-case error we will analyse the typical (expected error) while studying deterministic sensing matrices.

\subsection*{StRIP and UStRIP}
For dealing with deterministic sensing matrices, we will impose a weaker version of the Restricted Isometry Property, namely the Statistical Restricted Isometry Property.

\begin{defn}[$k, e, \delta$ - StRIP matrix]
An $m \times N$ sensing matrix $\Phi$ is said to be a $k, \epsilon, \delta $ - StRIP matrix, if for $k$ sparse vectors $\boldsymbol{\alpha} \in \mathbb{R}^N$, the following equation:
\[
    (1 - \epsilon)\lVert \boldsymbol{\alpha} \rVert^2 \leq \lVert \phi\boldsymbol{\alpha} \rVert^2 \leq (1 + \epsilon)\lVert \boldsymbol{\boldsymbol{\alpha}} \rVert^2
\]
holds with probability exceeding $1 - \delta$
\end{defn}

Note that this definition does not imply unique reconstruction, even with an exceedingly high probability. This is because the number of $s$-sparse vectors $\boldsymbol{\alpha} \in \mathbb{R}^N$ such that there is a different $s$-sparse $\boldsymbol{\beta} \in \mathbb{R}^N$ for which $\Phi\boldsymbol{\alpha} = \Phi\boldsymbol{\beta}$ being small is a much more strict condition than number of $s$-sparse vectors $\boldsymbol{\beta} \in \mathbb{R}^N$ and $\Phi\boldsymbol{\alpha} = \Phi\boldsymbol{\beta}$ be small. Note that the StRIP definition guaranteed the latter condition but not the former, and this is why we define the Unique Statistical Restricted Isometry Property as follows:

\begin{defn}[$k, e, \delta$ - UStRIP matrix]
An $m \times N$ sensing matrix $\Phi$ is said to be a $k, \epsilon, \delta $ - UStRIP matrix, if $\Phi$ is a ($k, e, \delta$ - StRIP matrix) and
\[
    \{\boldsymbol{\beta} \in \mathbb{R}^N, \Phi\boldsymbol{\alpha} = \Phi\boldsymbol{\beta}\} = \{\boldsymbol{\alpha}\}
\]
holds with probability exceeding $1 - \delta$
\end{defn}

\subsection*{StRIP-able Matrices}

We will now look at a few simple design rules, which are sufficient to guarantee that $\Phi$ is a UStRIP matrix, and these properties are satisfied by a large class of matrices.

\begin{defn}
An $m \times N$ - matrix $\Phi$ is said to be $\eta$ - StRIP-able, where $\eta$ satisfies $0 < \eta < 1$, if the following conditions are satisfied:
\begin{itemize}
    \item \textbf{St1}: The rows of $\Phi$ are orthogonal, and all the row sums are 0
    \item \textbf{St2}: The columns of $\Phi$ form a group under pointwise multiplication as follows:
    \[ \text{For all } j,j' \in \{1,2,....,N\}, \text{ there exists a } j'' \text{ such that } \phi_j\phi_{j'} = \phi_{j''}\]
    \item  \textbf{St3}: For all $j \in \{2,....,N\}$
    \[    |\sum_x \phi_j(x)|^2 \leq N^{2 - \eta}\]
\end{itemize}
\end{defn}

\subsection*{Main Result}
\begin{theorem}\label{theorem:thm1}
Suppose the $m \times N$ matrix $\Phi$ is $\eta$ StRIP-able, and suppose $k < 1 + (m - 1)\epsilon$ and $\eta > 1/2$. Then there exists a constant c such that, if $m \geq \left(c\dfrac{k\log(N)}{\epsilon^2}\right)^{1/\eta}$, then $\Phi$ is $(k, \epsilon, \delta) - UStRIP$ with $\delta = 2\exp\left(-\dfrac{[\epsilon - (k-1)/(N - 1)^2]m^{\eta}}{8k}\right)$
\end{theorem}

The authors also suggest the Quadratic Reconstruction Algorithm, which only requires vector-vector multiplication in the measurement domain, and thus has a sub-linear time complexity with respect to $N$ (the dimension of the signal). The StRIP property provides performance guarantees, because effectively the algorithm returns the location of one of the $k$ significant entries, and the property guarantees that the estimate is within $\epsilon$ of the true value.

\subsection*{Delsarte - Goethals Frames}
Having introduced deterministic matrices for the purpose of compressive classification, we will now look at a particular type of deterministic matrices - the Delsarte-Goethals Frames. We will then show that these matrices are StRIP-able by proving that they obey the properties mentioned above. We will also show that these matrices when used for the purpose of compressive classification using Support Vector Machines give performance bounds with respect to classification in the data domain.

\begin{defn}
A group is a set of elements $\Omega$ with some operation $*$ such that $*$ is associative, invokes an identity element in $\Omega$ and $\forall \ x, y   \in  \Omega, x*y \in \Omega$\\
\end{defn}

\begin{defn}
A Delsarte-Goethals set DG(m, r) is a set of $2^{(r+1)m}$ $m \times m$ binary symmetric matrices with the property that $\forall A, B \in DG(m, r)\ and\ A \neq B, rank(A - B) \geq m - 2r$\\
\end{defn}

\begin{defn}
A Delsarte-Goethals frame G(m, r) is a $2^m \times 2^{(r+2)m}$ matrix with each element defined as:
\[ a_{(P, b), t} = {i^{wt(d_P) + 2wt(b)}\cdot i^{tPt^T + 2bt^T}}/\sqrt{m}\]
where:
\[ t  \text{ is a binary m-tuple used to index rows of the matrix } \]
\[ P \text{ is a } m \times m \text{ matrix in } DG(m, r), \]
\[ b \text{ is a binary m-tuple } \]
\[ (P, b) \text{ indexes rows }\]
\[ wt(\cdot) \text{ is the Hamming weight of a binary vector (i.e.number of ones), } \]
\[ d_P \text{ is the diagonal of } P \]
\end{defn}

\begin{Lemma}\label{Lemma:dg1}
The set of unnormalized columns of G(m, r) denoted as $\Psi_{(P, b)}$ forms a group under point-wise multiplication ($*$).

\end{Lemma}
\begin{proof}
\begin{subLemma}
\begin{align*}
    For\ t \in  \mathbb{F}_2^m: \Psi_{(P, b)}(t) &= i^{wt(d_P) + 2wt(b)}\cdot i^{tPt^T + 2bt^T}\\
    \therefore \Psi_{(P, b)}(t)\Psi_{(P', b')}(t) &= i^{wt(d_P) + wt(d_{P'}) + 2wt(b) + 2wt(b')}\cdot\\
    & \ \ \ \ i^{tPt^T + tP't^T + 2bt^T + 2b't^T}\\
\end{align*}
\end{subLemma}

\begin{subLemma}
Let $\oplus$ denote binary addition, and construct $Q$ such that $P + P' = P \oplus P' + 2Q (mod 4)$. Hence, Q will be binary symmetric too, with $d_Q = dt* d_{P'}$. Moreover, it can be checked that $tQt^T = d_qt^T$, owing to Q being binary symmetric. This implies:
\[ tPt^T + tP't^T = t(P+P')t^T = t(P \oplus P')t^T + 2tQt^T\\ =t(P \oplus P')t^T + 2(d_P * d_{P'})t^T\]
\end{subLemma}

\begin{subLemma}
$2x + 2y = 2(x \oplus y) mod 4$ and $2wt(x) + 2wt(y) = 2wt(x \oplus y)\ (mod 4)$. This can be checked by trying out all 4 possible binary combinations for a corresponding element of $x$ and $y$. This implies:
\[ 2wt(b) + 2wt(b') = 2wt(b \oplus b') \]
\[ 2wt(b \oplus b') = 2wt(b \oplus b' \oplus 0) = 2wt(b \oplus b' \oplus d_P * d_{P'} \oplus d_P * d_{P'})\\= 2wt(b \oplus b' \oplus d_P * d_{P'}) + 2wt(d_P * d_{P'}) \]

\end{subLemma}

\begin{subLemma}
$wt(d_{P \oplus P'}) = wt(d_P) + wt(d_{P'}) + 2wt(d_P * d_{P'})$. This can be checked via taking cases for binary elements of $d_P, d_{P'}$.
\end{subLemma}

Combining all Sub Lemmas gives:
\begin{align*}
    \therefore \Psi_{(P, b)}(t)\Psi_{(P', b')}(t) &= i^{(wt(d_P) + wt(d_{P'}) + 2wt(d_P * d_{P'})) + 2wt(b \oplus b' \oplus d_P * d_{P'})}\cdot\\
    &\ \ \ \ i^{t(P \oplus P')t^T + 2(d_P * d_{P'})t^T + 2bt^T + 2b't^T}\\
    \therefore \Psi_{(P, b)}(t)\Psi_{(P', b')}(t) &= i^{(wt(d_P) + wt(d_{P'}) + 2wt(d_P * d_{P'})) + 2wt(b \oplus b' \oplus d_P * d_{P'})}\cdot\\
    &\ \ \ \ i^{t(P \oplus P')t^T + 2(d_P * d_{P'} \oplus b \oplus b')t^T}\\
    \therefore \Psi_{(P, b)}\Psi_{(P', b')} &= \Psi_{(P \oplus P', b \oplus b' \oplus d_P * d_{P'})}
\end{align*}
\end{proof}

\begin{Lemma}\label{Lemma:dg2}
G(m, r) is a tight frame with redundancy $\frac{n}{m}$. i.e. $G(m, r)G(m, r)^\dagger = \dfrac{nI_{m \times m}}{m}$\\
\end{Lemma}

\begin{proof}
We prove this showing that the inner product of two rows of $G(m, r)$ is $0$ if the rows are unequal and $\frac{n}{m}$ if they are equal.\\
Consider rows of index $t, t' \in \mathbb{F}_2^m$
\begin{align*}
    \therefore <G(m, r)[:, t], G(m, r)[:, t']> &= \sum_{P, b} \dfrac{\Psi_{P, b}(t)}{\sqrt{m}} \overline{\dfrac{\Psi_{P, b}(t')}{\sqrt{m}}}\\
    &= \frac{1}{m} \sum_{P, b} i^{tPt^T - t'Pt'^T + 2bt^T - 2bt'^T}\\
    &= \frac{1}{m} \sum_{P}  i^{tPt^T - t'Pt'^T} \sum_{b} i^{2bt^T - 2bt'^T}\\
    &= \frac{1}{m} \sum_{P}  i^{tPt^T - t'Pt'^T} \sum_{b} i^{2b(t \oplus t')^T}\\
    &= \frac{1}{m} \sum_{P}  i^{tPt^T - t'Pt'^T} \sum_{b} (-1)^{b(t \oplus t')^T}\\
    &= \frac{1}{m} \sum_{P}  i^{tPt^T - t'Pt'^T} \sum_{b' \in \mathbb{F}_2^{m'}} (-1)^{wt(b')}\\
    &\text{(where $m' = wt(t \oplus t')$)}\\
    &= \begin{cases}
    0\ if\ t \oplus t' \neq \textbf{0},\\
    \frac{n}{m}\ if\ t \oplus t' = \textbf{0}
    \end{cases}
\end{align*}
Notice that $m' = 0 \iff t \oplus t' = \textbf{0}$. Also, the pre-final step is justified because dot product with $(t \oplus t')$ is equivalent to adding those indexes of $b$ where $t \oplus t'$ is $1$, and since $b$ is varied over $\mathbb{F}_2^m$, this dot product is equivalent to the hamming weight of a binary sub-tuple of $b$ that selects the corresponding indices. Finally, it can easily shown that $\forall m' \geq 1, \mathbb{F}_2^{m'}$ has as many odd-hamming-weight vectors as even-hamming-weight ones, implying that the above sum is 0 for $m' \geq 1$.

\end{proof}

\begin{Lemma}\label{Lemma:dg3}
The column sums in the  $DG(m,r)$ satisfy:
\[ |\sum_{x}\phi_{P,b}(x)|^2 = 0 \text{ or } N^{2 - r/m}\]
\end{Lemma}
Now given Lemmas \ref{Lemma:dg1},\ref{Lemma:dg2} and \ref{Lemma:dg3}, we can conclude that $DG(m,r)$ frame is $r/m$ StRIP-able and thus obeys the UStRIP

\section*{Part 2 - Support Vector Machines for Compressive classification}

\subsection*{Classification using Support Vector Machines}
Before showing performance bounds on deterministic matrices, we will start off by first proving the error on performance bounds on matrices obeying the Restricted Isometry Property. Note that, random matrices obey the RIP property with overwhelming probability given that the number of measurments is large. The theorem can then be extended to deterministic matrices obeying the UStRIP.

\subsubsection*{Notation}

\textbf{Data domain:} $\mathcal{X} = \{(\boldsymbol{x}, y): \boldsymbol{x} \in \mathbb{R}^n, y \in \{-1, 1\}, ||\boldsymbol{x}||_0 \leq k, ||\boldsymbol{x}||_2 \leq R\}$.\\

\textbf{Compressed sensing measurement matrix} $A \in \mathbb{R}^{m \times n}$.\\

\textbf{Measurement domain} $\mathcal{M} = \{(A\boldsymbol{x}, y): (\boldsymbol{x}, y) \in \mathcal{X}\}$.\\

Let the data be drawn from some unknown distribution $\mathcal{D}$ over $\mathcal{X}$. $S =\ <(x_1, y_1), ..., (x_M, y_M)>$ is a set of $M$ i.i.d samples from $\mathcal{D}$. $AS =\ <(Ax_1, y_1), ..., (Ax_M, y_M)>$.
Any linear classifier $w(x)$ corresponds to some $w \in \mathbb{R}^n$ such that $\boldsymbol{w(x)} = sign(\boldsymbol{w^Tx})$.\\

We will now define the loss functions as follows:
\begin{defn}
\begin{align*}
\text{The True Hinge loss }    H_{\mathcal{D}}(\boldsymbol{w}) &= E_{(\boldsymbol{x, y}) \sim \mathcal{D}} [1 - y\boldsymbol{w}^T\boldsymbol{x}]\\
\text{The Empirical Hinge loss }
    \hat{H}_{\mathcal{S}}(\boldsymbol{w}) &= E_{(\boldsymbol{x_i}, y_i) \sim S} [1 - y_i\boldsymbol{w}^T\boldsymbol{x}_i]\\
\text{The True Reguralisation loss }
    L(\boldsymbol{w}) &= H_{\mathcal{D}}(\boldsymbol{\boldsymbol{w}}) + \dfrac{||\boldsymbol{w}||^2}{2C}\\
\text{The Emperical Reguralisation loss }
    \hat{L}(\boldsymbol{w}) &= \hat{H}_{\mathcal{S}}(\boldsymbol{w}) + \dfrac{||\boldsymbol{w}||^2}{2C}\\
\end{align*}
\end{defn}

The $\boldsymbol{w}$ for soft margin SVM classifier that minimizes $\hat{L}(\boldsymbol{w})$ can be expressed as:
\begin{align*}
     \boldsymbol{w} &= \sum_{i = 1}^M \alpha_i y_i \boldsymbol{x_i}\\
    where\ &0 \leq \alpha_i \leq \frac{C}{M}\ \forall i\\
    and\ &||\boldsymbol{w}||^2 \leq C\\
\end{align*}
\clearpage
We now define the following classifiers:
\begin{defn}
\begin{align*}
    \boldsymbol{w^*} &= arg\min_w L(\boldsymbol{w}) \text{ (in data domain)}\\
    \boldsymbol{z^*} &= arg\min_z L(\boldsymbol{z}) \text{ (in measurement domain)}\\
    \boldsymbol{\hat{w}}_S &= arg\min_w \hat{L}(\boldsymbol{w}) \text{ (in data domain)}\\
    \boldsymbol{\hat{z}}_{AS} &= arg\min_z \hat{L}(\boldsymbol{z}) \text{ (in measurement domain)}\\
\end{align*}
\end{defn}

\subsection*{Main Result}
We prove that the SVM classifier trained on the measurement domain performs, with high probability, almost as well as the best classifier in the data domain. This is done by bounding the difference of losses between (a) the best data domain classifier and the data domain trained SVM, (b) the data domain SVM and its projection onto the measurement domain, and (c) this projection and the SVM trained directly on the measurement domain.\\

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{svm_model.png}
    \caption{Hybrid Model for proving the performance bounds of the SVM classifier on the measurement domain - [3]}
    \label{fig:my_label}
\end{figure}

\begin{theorem}\label{theorem:thm2}
Let $A$ obey the RIP with $\delta_{2k} = \epsilon$, $AS, \hat{z}_{AS}$ be as defined above, and $w_0$ be the best data domain classifier. Then we have:
\[ H_D(\hat{\boldsymbol{z}}_{AS}) \leq H_D(\boldsymbol{w_0}) + \mathcal{O}\Bigg(\sqrt{||\boldsymbol{w_0}||^2\bigg(R^2 \epsilon + \frac{log(1/\delta)}{M}}\Bigg) \]

\end{theorem}

\begin{Lemma}\label{Lemma:lem10}
\[ (1 + \epsilon)\boldsymbol{x}^T\boldsymbol{x}' - 2R^2\epsilon \leq (A\boldsymbol{x})^T(A\boldsymbol{x}') \leq (1 - \epsilon)\boldsymbol{x}^T\boldsymbol{x}' + 2R^2\epsilon \]

\end{Lemma}

\ref{Lemma:lem10} essentially preserves the inner product in the measurement domain. This Lemma has been proved. We can now extend Lemma \ref{Lemma:lem10} to the inner product of linear combinations of sparse signals as follows.\\
\begin{Lemma}\label{Lemma:lem11}
Let $(\boldsymbol{x_1}, y_1), ..., (\boldsymbol{x_M}, y_M), (\boldsymbol{x_1}', y_1'), ..., (\boldsymbol{x_N}', y_N') \in \mathcal{X}$ and $\alpha_1, ..., \alpha_M, \beta_1, ..., \beta_N$ be non-negative numbers with $\sum_{i=1}^M \alpha_i \leq C$ and $\sum_{j=1}^N \beta_j \leq D$. Then for $\alpha = \sum_{i=1}^M \alpha_i y_i \boldsymbol{x_i}$, $\beta = \sum_{j=1}^N \beta_j y_j' \boldsymbol{x_j}' \leq D$, we have:
\[ |\boldsymbol{\beta}^T \boldsymbol{\alpha} - (A\boldsymbol{\beta})^T (A\boldsymbol{\alpha})| \leq 3CDR^2\epsilon \]
\end{Lemma}

The above result follows easily from the previously mentioned Lemma, and has been proved in Ref. 2

We will now prove one of the core results that connects the regularization loss of the data domain and the measurement domain
\begin{theorem}\label{theorem:thm3}
Let $\hat{\boldsymbol{w}}_S$ be as defined previously, and $A\hat{\boldsymbol{w}}_S$ be its projection. Then:
\[ L(A\hat{\boldsymbol{w}}_S) \leq L(\hat{\boldsymbol{w}}_S) + O(CR^2\epsilon) \]
Therefore, there exists at least one classifier in the measurement domain that doesn't perform much worse than the SVM classifier in the data domain.
\end{theorem}

\begin{proof}
We can write $\hat{\boldsymbol{w}}_S = \sum_{i = 1}^M \alpha_i y_i \boldsymbol{x_i}$, as seen above. Using Lemma \textcolor{red}{\ref{Lemma:lem11}}, with $N=M, D=C, (\boldsymbol{x_i}', y_i') = (\boldsymbol{x_i}, y_i)$, we get:
\begin{align*}
    (A\boldsymbol{w})^T(A\boldsymbol{w}) &\leq \boldsymbol{w}^T\boldsymbol{\boldsymbol{w}} + 3C^2R^2\epsilon\\
    \therefore \dfrac{||A\hat{\boldsymbol{w}}_S||^2}{2C} &\leq \dfrac{||\hat{\boldsymbol{\boldsymbol{w}}}_S||^2}{2C} + \mathcal{O}(CR^2\epsilon)\\
\end{align*}
Using Lemma \textcolor{red}{\ref{Lemma:lem11}}, with $N=M=1, D=C=1, (\boldsymbol{x_1}', y_1') = (\boldsymbol{x}, y)$, we get:
\begin{align*}
    1 - y(A\hat{\boldsymbol{w}}_S)^T(A\boldsymbol{x}) &\leq 1 - y\hat{\boldsymbol{w}}_S^T x + \mathcal{O}(CR^2\epsilon)\\
    \therefore H_D(-y(A\hat{\boldsymbol{w}}_S)^T(A\boldsymbol{x})) &\leq H(- y\hat{\boldsymbol{w}}_S^T \boldsymbol{x})+ \mathcal{O}(CR^2\epsilon)\\
\end{align*}

Since $A$ is one-one mapping (due to RIP), we can take expectation over $\mathcal{D}$ to complete the proof.
\end{proof}

As the pre-final step, we will now show that the regularisation loss of the SVM's classifier in the measurement domain ($\hat{\boldsymbol{z}}_{AS}$) is close to the projected regularisation loss $A\hat{\boldsymbol{w}}_S$.
We will also show the regularisation loss of the SVM's classifier ($\hat{\boldsymbol{w}_S}$) in the data domain is close to the best classifier in the data domain ($\boldsymbol{w_0}$)

\begin{theorem}[\textbf{Sridharan 2008 [4]}]
For all $\boldsymbol{w}$ with $\lVert \boldsymbol{w} \rVert^2 \leq 2C$, with probability at least 1 - $\delta$ over training set:
\[
    L_{\mathcal{D}}(\boldsymbol{w}) - L_{\mathcal{D}}(\boldsymbol{w^*})\leq 2[\hat{L_{\mathcal{S}}}(\boldsymbol{w}) - \hat{L_{\mathcal{S}}}(\boldsymbol{w^*})]_+ + O\left(\dfrac{C\log(1/ \delta)}{M}\right)
\]
\end{theorem}

\begin{corollary}\label{cor:cor1}
Let $\hat{\boldsymbol{w}}_S$ be the SVM's classifier. Then with probability $1 - \delta$
\[ L_{\mathcal{D}}(\hat{\boldsymbol{w}}_S) \leq L_{\mathcal{D}}(\boldsymbol{w^*}) + O\left(\dfrac{C\log(1/ \delta)}{M}\right)\]
\end{corollary}

\begin{proof}
This directly follows from the fact that the SVM's classifier minimises the empirical regularisation loss, thus we have:
\[ \hat{L_{\mathcal{S}}}(\boldsymbol{w}) \leq \hat{L_{\mathcal{S}}}(\boldsymbol{w^*})\]

This can be now used to deduce the corollary.
\end{proof}

We are now ready to prove the main result of this section, i.e Theorem \ref{theorem:thm2}

\begin{proof}
By corollary \ref{cor:cor1}, we have a relation between the regularisation loss of the SVM's classifier in the measurement domain ($\hat{\boldsymbol{z}}_{AS}$) and the regularisation loss of the best classifier in the measurement domain ($\boldsymbol{z^*}$)
\[ L_{\mathcal{D}}(\hat{\boldsymbol{z}}_{AS}) \leq L_{\mathcal{D}}(\boldsymbol{z^*}) + O\left(\dfrac{C\log(1/ \delta)}{M}\right)\]

We also know that $\boldsymbol{z^*}$ is the best classifier in the measurement domain, thus we have:
\[ L_{\mathcal{D}}(\boldsymbol{z^*}) \leq L_{\mathcal{D}}(\boldsymbol{A\hat{w}_S})\]

We have bounded $L_{\mathcal{D}}(\boldsymbol{A\hat{w}_S})$ in Theorem \ref{theorem:thm3} by the regularization loss of the best classifier $\boldsymbol{w^*}$

\[ L_{\mathcal{D}}(\boldsymbol{A\hat{w}_S}) \leq L_{\mathcal{D}}(\boldsymbol{\hat{w}_S}) + O(CR^2\epsilon)\]
We can now apply the same corollary to the data domain to connect the the regularisation loss of the SVM's classifier in the data domain to the regularisation loss of the best classifier in the data domain ($\boldsymbol{w}^*$)
\[ L_{\mathcal{D}}(\hat{\boldsymbol{w}}_{AS}) \leq L_{\mathcal{D}}(\boldsymbol{w^*}) + O\left(\dfrac{C\log(1/ \delta)}{M}\right)\]

We can now complete the proof by noting that for any good classifier on $\mathcal{S}$ (say  $\boldsymbol{w_0}$), we have:
\[ L_{\mathcal{D}}(\boldsymbol{w^*}) \leq L_{\mathcal{D}}(\boldsymbol{w_0})\]
We can now combine all the inequalities to get:
\[ H_{\mathcal{D}}(\hat{\boldsymbol{z}}_{AS}) \leq H_{\mathcal{D}}(\boldsymbol{w_0}) + \dfrac{1}{2C}\lVert \boldsymbol{w_0} \rVert^2 + O\left(CR^2\epsilon + \dfrac{Clog(1/\delta)}{M}\right) \]

\end{proof}

\subsection*{Result for Deterministic Matrices}
We will now extend Theorem \ref{theorem:thm2} for deterministic matrices. Unfortunately, this theorem has just been mentioned in the paper but not proven by the authors.
\begin{theorem}\label{theorem:thm4}
Let $o$ be an odd integer, and let $r \leq \dfrac{o-1}{2}$. Let A be an $m \times n$ DG frame with $m = 2^o$, and $n = 2^{(r+2)o}$. Let $\boldsymbol{w_0}$ denote the data-domain oracle classifier. Also let S = $\langle (\boldsymbol{x_1, l_1}), (\boldsymbol{x_M, l_M})\rangle$ represent $M$ training examples in the data-domain, and let AS = $\langle (\boldsymbol{y_1, l_1}), (\boldsymbol{y_M, l_M})\rangle$ denote the training examples in the measurement domain. Let $\hat{\boldsymbol{z}}_{AS}$ denote the measurement domain SVM classifier trained on S. Then there exists a universal constant $C$ such that if:
\[ m \geq \left( \dfrac{2^{r+1}C\log n}{\epsilon_1}\right)^2\] then with probability atleast $1 - \dfrac{6}{n}$, $H_{\mathcal{D}}(\boldsymbol{\hat{z}}_{AS}) - H_{\mathcal{D}}(\boldsymbol{\hat{w}}_{0})$ is \[ O\left(  R\lVert \boldsymbol{w_0}\rVert\sqrt{\left(
\dfrac{2^r(\log M + \log N)}{\sqrt{m}} + \sigma + \dfrac{((1+\epsilon_1)\log n)}{M}\right)}\right) \]
\end{theorem}
\subsection*{Results}
We have used the Brodatz Texture Database that contains 111 images of the textures - Horizontal, Vertical and None. We used a 50 \% train-test split and used an SVM classifier on the data domain (on the wavelet coefficients) and the compressed domain (randomly generated Gaussian matrices as sensing matrices)
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{Figure_1_1.png}
    \caption{Training Accuracy vs Number of Measurements in Uncompressive and Compressive Sensing using an SVM classifier. We see that as $m$, the number of compressive measurements increases, the accuracy of SVM on the compressed data approaches that on the original, uncompressed data. Moreover, we see that for $m=200$, we have almost equal classification accuracy, implying a compression of $\frac{38*38}{200}$ = 7.22}
    \label{fig:my_label}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{Figure_2.png}
    \caption{Test Accuracy vs Number of Measurements in Uncompressive and Compressive Sensing using an SVM classifier}
    \label{fig:my_label}
\end{figure}
\section*{Part 3 - An Attempt at Using Fischers Linear Discriminant on Image Classification}
FLD is a transform learning based image classification technique that learns a transformation vector $\boldsymbol{W}$ given a set of input points $\boldsymbol{x_i}$, and classifies the transformed points $\boldsymbol{y_i}$ given by:
        \[ \boldsymbol{y_i = w^tx_i}\]
        Let $\boldsymbol{m_i}$ denote the mean vector of each class and $\boldsymbol{m}$ denote the mean vector of the entire data, i.e:
        \[ \boldsymbol{m_i} = \sum_{x_j \in C_i} \boldsymbol{x_j}/n_i\]
        \[ \boldsymbol{m} = \sum_{i=1}^{c}\sum_{x_j \in C_i} \boldsymbol{x_j}/N\]

        We define the following three matrices:
        \begin{enumerate}

        \item \[ \boldsymbol{S_W} = \sum_{i=1}^{c}\sum_{x_j \in C_i}(\boldsymbol{x_j - m_i})(\boldsymbol{x_j - m_i})^T \]
        ($\boldsymbol{S_W}$ is the within class scatter matrix)
        \item \[ \boldsymbol{S_B} = \sum_{i=1}^{c}\sum_{x_j \in C_i}(\boldsymbol{m_i - m})(\boldsymbol{m_i - m})^T\]
        ($\boldsymbol{S_B}$ is the between class scatter matrix)
        \item \[ \boldsymbol{S_T} = \boldsymbol{S_B} + \boldsymbol{S_W} = \sum_{j = 1}^{N}(\boldsymbol{x_j - m})(\boldsymbol{x_j - m})^T\]
        ($\boldsymbol{S_T}$ is the between class scatter matrix)
        \end{enumerate}

         The optimisation function (Fischers criterion) which we maximise with respect to the transformation vector $\boldsymbol{w}$ is given by:
    \[ \boldsymbol{J(W) = \dfrac{|W^TS_BW|}{|W^TS_WW|}} \]

Essentially, a good $\boldsymbol{W}$ should transform the mean vectors of each class as far as possible \textbf{and, (this is also what sets apart FLD from PCA)} relative to the variances of the transformed vectors, the difference between mean vectors have to be large (to avoid conflation - overlap of points).

The Fischers criterion is a generalised eigenvalue problem, and the solution is a set of $c-1 \ (\boldsymbol{S_B} \text{ is a sum of rank 1 matrices})$ generalised eigenvectors of the following form:
\[ \boldsymbol{S_Bw_i} = \lambda_i\boldsymbol{S_Ww_i}\]
\[ \boldsymbol{S_W^{-1}S_Bw_i} = \boldsymbol{S_Ww_i}\]

We can see that by definition of $\boldsymbol{S_W} \in \mathbb{R}^{n \times n}$, its rank is at most $N - c$ (we subtract $c$, since the vectors are normalised with respect to their mean vector $\boldsymbol{m_i}$. This is results in $\boldsymbol{S_W}$ becoming singular and thus non-invertible.

 To circumvent this problem, we project the points $\boldsymbol{x_i}$ onto an $N-c$ dimensional space and then perform FLD to reduce the dimensionality to $c-1$
        Let the transformation learned by PCA be given by $W_{pca}$ and let the transformation learned by FLD be given by $W_{fld}$. The modified FLD criterion becomes:
        \[ \boldsymbol{W_{pca} = argmax_{\boldsymbol{W}} |\boldsymbol{W^TS_WW}|}\]

        \[ \boldsymbol{W_{fld} = argmax_{\boldsymbol{W}} \dfrac{|\boldsymbol{W^TW_{pca}^TS_BW_{pca}W}|}{|\boldsymbol{W^TW_{pca}^TS_WW_{pca}W}|}}\]

\subsection*{Results}

We were successfully able to implement the FLD (and PCA) algorithm on the training set (as shown in Figure \ref{fig:fig3}), but unfortunately we did not get the desired results on the test set. More specifically, as shown in Figure \ref{fig:fig4}, we did not get the desired clusters of points.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.8]{fld_1.png}
    \caption{FLD (+ PCA) algorithm on the training set}
    \label{fig:fig3}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.8]{fld_2.png}
    \caption{Training Accuracy vs Number of Measurements in Uncompressive and Compressive Sensing using an SVM classifier}
    \label{fig:fig4}
\end{figure}

\section*{References}
\begin{enumerate}
    \item \href{https://ieeexplore.ieee.org/document/6288656}{FINDING NEEDLES IN COMPRESSED HAYSTACKS}, \textit{Robert Calderbank and Sina Jafarpour}

    \item \href{https://ieeexplore.ieee.org/document/5419073}{Construction of a Large Class of Deterministic Sensing Matrices that Satisfy a Statistical Isometry Property} \textit{Robert Calderbank, Fellow, IEEE, Stephen Howard, Member, IEEE, and Sina Jafarpour, Student Member, IEEE}

    \item \href{http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.154.7564}{Compressed Learning:
Universal Sparse Dimensionality Reduction and Learning in the Measurement
Domain}
\textit{Robert Calderbank, Sina Jafarpour, Robert Schapire}

\item \href{http://ttic.uchicago.
edu/˜karthik/con.pdf, 2008.}{Fast
convergence rates for excess regularized risk with application
to SVM}\textit{K. Sridharan, S. Shalev-Shwartz, and N. Srebro. }

\item \href{https://cseweb.ucsd.edu/classes/wi14/cse152-a/fisherface-pami97.pdf}{Eigenfaces vs. Fisherfaces: Recognition
Using Class Specific Linear Projection} \textit{Peter N. Belhumeur, Joao~ P. Hespanha, and David J. Kriegman}
\end{enumerate}
}
\end{document}