Quantization and Compressive Sensing

Abstract: Quantization is an essential step in digitizing signals, and, therefore, an indispensable component of any modern acquisition system. This book chapter explores the interaction of quantization and compressive sensing and examines practical quantization strategies for compressive acquisition systems. Specifically, we first provide a brief overview of quantization and examine fundamental performance bounds applicable to any quantization approach. Next, we consider several forms of scalar quantizers, namely uniform, non-uniform, and 1-bit. We provide performance bounds and fundamental analysis, as well as practical quantizer designs and reconstruction algorithms that account for quantization. Furthermore, we provide an overview of Sigma-Delta ($\Sigma\Delta$) quantization in the compressed sensing context, and also discuss implementation issues, recovery algorithms and performance bounds. As we demonstrate, proper accounting for quantization and careful quantizer design has significant impact in the performance of a compressive acquisition system.

\subsection{1-Bit Compressive Sensing}
\label{sec:1bitCS}

The simplest scalar quantizer design to implement in hardware is a
1-bit quantizer, which only computes the sign of its input. Its
simplicity makes it quite appealing for compressive sensing systems.

The sensing model of 1-bit CS, first introduced in~\cite{BB_CISS08},
is very similar to the standard scalar quantization model
\begin{equation}
  \label{eq:1bit-CS}
  \ensuremath{q} = \sign(A x),
\end{equation}
where $\sign(x_i)$ is a scalar function applied element-wise to its
input and equals $1$ if $x_i\ge 0$ and $-1$ otherwise.

One of the challenges of this model is that it is invariant under changes of the
signal amplitude since $\sign(cx)=\sign(x)$ for any positive $c$. For
that reason, enforcing consistency is not straightforward. A signal
can be scaled arbitrarily and still be consistent with the
measurements. Thus, a magnitude constraint is typically necessary. Of
course, the signal can only be recovered within a positive scalar
factor.

Similarly to multi-bit scalar quantization models, the literature in
this area focuses on deriving lower bounds for the achievable
performance, reconstruction guarantees, as well as practical
algorithms to invert this problem.

\begin{figure}[!Htb]
  \centering
  \subfigure[Orthants in measurement space]{
    \includegraphics[width=.43\linewidth]{Orthant_octants}
  }
  \subfigure[Orthant intersection]{
    \includegraphics[width=.43\linewidth]{Orthant_qpoints}
  }
  \subfigure[Signal space consistency]{
    \includegraphics[width=.5\linewidth]{1bit-cs-intuition}
  }
   \caption{Behavior of 1-bit measurements. (a) The measurement space,
     $\mathbb{R}^m$ is separated to high-dimensional orthant,
     according to the sign of each orthant. (b) Signals in a
     $k$-dimensional space ($k<m$) will only map to a $k$ dimensional
     subspace of $\mathbb{R}^m$ and intersect only a few orthants of
     the measurement space. (c) The same behavior in the signal
     space. Each measurement vector defines its orthogonal
     hyperplane. The measurement sign identifies which side of the
     hyperplane the signal lies on; all signals in the shaded region
     have consistent measurements. Newer measurements provide less and
     less information; the chance of intersecting the consistency
     region decreases.}
  \label{fig:1bit-cs-intuition}
\end{figure}

\subsubsection{Theoretical Performance Bounds}
\label{bounds}

A lower bound on the achievable performance, can be derived using a
similar analysis as in Sec.~\ref{sec:perf-bounds-sparse}. The main
difference is that the quantization cells are now orthants in the
$m$-dimensional space, shown in Fig.~\ref{fig:1bit-cs-intuition}(a),
corresponding to each measured sign pattern. Each subspace of the ${n
  \choose k}$ possible ones intersects very few of those orthants, as
shown in Fig.~\ref{fig:1bit-cs-intuition}(b), \emph{i.e.},  uses very few
quantization points. In total, at most $I \le 2^k {n \choose k}{m
  \choose k}$ quantization cells are intersected by the union of all
subspaces~\cite{jacques2013robust}.

Since the signal amplitude cannot be recovered, the lower bound is
derived on $k$-dimensional spheres and coverings using spherical caps
instead of balls. The derivation ensures that the spherical caps have
radius sufficiently large to cover the ${n \choose k}$
spheres. Despite the similarity to the argument in
Sec.~\ref{sec:perf-bounds-sparse}, this case requires a little bit
more care in the derivation; details can be found
in~\cite{jacques2013robust}. Still, the result is very similar in
nature. Defining $\Sigma_{k}^{*}:=\{x\in \Sigma_k, \|x\|_2 =1 \},$ we have:
\begin{theorem}[\cite{jacques2013robust}]
  \label{res:lower_bound}
  Given $x \in \Sigma_{k}^{*}$, any
  estimation $\hat x \in \Sigma^*_k$ of $x$ obtained from $q = \sign(A x)$ has
  a reconstruction error of at least
  $$
    \|\hat x - x\|\ \gtrsim\ \frac{k}{m + k^{3/2}},
  $$
  which is on the order of $\tfrac{k}{m}$ as $m$ increases.
\end{theorem}

If the sensing matrix $A$ is Gaussian, \emph{i.e.},  if $a_{ij} \sim_{\rm iid}
\mathcal N(0,1)$, any $k$-sparse signal that has consistent measurements
will not be very far from the signal producing the measurements,
assuming a sufficient number of them. This guarantee approaches the
lower bound of Theorem~\ref{res:lower_bound} within a logarithmic factor.

\begin{theorem}[\cite{jacques2013robust}]
\label{thm:hash}
Fix $0\leq \eta\leq 1$ and $\epsilon_{o}>0$.  If the number of
measurements is
\begin{equation}
\label{eq:gaussian-number-meas-consist}
m \geq \tfrac{2}{\epsilon_{o}}\,\big(2k\,\log(n) + 4k
\log(\tfrac{17}{\epsilon_{o}}) + \log
\tinv{\eta}\big),
\end{equation}
then for all $x, x'\in \Sigma^*_k$  we have that
\begin{equation}
\label{eq:radius-implication}
\|x - x'\|_{2} > \epsilon_{o}\ \Rightarrow\ \sign( A x) \neq \sign (A
x'),
\end{equation}
with probability higher than $1-\eta$. Equivalently, if $m$ and $k$ are given, solving for $\epsilon_0$ above leads to
\begin{equation}
\label{eq:radius-implication-bound}
\|x - x'\|_{2} \lesssim \tfrac{k}{m} \log \tfrac{mn}{k},
\end{equation}
with the same probability.
\end{theorem}

Fig.~\ref{fig:1bit-cs-intuition}(c) provides further intuition on
these bounds by illustrating how 1-bit measurements operate in the
signal space. Specifically, each measurement corresponds to a
hyperplane in the signal space, orthogonal to the measurement
vector. The measurement sign determines on which side of the
hyperplane the signal lies. Furthermore, the signal is sparse, \emph{i.e.},
lies in $\Sigma_k^*$. A consistent sparse reconstruction algorithm can
produce any sparse signal in the indicated shaded region.

A new measurement provides new information about the signal only if
the corresponding hyperplane intersects the region of consistent
signals and, therefore, shrinks it. However, as more measurements are
obtained and the consistency region shrinks, newer measurements have
lower probability of intersecting that region and providing new
information, leading to the $1/m$ decay of the error.

\
Consistency can be quantified using the normalized hamming distance
between measurements
$$
d_H(\ensuremath{q},\ensuremath{q}')=\frac{1}{m}\sum_i\ensuremath{q}_i\oplus\ensuremath{q}_i',
$$ where $\oplus$ denotes the exlusive-OR operator.  It is, thus,
possible to show that if $x$ and $x'$ above differ by no more than $s$
bits in their 1-bit measurements, \emph{i.e.},  if $d_H(\sign(Ax),\sign(Ax'))
\leq s/m$, then, with $m \gtrsim \tfrac{1}{\epsilon_0}\,k
\log \max(m,n)$ and with high probability \cite{QIHT},
$$
\|x - x'\|_{2} \leq \tfrac{k+s}{k} \epsilon_{o}.
$$

A bound similar to \eqref{eq:radius-implication-bound} exists for sign
measurements of non-sparse signals in the context of quantization
using frame permutations~\cite{vivekQuantFrame}. In particular,
reconstruction from sign measurements of signals exhibits (almost
surely) an asymptotic error decay rate arbitrarily close to
$O(1/m)$. However, in contrast to Theorem~\ref{thm:hash} this result
holds only for a fixed signal and not uniformly for all signals of
interest.

Note that these results focus on matrices generated using the normal
distribution. It has been shown that matrices generated from certain
distributions do not perform well in this setting, even though they
can be used in standard compressive
sensing~\cite{plan2011dimension}. For instance, consider a random
Bernoulli matrix $A$ such that $a_{ij} = 1$ or $-1$ with equal
probability. In this case, the two distinct sparse vectors
$(1,0,\cdots,0)^T$ and $(1,\lambda,0,\cdots,0)^T$ with $0\leq \lambda
<1$ are $\lambda$ apart and they generate the same quantization vector
$q=\sign(A_1)$, where $A_1$ is the first column of $A$. It is not
possible, therefore, to distinguish those two vectors from their 1-bit
observations by increasing $m$ and guarantee that the reconstruction
error will decay as measurements increase. This counterexample,
however, is exceptional in the sense that such failures can only
happen if the signal can have a very large entry. Under mild flatness
assumptions on the $\ell_\infty$-norm of the signal, arbitrary
subgaussian measurements can be utilized~\cite{ai2014one}.

These results establish lower and upper bounds on distances between
two sparse signals that have (almost) consistent 1-bit
measurements. It is also possible to provide an embedding guarantee
similar to the RIP~\cite{candes2006ssr}. Since the measurement does
not preserve the signal magnitude, we should not expect distances of
signals to be preserved. However, the measurements do preserve angles
between signals.
Defining $
d_S(u,v) = \tinv{\pi} \arccos(u^T v),\quad u,v\in S^{n-1},$ we have:

\begin{theorem}[Binary $\epsilon$-Stable Embedding (B$\epsilon$SE) \cite{jacques2013robust}]
\label{thm:1-bit-stable-embed}
Let $A \in \mathbb R^{m\times n} $ be a random Gaussian matrix such that
$a_{ij} \sim_{\rm iid} \mathcal N(0,1)$.  Fix $0\leq \eta\leq 1$ and $\epsilon>0$.  If the
number of measurements satisfies
\begin{equation}
\label{eq:numM}
m\ \geq\ \tfrac{2}{\epsilon^2}\big(k\,\log(n) +
2k\,\log(\tfrac{35}{\epsilon}) + \log(\tfrac{2}{\eta})
\big),
\end{equation}
then with probability exceeding $1-\eta$
\begin{equation}
  \label{eq:bse-def}
  d_S(x,x') - \epsilon \leq d_H(\sign(Ax), \sign(Ax')) \leq d_S(x,x')
  + \epsilon,
\end{equation}
for all $x,x' \in \Sigma^*_k$.
\end{theorem}

In other words, up to an additive distortion that decays as
$\epsilon\lesssim (\tfrac{k}{m} \log \tfrac{mn}{k})^{1/2}$, the Hamming
distance between $\sign(Ax)$ and $\sign(Ax')$ tends to concentrate
around the angular distance between $x$ and $x'$. Notice that, in
contrast to the RIP, a vanishing distance between the quantized
measurements of two signals does not imply they are equal, \emph{i.e.},  we observe a (restricted) \emph{quasi-isometry}
between $\Sigma^*_k$ and $\sign(A \Sigma^*_k)$ instead of the common
RIP \cite{jacques2013quantized}. This comes from the additive nature of the distortion in
\eqref{eq:bse-def} and is a direct effect of the inherent ambiguity
due to quantization.

This embedding result has been extended to signals belonging to
convex sets $\mathcal K\subset \mathbb R^n$ provided that their Gaussian mean
width
\begin{equation}
  \label{eq:gaussian-width}
  w(\mathcal K) = \mathbb E\, \sup\{ u^T g: u\,\in\,\mathcal K\! -\! \mathcal K\},\quad g \sim
  \mathcal N(0,{\rm I}_{n\times n}),
\end{equation}
with \mbox{$\mathcal K\!\!-\!\mathcal K :=\{v - v': v,v' \in \mathcal K\}$}, can be computed
\cite{plan2013one,plan2012robust,plan2011dimension}. In particular, if
$$
m \geq C \epsilon^{-6} w^2(\mathcal K)
$$ for some constant $C>0$, then \eqref{eq:bse-def} holds with high
probability for any $x,x' \in \mathcal K \cap S^{n-1}$. In particular, for
$$
\mathcal K = K_{n,k} := \{u \in \mathbb R^n: \|u\|_1 \leq k^{1/2}, \|u\|_2 \leq 1\},
$$ since $w^2(K_{n,k}) = O(k \log n/k)$ \cite{plan2012robust}, an
embedding exists between the set of compressible vectors modeled by $K_{n,k}$
and  $\{-1,+1\}^m$ provided that $m \geq C \epsilon^{-6}\,k \log n/k$.

Note that generalizations of these embeddings to non-linear functions
other than the $\sign$ operator, or to stochastic processes whose
expectation is characterizable by such functions, are also possible
\cite{plan2012robust}.

\subsubsection{Reconstruction from 1-Bit Measurements}
\label{sec:reconstr-greedy-conv}
The original efforts in reconstructing from 1-bit measurements
enforced $\|x\|_2=1$ as a reconstruction constraint, formulating the
non-convex $\ell_1$ minimization problem
\begin{align}
  \hat{x}=\arg\min_x\|x\|_1,~\mathrm{s.t.}~\ensuremath{q}=\sign(Ax),~\|x\|_2=1. \label{eq:1bmin}
\end{align}
Even though the problem is not convex, a number of algorithms have
been shown experimentally to converge to the
solution~\cite{BB_CISS08,LasWenYin::2010::Trust-but-verify:}. More
recently, a number of greedy algorithmic alternatives have also been
proposed~\cite{Bou::2009::Greedy-sparse,bahmani2013robust,jacques2013robust}.

Most of these algorithms attempt to enforce consistency by introducing
a one-sided penalty for sign violations
\begin{align}
  J(Az,\ensuremath{q}) = \| (\ensuremath{q} \circ A z)_- \|_q,
  \label{eq:one-sided}
\end{align}
where $\circ$ is the element-wise product between vectors, $(y_i)_- =
y_i$ if $y_i$ is negative and 0 otherwise, also applied element-wise,
and the $\ell_q$ norm is typically the $\ell_1$ or the $\ell_2$
norm. Typically, a descent step is performed using the gradient
of~\eqref{eq:one-sided}, followed by a support identification and
sparsity enforcement step. Often, care is taken in selecting the
descent step, especially considering the signal is on the unit
$\ell_2$ sphere~\cite{LasWenYin::2010::Trust-but-verify:}. Assuming a
certain noise level, a maximum likelihood formulation can also be used
to remove the norm constraint~\cite{bahmani2013robust}.

For example, the Binary IHT (BIHT), a variation of the popular Iterative Hard
Thresholding (IHT)~\cite{BluDav::2008::Iterative-hard}, uses the
one-sided $\ell_1$ norm in~\eqref{eq:one-sided} and follows its
subgradient $\tinv{2}A^T(q-\sign(Az))$. The algorithm is defined by
the iteration
\begin{equation}
  \label{eq:biht-def}
  z^{n+1} = \mathcal H_k\big(z^{n} + \tinv{2} A^T(y - \sign(A z^{n}))\big), \quad z^{0} = 0,
\end{equation}
where $\mathcal H_k(\cdot)$ is a hard threshold, keeping the largest $k$
coefficients of its input and setting the remaining ones to zero.

The BIHT does not have convergence or reconstruction guarantees to a
consistent output. Still, as shown in Fig.~\ref{fig:AngVsHamm}, it
works surprisingly well compared to other greedy approaches. Moreover,
variations exist to make it more robust to potential binary errors in
the knowledge of~$q$~\cite{jacques2013robust} or to extend it to
multi-bit scalar quantization \cite{QIHT}.

\begin{figure}[t]
   \centering
   \subfigure[\label{fig:AngVsHamm-01}m/n = 0.1]{\includegraphics[width=.305\textwidth]{fig5}}
   \subfigure[\label{fig:AngVsHamm-07}m/n = 0.7]{\includegraphics[width=.32\textwidth]{fig6}}
   \subfigure[\label{fig:AngVsHamm-15}m/n =1.5]{\includegraphics[width=.32\textwidth]{fig7}}
   \caption{Angular reconstruction error $\epsilon_S = d_S(x,\hat x)$
     vs.\ consistency error $\epsilon_H = d_H(\sign(A\hat x), q)$ for different
     greedy reconstructions (MSP, RSS and BITH).  BIHT returns a
     consistent solution in most trials.  When $A$ is a B$\epsilon$SE,
      \eqref{eq:bse-def} predicts that the angular
     error $\epsilon_S$ is bounded by the hamming error $\epsilon_{H}$
     (and conversely) in addition
     to an offset $\epsilon$. This phenomenon is confirmed by an
     experimental linear trend (in dashed) between the two errors that improves
     when $m/n$ increases \cite{jacques2013robust}. \label{fig:AngVsHamm}}
\end{figure}

The first iteration of BIHT is a simple truncated back-projection,
$\hat x_0 = \mathcal H_k(A^T q)$ whose distance to $x$ is known to decay
asymptotically as
$\sqrt{k/m}$ for a Gaussian matrix $A$
\cite{QIHT,bahmani2013robust}. Furthermore, $\hat x_0$ matches the
solution of the (feasible) problem
$$ \arg\max_z q^T Az\ {\rm s.t.}\ z\in\Sigma_k^*,
$$
where maximizing $q^T Az$ also promotes the 1-bit consistency of $z$
with $q$.

This optimization can be generalized to any convex sets $\mathcal K \subset
\mathbb R^n$ where $x$ can lie, such as the set $\mathcal K = K_{n,k}$ of
compressible signals with Gaussian width $w(K_{n,k}) \asymp k \log
n/k$~\cite{plan2012robust}. If $m \geq C \epsilon^{-2} w(\mathcal K)^2$
for some $C>0$, and a fixed $x$ is sensed using~\eqref{eq:1bit-CS}
with a Gaussian sensing matrix $A$, then the solution to
$$
\hat x\ =\ \arg\max_z\ q^T Az\ {\rm s.t.}\ z\in\mathcal K,
$$ satisfies $\|\hat x - x\|^2 = O(\epsilon)$ with high probability.
Interestingly, under certain conditions, this holds also for sensing
models other than \eqref{eq:1bit-CS}, where the sign operator is
replaced, for instance, by the logistic function
\cite{plan2012robust}.

What makes it difficult to provide reconstruction error estimates for algorithms
motivated by the problem
\eqref{eq:1bmin} is the non-convex constraint $\|x\|_2=1$, whose convex relaxation
allows for the zero solution and is hence meaningless.
To overcome this obstacle, it has been proposed in~\cite{plan2013one,plan2012robust} to
impose a norm constraint to prevent trivial solutions on the measurements rather than the signal.
This results in a different problem, which allows for a meaningful convex
relaxation. Namely, since $\ensuremath{q}=sign(Ax)$, it
follows that at the solution $\ensuremath{q}^T(Ax)=\|Ax\|_1$. Thus, by
constraining this norm, the following convex problem can be
formulated:
\begin{align}
  \hat{x}=\arg\min_x\|x\|_1,~\mathrm{s.t.}~\ensuremath{q}=\sign(Ax),~\ensuremath{q}^TAx=1
  \label{eq:1bit_convex}
\end{align}
As shown in \cite{plan2013one}, the problem in \eqref{eq:1bit_convex}
does allow for reconstruction guarantees: If $m \sim \epsilon^{-5} k
\log (n/k)$, the solution $\hat x$ recovered from quantized Gaussian
measurements of a sparse signal $x$ is such that $d_S(x,\hat x) \leq
\epsilon$ with high probability. This holds uniformly for all signals
$x \in \mathbb R^n$. Under flatness assumptions on the signal, recovery
guarantees can also be proved for arbitrary subgaussian measurements
\cite{ai2014one}.

\section{Discussion and Conclusion}
\label{sec:discussion}
Quantization is an essential component of any acquisition system, and,
therefore, an important part of compressive sensing theory and
practice. While significant work has been done in understanding the
interaction of quantization and compressive sensing, there are several
open problems and questions.

One of the most interesting open problems is the interaction of
quantization with noise. While the discussion and references in
Sec.~\ref{sec:extensions} provides some initial results and theoretical analysis, a
comprehensive understanding is still missing. An understanding of the
optimal bit allocation and the optimal quantizer design, uniform or
non-uniform scalar, or \ensuremath{\Sigma\Delta}\xspace, given the noise level, as well as the
robustness of the reconstruction to noise and quantization is still
elusive.

While \ensuremath{\Sigma\Delta}\xspace\ can be used to improve the rate efficiency of compressive
sensing, compared to scalar quantization, the performance is still not
comparable to the state-of-the-art in conventional \ensuremath{\Sigma\Delta}\xspace\ methods. For
example, conventional \ensuremath{\Sigma\Delta}\xspace\ quantization of band-limited functions can achieve error that decays exponentially
as the sampling rate increases, not currently possible with existing
compressive sensing \ensuremath{\Sigma\Delta}\xspace. Furthermore, the analysis in
Sec.~\ref{sec:SDCS} does not hold for 1-bit quantization, often
desirable in practical systems due to its simplicity. Such an
extension has significant practical importance.

Even with \ensuremath{\Sigma\Delta}\xspace\ approaches, the rate efficiency of compressive sensing
systems is not ideal. As evident from the fundamental bounds in
Sec.~\ref{sec:intro_fundamentals}, compressive sensing is not
rate-efficient compared to classical methods such as transform
coding. In others word, while compressive sensing is very promising
in building sensing systems because it can significantly reduce the
number of measurements and the sampling burden, it is not a good data
compression approach if the measurements have already been obtained
and the achievable bit-rate is important. That said, due to the intimate connection between frame quantization and quantization for compressed sensing, promising results in the finite frames context, e.g., \cite{iwen2013near} can inform future developments in compressed sensing.

The potential encoding simplicity of a compressive sensing
system is very appealing. Acquiring generalized linear measurements and quantizing
them can be less complex than typical transform-coding
approaches and much more attractive in low-power and
computationally-restricted sensing applications. The complexity is
shifted to the reconstruction, which, in many applications, can bear
significantly more computational complexity. Nevertheless, the rate inefficiency of
compressive sensing can be a barrier in such applications.

A number of promising approaches have been proposed to overcome this
barrier using modifications of the quantizer that produce
non-contiguous quantization
regions~\cite{bib:Pai06,bib:BoufSAMPTA2011,B_TIT_12,Kamilov12}. Initial
theoretical analysis and experimental results are promising. However,
our understanding is still limited. One of the drawbacks of such
approaches is that the reconstruction is no longer convex and,
therefore, not as simple to provide guarantees for.

Alternatively, recent work on adaptive quantization strategies has
shown that error decay exponential in the bit-rate can be achieved,
even using a 1-bit quantizer, at the cost of adaptivity in the
measurements and -- in contrast with the methods presented in this chapter -- significant computation at the encoder. Specifically,~\cite{baraniuk2014exponential} shows that
adaptively choosing the threshold of a 1-bit quantizer allows
the error to decay exponentially with the number of measurements. The cost is that the thresholds are updated by solving an $\ell_1$ minimization problem, or running an iterative hard thresholding scheme. It is thus interesting to quantify the tradeoff between computational complexity at the quantizer, and achievable reconstruction accuracy.

Another important aspect is that while the best recovery guarantees in
compressed sensing are obtained for Gaussian and subgaussian
measurement matrices, which are also mainly considered in this
article, applications usually require structured matrices, such as
subsampled Fourier matrices, e.g., as a model for subsampled MRI
measurements \cite{ldp07}, or subsampled convolution, e.g., as a model
for coded aperture imaging \cite{mawi08}. In both cases, when the
subsampling is randomized, near-optimal recovery guarantees are known
for unquantized compressed sensing \cite{RV08:sparse,krmera12}.
Combined with quantization, however, hardly anything is known for such
matrices. Such results would be of great importance to move the
approaches discussed in this survey closer to the application
scenarios.

Quantization is also important when considering randomized embeddings,
an area of research intimately related to compressive
sensing~\cite{baraniuk2008simple,krahmer2011new}. Embeddings are
transformations that preserve the geometry of the space they operate
on; reconstruction of the embedded signal is not necessarily the
goal. They have been proven quite useful, for example, in signal-based
retrieval applications, such as augmented reality, biometric
authentication and visual
search~\cite{LRB_MMSP12,BR_DCC13,SBV_SPIE13_Embeddings}.

These applications require storage or transmission of the embedded
signals, and, therefore, quantizer design is very important in
controlling the rate used by the embedding. Indeed, significant
analysis has been performed for embeddings followed by conventional
scalar quantization, some of it in the context of quantized
compressive
sensing~\cite{jacques2013robust,plan2012robust,plan2011dimension} or
in the study of quantized extensions to the Johnson Lindenstrauss Lemma~\cite{johnson1984extensions,LRB_MMSP12,SBV_SPIE13_Embeddings,jacques2013quantized}. Furthermore,
since reconstruction is not an objective anymore, non-contiguous
quantization is more suitable, leading to very interesting quantized
embedding designs and significant rate reduction~\cite{BR_DCC13}. In
this context, quantization can also provide to significant computation
savings in the retrieval, leading to Locality Sensitive Hashing (LSH)
and similar methods~\cite{Andoni08LSH}.
\section{Fundamentals of Quantization}
\label{sec:intro_fundamentals}
For the purposes of this section, a quantizer operates on signals $x$, viewed as vectors
in a bounded set $\ensuremath{V}\subset\mathbb{R}^n$. The goal of a quantizer
$\ensuremath{Q}(\cdot)$ is to represent those signals as accurately as possible
using a rate of \ensuremath{R}\ bits, \emph{i.e.},  using a quantization point
$\ensuremath{q}=\ensuremath{Q}(x)$ chosen from a set of $2^\ensuremath{R}$ possible ones often
referred to as codebook. Of course, when $\ensuremath{V}$ contains an infinite
number of signals, signals will be distorted through this
representation.

In this section, we first define common quantization performance
metrics and determine fundamental bounds on the performance of a
quantizer. Then, in preparation for the next sections, we examine
common approaches to quantization, namely scalar and Sigma-Delta
quantization, which are very useful in compressive sensing
applications.

\subsection{Quantization Performance Bounds}
\label{sec:intro_vector}
To measure the accuracy of the quantizer we consider the distortion, i.e.,  the $\ell_2$ distance of a
quantization point from its original signal $\|x-\ensuremath{Q}(x)\|_2$. The
overall performance of the quantizer is typically evaluated either using the
average distortion over all the signals---often computed using a
probability measure on the signal space \ensuremath{V}---or using the worst case
distortion over all signals in \ensuremath{V}. In this chapter, in the spirit of
most of the compressed sensing literature, we quantify the performance
of the quantizer using the worst case distortion on any signal, \emph{i.e.},
\begin{align}
\ensuremath{\varepsilon}=\sup_{x\in \ensuremath{V}}\|x-\ensuremath{Q}(x)\|_2.
\end{align}
This choice enables very strong guarantees, irrespective of the
accuracy of any probabilistic assumption on the signal space.

A lower bound on the distortion of any quantizer can be derived by
constructing a covering of the set \ensuremath{V}. A covering of radius $r$ is a
set of points \ensuremath{q}\ such that each element in \ensuremath{V}\ has distance at most
$r$ from its closest point in the covering. If we can construct a
covering using $P$ points, then we can also define a quantizer that
uses $\ensuremath{R}=\lceil\log_2 P\rceil$ bits and has worst case distortion $\ensuremath{\varepsilon}=r$ as
each signal is quantized to the closest point in the covering.

To determine a lower bound for the number of points in such a
covering, we consider balls of radius $r$ centered {at} $\ensuremath{q}$, defined as
\begin{align}
  \ball{r}{\ensuremath{q}}=\left\{\left.x\in\mathbb{R}^n\right|\|\ensuremath{q}-x\|_2\le r\right\}.
\end{align}
Since each signal in \ensuremath{V}\ is at most $r$ away from some point in the
covering, if we place a ball of radius $r$ at the center of each point
of the covering, then the union of those balls covers \ensuremath{V}. Thus, the total volume of the balls should be at
least as large as the volume of the set, denoted
$\mathrm{vol}(\ensuremath{V})$. Since the volume of a ball of radius $r$ in $n$
dimensions is
$\mathrm{vol}(\ball{r}{\ensuremath{q}})=r^n\pi^{n/2}/\Gamma(1+n/2)$, where
$\Gamma(\cdot)$ is the Gamma function, the best possible error given
the rate \ensuremath{R}\ can be derived using
\begin{align}
  \mathrm{vol}(\ensuremath{V})\le
  \tfrac{\pi^{n/2}}{\Gamma\left(1+\frac{n}{2}\right)}\,2^\ensuremath{R}\,r^n
  \Rightarrow r \gtrsim 2^{-\frac{\ensuremath{R}}{n}}.
\end{align}
In other words, the worst-case error associated with an optimal quantizer can, at best,
decay exponentially as the bit rate increases. Moreover, the decay rate
depending on the ambient dimension of the signal. In short,
\begin{align} \ensuremath{\varepsilon}&\gtrsim 2^{-\frac{\ensuremath{R}}{n}}.
\end{align}
The smallest achievable worst case distortion for a set is also known
as the $(\ensuremath{R}+1)$-dyadic entropy number of the set, whereas the number
of bits necessary to achieve a covering with worst-case distortion
equal to $\ensuremath{\varepsilon}$ is known as the Kolmogorov $\ensuremath{\varepsilon}$-entropy or metric
entropy of the set.

For the models commonly assumed in compressive sensing, these
quantities are not straightforward to calculate and depend on the
sparsity model assumed. For example, compressible signals are commonly modeled as
being drawn from a
unit $\ell_p$ ball, where $0<p<1$ (cf.~Chapter 1 for a discussion on compressibility). In this case, the worst case distortion is bounded by
\begin{align}
  \ensuremath{\varepsilon}\ \gtrsim\ \left\{
  \begin{array}{cl}
    1&\mbox{if}~1\le \ensuremath{R}\le \log_2 n\\    \,\left(\tinv{\ensuremath{R}}\log_2(\tfrac{n}{\ensuremath{R}}+1)\right)^{\inv{p}-\inv{2}}&\mbox{if}~\log_2 n\le \ensuremath{R}\le n\\[1mm]
    \,2^{-\frac{\ensuremath{R}}{n}}n^{\inv{2}-\inv{p}}&\mbox{if}~\ensuremath{R}\ge n,
  \end{array}
  \right.
\end{align}
where the constant implicit in our nation is independent of \ensuremath{R}\ and
$n$~\cite{schutt1984entropy,edmunds1996function,kuhn2001lower,candes2006encoding}.

In the case of exactly $k$-sparse signals, the volume of the union of
subspaces they occupy has measure zero in the $n$-dimensional ambient
space. However, by considering the $\binom{n}{k}$ $k$-dimensional
subspaces and coverings of their unit balls, a lower bound on the error can be
derived~\cite{BB_DCC07}, namely
\begin{align}
  \ensuremath{\varepsilon}\gtrsim \frac{2^{-\frac{\ensuremath{R}}{k}}n}{k}.
  \label{eq:sparse_lower}
\end{align}
Note that this lower bound can be achieved in principle using standard
transform coding (TC), \emph{i.e.},  by first representing the signal using its
sparsity basis, using $\log_2\binom{n}{k} \lesssim k\log_2(n/k)$ bits to
represent the support of the non-zero coefficients and using the
remaining bits to represent the signal in the $k$-dimensional subspace
at its Kolmogorov entropy
\begin{align}
  \ensuremath{\varepsilon}_{\mathrm{TC}} \lesssim 2^{-\frac{\ensuremath{R}-k\log_2(n/k)}{k}} =\frac{2^{-\frac{\ensuremath{R}}{k}}n}{k}.
\end{align}
Unfortunately, compressive sensing systems do not have direct access to
the sparse vectors. They can only access the measurements, $y=Ax$,
which must be quantized upon acquisition---in practice using analog
circuitry. Thus, transform coding is not possible. Instead, we must
devise simple quantization algorithms that act directly on the
measurements in such a way that permits accurate reconstruction.
\subsection{Scalar Quantization}
\label{sec:intro_scalar}
The simplest approach to quantization is known as {\em scalar
  quantization} and often referred to as {\em pulse code modulation}
(PCM), or {\em memoryless scalar quantization} (MSQ). Scalar
quantization directly quantizes each measurement of the signal,
without taking other measurements into account. In other words a
1-dimensional, \emph{i.e.},  scalar, quantizer is applied separately to each
measurement of the signal.

\subsubsection{Measurement and Scalar Quantization}
\label{sec:meas-scal-quant}

A scalar quantizer can be defined using a set of levels, $\ensuremath{\mathcal{Q}}
=\{\ensuremath{l}_i \in \mathbb R: \ensuremath{l}_j < \ensuremath{l}_{j+1}\}$, comprising the
quantization codebook, and a set of thresholds $\ensuremath{\mathcal{T}} = \{\ensuremath{t}_i \in
\overline{\mathbb R}: \ensuremath{t}_j < \ensuremath{t}_{j+1}\}$, implicitly defining the
quantization intervals $\ensuremath{\mathcal{C}}_j = [\ensuremath{t}_j,\ensuremath{t}_{j+1})$. Assuming no
  measurement noise, the quantizer is applied element-wise to the
  measurement coefficients, $y=Ax$, to produce the quantized
  measurements $\ensuremath{q}=Q(y),~\ensuremath{q}_i=Q(y_i)$. Using a rate of \ensuremath{B}\ bits per
  coefficient, \emph{i.e.},  $\ensuremath{R}=m\ensuremath{B}$ total bits, the quantizer represents
  $\ensuremath{L}=2^\ensuremath{B}$ total levels per coefficient. A scalar value $y_i$
  quantizes to the quantization level corresponding to the
  quantization interval in which the coefficient lies.
\begin{equation}
  \label{eq:scalar-quant-def}
  \ensuremath{Q}(y_i) = \ensuremath{l}_j\quad\Leftrightarrow\quad y_i \in \ensuremath{\mathcal{C}}_j.
\end{equation}

A scalar quantizer is designed by specifying the quantization levels
and the corresponding thresholds. Given a source signal with
measurements modeled as a continuous random variable $X$, a
\emph{(distortion) optimal} scalar quantizer minimizes the error
\begin{equation}
  \label{eq:distortion-optimal}
  \mathbb E|X - \ensuremath{Q}(X)|^2.
\end{equation}
Such an optimal quantizer necessarily satisfies the Lloyd-Max
conditions~\cite{Lloyd1982,Max1960}
\begin{equation}
  \label{eq:LM-condition}
  \ensuremath{l}_j = \mathbb E\left\{X|X \in \ensuremath{\mathcal{C}}_j\right\},\qquad  \ensuremath{t}_j =
  \tinv{2}(\ensuremath{l}_j+\ensuremath{l}_{j+1}),
\end{equation}
which define a fixed point equation for levels and thresholds and the
corresponding fixed-point iteration---known as the Lloyd-Max
algorithm---to compute them.

\begin{figure}[t]
  \centerline{\includegraphics[width=.7\linewidth]{QuantGrid}}
  \caption{A finite uniform scalar quantizer and the uniform grid it
    generates in 2 and 3 dimensions}
  \label{fig:quant_grid}
\end{figure}

Alternatively, a simpler design approach is the uniform scalar
quantizer, which often performs almost as well as an optimal scalar
quantizer design. It is significantly less complex and can be shown to
approach optimality as the bit-rate increases~\cite{gray1998quantization}. The
thresholds of a uniform scalar quantizer are defined to be
equi-spaced, \emph{i.e.},  $\ensuremath{t}_{j+1} - \ensuremath{t}_j = \ensuremath{\Delta}$, where $\ensuremath{\Delta}$ is
referred to as the quantization bin width or \emph{resolution}. The
levels are typically set to the mid-point $\ensuremath{l}_j = \tinv{2}(\ensuremath{t}_j
+ \ensuremath{t}_{j+1})$ of the quantization bin $\ensuremath{\mathcal{C}}_j$. Thus, the
quantization error introduced to each coefficient is bounded by
$\ensuremath{\Delta}/2$. A uniform quantizer defines a uniform grid in the
$m$-dimensional measurement space, as shown in
Fig.~\ref{fig:quant_grid}.

In practical systems, the scalar quantizer has finite range, \emph{i.e.},  it
saturates if the signal exceeds a saturation level \ensuremath{S}. In particular,
a uniform finite-range scalar quantizer using $B$ bits per coefficient
has quantization interval $\ensuremath{\Delta}=\ensuremath{S} 2^{-B+1}$. If a coefficient exceeds
$S$, the quantizer maps the coefficient to the largest quantization
level, \emph{i.e.},  it saturates.  Depending on the magnitude of the
coefficient, this may introduce significant error. However, it is
often convenient in theoretical analysis to assume an infinite
quantizer that does not saturate. This assumption is often justified,
as \ensuremath{S}\ in practice is set large enough to avoid saturation given a signal
class. As described in Sec.~\ref{sec:unif-scal-quant-with-saturation},
this is often suboptimal in compressive sensing applications.

Compared to classical systems, optimal scalar quantizer designs for
compressive sensing measurements require extra care. An optimal design with respect to the measurement error is
not necessarily optimal for the signal, due to the non-linear reconstruction inherent in compressed sensing. While specific designs have been derived for very
specific reconstruction and probabilistic signal models,
\emph{e.g.}, ~\cite{sun2009optimal,kamilov2011optimal}, a general optimal design
remains an open problem. Thus the literature has focused mostly, but
not exclusively, on uniform scalar quantizers.

\begin{figure}[t]
  \centerline{\includegraphics[width=.8\linewidth]{CellIntersection}}
  \caption{A $k$-dimensional space measured using $m$ measurements
    spans a $k$-dimensional subspace of $\mathbb{R}^m$ and intersects
    only a few of the $L^m$ available quantization cells.}
  \label{fig:cell_intersect}
\end{figure}

\subsubsection{Scalar Quantization and Oversampling}
\label{sec:scal-quant-overs}

When a signal is oversampled, a scalar quantizer makes suboptimal use
of the bit-rate. The $k$-dimensional signal space mapped through the
measurement operator to an $m$-dimensional measurement space, where
$m>k$, spans, at most, a $k$-dimensional subspace of $\mathbb{R}^m$,
as shown in Fig.~\ref{fig:cell_intersect}. As evident from the figure,
this subspace intersects only a few of the available quantization
cells and, therefore, does not use the available bits effectively. For
an $\ensuremath{L}$-level quantizer, the number of quantization cells intersected
$I_{k,m,L}$ is bounded
by~\cite{bib:Thao96,goyal_1998_lowerbound_qc,bib:Boufounos06thesis}
\begin{align}
  I_{k,m,L}\lesssim\left(\frac{Lm}{k}\right)^k
\end{align}
Using a simple covering argument as in Sec.~\ref{sec:intro_vector}, it
is thus possible to derive a lower bound on the error performance as a
function of the number of measurements $m$
\begin{align}
  \ensuremath{\varepsilon}\gtrsim\left(\frac{2^{-\ensuremath{B}}k}{m}\right) \label{eq:scalar_bound1}
\end{align}
The bounds hold for any scalar quantizer design, not just uniform
ones.

Linear reconstruction, \emph{i.e.},  reconstruction using a linear operator
acting on the scalar quantized measurements, does not achieve the bound
~\eqref{eq:scalar_bound1}~\cite{bib:Thao94,goyal_1998_lowerbound_qc}. The
quantization error using linear reconstruction can only decay as fast
as
\begin{equation}
  \ensuremath{\varepsilon}\gtrsim\frac{2^{-\ensuremath{B}}k}{\sqrt{m}}.
  \label{eq:scalar_bound2}
\end{equation}

Instead, {\em consistent}
reconstruction achieves the optimal bound in a number of
cases. Consistent reconstruction treats the quantization regions as reconstruction
constraints and ensures that the reconstructed signal $\hat{x}$
quantizes to the same quantization points when measured using the same
system. Thus in the oversampled setting where $A$ is an $m\times k$
matrix with $m>k$, and where $\ensuremath{q}=\ensuremath{Q}(Ax)$ one solves the problem:
\begin{align}
\label{eq:consistent-algo}
  \mathrm{find~any}~~\hat{x}&~~\mathrm{s.t.}~~\ensuremath{q}=\ensuremath{Q}(A\hat{x}).
\end{align}
If the measurement operator $A$ is a tight frame formed by an
  oversampled Discrete Fourier Transform (DFT), the root mean square
  error (RMSE) of such a reconstruction (with respect to a random
  signal model) decays as $O(1/m)$
  \cite{bib:Thao94,goyal_1998_lowerbound_qc}, \emph{i.e.},  as
  \eqref{eq:scalar_bound1}. In the case of random
  frames with frame vectors drawn independently from a Gaussian distribution \cite{jacques2014error} or from a suitable distribution on the
  $(m-1)$-sphere \cite{powell_consistent}, the reconstruction method
  in \eqref{eq:consistent-algo}
  also displays RMSE and worst
  case reconstruction error decreasing as $O(1/m)$ and $O((\log m)/m)$, respectively.

The constraints imposed by consistent reconstruction are convex and
can be imposed on any convex optimization algorithm. This makes them
particularly suitable for a number of reconstruction algorithms
already used in compressive sensing systems, as we explore in
Sec.~\ref{sec:scalarCS}.

The bounds \eqref{eq:scalar_bound1} and
\eqref{eq:scalar_bound2}---which can be achieved with proper design of
the measurement process and the reconstruction algorithm---demonstrate
that the most efficient use of the rate $\ensuremath{R}=m\ensuremath{B}$ is in refining each
measurement using more bits per measurement, \ensuremath{B}, rather than in
increasing the number of measurements, $m$. They suggest that in terms
of error performance, by doubling the oversampling it is possible to
save 0.5 bits per coefficient if linear reconstruction is used and 1
bit per coefficient if consistent reconstruction is used. This means
that a doubling of the rate by doubling the oversampling factor, is
equivalent to a linear increase in the rate by $m/2$ or $m$ through an
increase in \ensuremath{B}, for linear and consistent reconstruction,
respectively. So in principle, if rate-efficiency is the objective, the
acquisition system should only use a sufficient number of measurements
to reconstruct the signal and no more. All the rate should be devoted
to refining the quantizer. However, these bounds ignore the practical advantages in oversampling
a signal, such as robustness to erasures, robustness to measurement
noise and implementation complexity of high-rate scalar quantizers. Thus in
practice, oversampling is often preferred, despite the
rate-inefficiency. Techniques such as Sigma-Delta quantization, which we discuss in Sec.~\ref{sec:intro_SD}, have been developed to
improve some of the trade-offs and are often used in conjunction with oversampling.

\subsubsection{Performance Bounds on Sparse Signals}
\label{sec:perf-bounds-sparse}

Scalar quantization in compressive sensing exhibits similar bounds as
scalar quantization of oversampled signals. Signals that are
$k$-sparse in $\mathbb R^n$ belong
to a union of $k$-dimensional subspaces. When measured using $m$
linear measurements, they occupy a union of $k$-dimensional subspaces
of $\mathbb{R}^m$, $\binom{n}{k}$ of them. Using the same counting
argument as above, it is evident that the number of quantization cells
intersected, out of the $L^m$ possible ones, is at most
\begin{align}
  \binom{n}{k}I_{k,m,L}\gtrsim\left(\frac{Lmn}{k^2}\right)^k
\end{align}
The resulting error bound is
\begin{align}
  \ensuremath{\varepsilon}&\gtrsim\frac{2^{-\ensuremath{B}}k}{m}\\
  &\gtrsim\frac{2^{-\frac{\ensuremath{R}}{m}}k}{m},
\end{align}
which decays slower than \eqref{eq:sparse_lower} as the rate increases
keeping the number of measurements $m$ constant. Furthermore, as the
rate increases with the number measurements $m$, keeping $\ensuremath{B}$, the
number of bits per measurement constant, the behavior is similar to
quantization of oversampled frames: the error can only decay linearly
with $m$.

These bounds are not surprising, considering the similarities of
oversampling and compressive sensing of sparse signals. It should,
therefore, be expected that more sophisticated techniques, such as
Sigma-Delta ($\Sigma\Delta$) quantization should improve performance,
as they do in oversampled frames. However, their application is not as
straightforward. The next section provides an overview of
$\Sigma\Delta$ quantization and Sec.~\ref{sec:SDCS} discusses in
detail how it can be applied to compressive sensing.
\subsection{Sigma-Delta Quantization}
\label{sec:intro_SD}
An alternative approach to the scalar quantization techniques detailed
in the previous section is feedback quantization. The underlying idea
is that the fundamental limits for the reconstruction accuracy
discussed above can be overcome if each quantization step takes into
account errors made in previous steps. The most common feedback
quantization scheme is $\ensuremath{\Sigma\Delta}\xspace$ quantization, originally introduced for
bandlimited signals in \cite{IYM62} (cf. \cite{IY63}). A simple $\ensuremath{\Sigma\Delta}\xspace$ scheme, illustrated in Figure \ref{fig:SD_blk}, shows this feedback structure.
\begin{figure}
\begin{center}
\includegraphics{sd_block_diagram.pdf}\label{fig:SD_blk}
\caption{A block diagram of a simple $1$st order $\ensuremath{\Sigma\Delta}\xspace$ scheme: The input $y_i$ is added to the state variable $u_{i-1}$ (initialized as $u_0 =0 $) and the sum is scalar quantized. Subsequently, the state variable is updated as the difference between the scalar quantizer's input and its output. More complex designs, featuring higher order $\ensuremath{\Sigma\Delta}\xspace$ quantization with more feedback loops are possible. We discuss such designs in more detail in  Section~\ref{sec:SDCS}.}
\end{center}
\end{figure}

A motivation
in \ensuremath{\Sigma\Delta}\xspace\ quantization is that, in some applications, reducing circuit
complexity is desirable, even at the expense of a higher sampling
rate. Indeed, \ensuremath{\Sigma\Delta}\xspace\ designs drastically reduce the required bit depth
per sample while allowing for accurate signal reconstruction using
simple circuits. In fact, since its introduction, $\ensuremath{\Sigma\Delta}\xspace$ quantization has seen widespread use (see,
\emph{e.g.},  \cite{NST96} and the references therein) in applications ranging from audio coding to wireless communication.

Nevertheless, a mathematical analysis of $\ensuremath{\Sigma\Delta}\xspace$ quantization in its full generality has been challenging. A preliminary
analysis of simple $\ensuremath{\Sigma\Delta}\xspace$ schemes for restricted input
classes (including constant input and sinusoidal input) was presented in \cite{G87} and follow-up works. However, most
of these results were limited to linear, or at best low-order polynomial error
decay in the oversampling rate.  This type of error decay is sub-optimal (albeit better than scalar quantization), and  rather far from the optimal exponential error decay.
Specifically, a major difficulty that prevented a
more comprehensive treatment was understanding the instabilities caused by
the positive feedback inherent to the $\ensuremath{\Sigma\Delta}\xspace$ circuit designs. For example, depending on the design of the $\ensuremath{\Sigma\Delta}\xspace$ scheme, the state variables could grow without bound. A crucial idea to
prevent such phenomena for arbitrary \emph{band-limited} inputs was
developed in \cite{DD03}; their analysis led, for the first-time, to
super-polynomial bounds for the error decay. To date, the best known
error bounds decay exponentially in the oversampling rate \cite{G03,
  DGK11}. While this is near-optimal (optimal up to constants in the exponent), it has been shown that with a fixed
bit budget per sample, the achievable rate-distortion relationship is
strictly worse than for scalar quantization of Nyquist rate samples
\cite{KW12}. That said, increasing the bit budget per sample entails more expensive and complex circuitry, which grows increasingly costly with every added bit (in fact, the best current quantizers provide a resolution of about 20 bits per sample). Thus, for quantizing bandlimited functions, if one wishes to improve the performance or reduce the cost, one must revert to oversampling-based methods such as $\ensuremath{\Sigma\Delta}\xspace$ quantization.

The accuracy gain of $\ensuremath{\Sigma\Delta}\xspace$ quantization is most prominent when a
significant oversampling rate and, therefore, a high redundancy of
samples is inherent or desired. Such redundant representations can
also be encountered in a finite-dimensional discrete context. Namely,
this corresponds to a finite frame expansion in the sense of
(1.32). This observation served as a motivation to devise $\ensuremath{\Sigma\Delta}\xspace$
schemes for finite-frame expansions, and the first such construction was
provided in \cite{BPY06}.  In contrast to oversampled representations
of bandlimited signals, which directly correspond to a temporal
ordering, finite frames generally do not have an inherent order, nor
are the frame vectors necessarily close enough to each other to allow
for partial error compensation.  Due to this difficulty, the first
works on $\ensuremath{\Sigma\Delta}\xspace$ quantization for finite frame expansions focus on
frames with special smoothness properties. Namely, they assume that
the frame $\Phi=\{\phi_j\}_{j=1}^N$ has a well controlled {\em frame
  variation} \[v_\Phi:=\sum_{j=1}^{N-1} \|\phi_{j+1}-\phi_j\|_2.\] The
constructions in \cite{BPY06} coupled with (linear) reconstruction via
the canonical dual frame (that is, the Moore-Penrose pseudo-inverse of the matrix that generates the redundant representation) was shown to yield an error decay on the
order of $v_{\Phi} N^{-1}$, \emph{i.e.},  linear error decay whenever the frame
variation is bounded by a constant. By using more sophisticated $\ensuremath{\Sigma\Delta}\xspace$
schemes these results were later improved to higher order polynomial
error decay \cite{BPY06-2, BP07, BPA07} in the number of measurements,
thereby beating the bound \eqref{eq:scalar_bound1} associated with
scalar quantization. Again, these constructions require certain
smoothness conditions on the frame and employ the canonical dual frame
for recovery. In a slightly different approach, the design of the feedback
and the ordering of the frame vectors has been considered as part of the quantizer design~\cite{BO_JASP06,bib:Boufounos06thesis}.

A new take on the frame quantization problem was initiated in
\cite{LPY10, BLPY10} where the authors realized that reconstruction
accuracy can be substantially improved by employing an appropriate
alternative dual frame (\emph{i.e.}, , a different left-inverse) for recovery. At the core of this approach is
still a smoothness argument, but this time for the dual frame. Given a
frame, an appropriate dual frame, the so-called Sobolev dual, can be
obtained by solving a least-squares problem over the space of all
duals \cite{ BLPY10}. Again, this yields polynomial error decay,
albeit now in more general settings. Moreover, by optimizing over such
constructions, root-exponential error decay can be achieved
\cite{KSW12}.

While the definition of the Sobolev dual does not require any
smoothness of the frame, the concrete examples discussed in the
aforementioned works still exclusively focused on smooth
frames. Similar results on recovery guarantees for frames without
smoothness properties were first obtained for frames consisting of
independent standard Gaussian vectors \cite{GLPSY13} and subsequently
generalized to vectors with independent subgaussian entries
\cite{KSY13}.

The underlying constructions also form the basis for the $\ensuremath{\Sigma\Delta}\xspace$
quantization schemes for compressed sensing measurements. Details on
such schemes are given in Sec.~\ref{sec:SDCS}. The insight behind
the schemes is that the number of measurements taken in compressed
sensing is typically larger than the support size by at least a
logarithmic factor in the dimension, and there is an interest in
choosing it even larger than that, as this induces additional
stability and robustness. Thus, once the support of the signal has been
identified and only the associated signal coefficients need to be
determined, one is dealing with a redundant representation. The goal
is now to employ frame quantization schemes to exploit this
redundancy.

For typical compressed sensing matrices, any $k$ columns indeed form a
frame; this follows for example from the restricted isometry property.
However, as the support of the signal is not known when quantizing the
measurements, it is crucial that $\ensuremath{\Sigma\Delta}\xspace$ quantization is universal. That
is, it must not require knowledge regarding which of a given
collection of frames (namely, those forming the rows of an $m \times
k$ submatrix of $A$) has been used for encoding. The reconstruction
from the resulting digital encodings then typically proceeds in two
steps. First the support is identified using standard compressed
sensing recovery techniques, just treating the quantization error as
noise. In a second step, only the restriction of the measurement
matrix to the identified support columns is considered. For the frame
consisting of the rows of this matrix, one then applies frame
quantization reconstruction techniques. Recovery guarantees for such
an approach have been proven for Gaussian measurements \cite{GLPSY13}
and measurements with independent subgaussian entries \cite{KSY13}. It
is of great importance that the dual frame used for recovery is chosen
properly (\emph{e.g.},  the Sobolev dual), as it follows from the RIP that the
frames never have a small frame variation. Here again the recovery
error bounds decay polynomially in the number of measurements and beat
the analogous bounds for scalar quantization.

Preliminary steps towards a unified approach to support and signal
recovery have been considered in \cite{C13}. The reconstruction
techniques studied in this work, however, intrinsically rely on
certain non-convex optimization problems, for which no efficient
solution methods are known. Thus the quest remains open for an
integrated approach to reconstruction from $\ensuremath{\Sigma\Delta}\xspace$-quantized compressed
sensing measurements that combines numerical tractability and
guaranteed recovery.
\section{Introduction}
\label{sec:intro}

In order to store and manipulate signals using modern devices, it is
necessary to digitize them. This involves two steps: sampling (or
measurement) and quantization. The compressed sensing theory and
practice described in the remainder of this book provides a novel
understanding of the measurement process, enabling new technology and
approaches to reduce the sampling burden. This chapter explores the
very interesting interaction of compressed sensing with quantization.

Sampling maps a signal to a set of coefficients, typically using
linear measurements. This map can often be designed to be lossless,
\emph{i.e.},  to perfectly represent all signals in a certain class, as well as
robust to noise and signal modeling errors. The Nyquist
theorem, as well as more recent compressive sampling theorems are
examples of such sampling
approaches~\cite{candes2006ssr,blu2008sparse,mishali2011sub}.

The guarantees in sampling theorems are typically stated in terms of
the critical measurement rate, \emph{i.e.},  the number of measurements necessary to
perfectly represent signals in a given class. Oversampling, compared
to that minimum rate, typically provides robustness to errors in the
representation, noise in the acquisition, and mismatches in signal
models. The latter is especially important in compressive sensing
systems as they provide perfect reconstruction guarantees for exactly
sparse signals; in practice, the acquired signal is almost never
exactly sparse.

Quantization, on the other hand, is the process of mapping the
representation coefficients---which potentially belong to an
uncountably infinite set---to elements in a finite set, and
representing them using a finite number of bits. Due to the
many-to-one nature of such a map, the quantized representation is in
general lossy, \emph{i.e.},  distorts the representation and, therefore, the
signal. This distortion occurs even if the measurement process is lossless.

The interaction of quantization with sampling introduces interesting
trade-offs in the acquisition process. A system designed to sample
signals at (or slightly above) the critical rate may be less
robust to errors introduced by quantization. Consequently, it requires a
sophisticated quantizer design that ensures very small quantization errors.
On the other hand, a simpler quantizer architecture (e.g., with fewer bits per measurement) could introduce significant error to the representation
and require some oversampling to compensate. Practical systems
designs navigate this trade-off, for example, according to the complexity of the
corresponding hardware.

Compressive acquisition systems amplify the importance of the
trade-off between quantizer complexity and oversampling. The sampling rate is significantly reduced in such systems, at the expense of increased sensitivity to noise and signal model
mismatch. Thus, loss of information due to quantization
can be detrimental, especially when not properly handled. One may revert to oversampling here as well, however
the incoherent and often randomized nature of compressive
measurements poses challenges.  Thus, powerful oversampling based quantization approaches, such as Sigma-Delta quantization can be applied, but only after careful consideration.

Nevertheless, the sparse signal models and the computational
methods developed for compressed sensing can alleviate a number of
performance bottlenecks due to quantization in conventional
systems. Using computational approaches originating in frame theory
and oversampling, it is possible to significantly reduce the
distortion due to quantization, to significantly improve the
performance due to saturation, and to enable reconstruction from
measurements quantized as coarsely as 1~bit. The theory and practice
for such methods are described in Sec.~\ref{sec:scalarCS}.

It might seem counter-intuitive that compressed sensing
attempts to remove sampling redundancy, yet successful reconstruction
approaches employ tools developed for oversampled representations. In
fact there is a strong connection between compressed sensing and
oversampling, which we explore in various points in this
chapter. Furthermore, with sufficient care, this connection can be
exposed and exploited to implement Sigma-Delta quantization in
CS-based acquisition systems, and significantly improve performance
over scalar quantization. The details are discussed in
Sec.~\ref{sec:SDCS}.

The next section presents general principles of quantization,
including a brief background on vector, scalar, and Sigma-Delta
quantization for general acquisition systems. It is not an exhaustive
survey of the topic. For this we refer the reader
to~\cite{gray1998quantization,DD03,NST96}. Instead, it serves to
establish notation and as quick reference for the subsequent
discussion. Sec.~\ref{sec:scalarCS} and Sec.~\ref{sec:SDCS} examine the
interaction of compressive sensing and quantization in significant
detail. Sec.~\ref{sec:discussion} concludes with some discussion of
the literature, promising directions and open problems.

\input{conventions}
\input{intro_fundamentals}
\input{intro_scalar}
\input{intro_SD}

\section*{Acknowledgement}
Petros T. Boufounos is exclusively supported by
Mitsubishi Electric Research Laboratories. Laurent Jacques is a Research Associate funded by the Belgian
F.R.S.-FNRS. Felix Krahmer acknowledges support by the German Science Foundation (DFG) in the context of the Emmy-Noether Junior Research Group KR 4512/1-1 ``RaSenQuaSI''.
Rayan Saab is an assistant professor of mathematics with the University of California, San Diego.

\section{Reviews}

We would like to thank the two reviewers for their thorough evaluation
of our chapter.  In the core of this document, we respond to each of
their comments and mention related changes in the manuscript.

\section{Answers to the 1$^{\rm st}$ reviewer}

{\em This chapter exposes in a very comprehensive and balanced manner the principles of quantization in compressive sensing. It is very well written and it is also well integrated with Chapter 1 on ``A Survey of Compressed Sensing''.
I have only minor suggestions that the author may or may not choose to
address, as they see fit.}

\paragraph*{\em Suggestions:}

\begin{itemize}
\item\em There are some references that are used both in Chapter 1 and
  this Chapter. Examples include references 23, 24 in this chapter
  (references 15, 11 in chapter 1), reference 10 in this chapter
  (reference 8 in chapter 1), reference 4 in this chapter (reference 4
  in chapter 1), and so on.
\end{itemize}

\resp{Due to the nature of the book there will be redundancy in the
references. Unless the editors decide to collect all references at the
end of the book, this redundancy is unfortunately necessary. We will
be happy to use a common reference list if requested. However,
pointing to reference in Ch. 1 as it currently stands will be very
inconvenient to the reader (having to go back and forth) and cause a
problem with citation numbering (not clear how to cite references in
another chapter, unless there is a sequential numbering scheme from
chapter to chapter).}

\begin{itemize}
\item\em It is not always clear the source of some of the theorems. I
  would suggest the authors to adopt the form Theorem [reference] – as
  they do e.g. in Theorem 4 – in order to make this clear.
\end{itemize}

\resp{References were added to all the theorems to make presentation
consistent.}

\begin{itemize}
\item\em P. 4 – ``A lower bound on the accuracy of any quantizer...''
  would (in view of eq. (1)) likely read better as ``An upper bound on
  the distortion of any quantizer...''
\end{itemize}

\resp{In fact it should have been ``A lower bound on the distortion of any
quantizer...'' Thank you for noticing it.}

\begin{itemize}
\item\em P. 4 – ``...a quantizer that uses $R = log_2 P$ bits...'' – Is
  the ceiling function needed?
\end{itemize}

\resp{Indeed. Corrected.}

\begin{itemize}
\item\em P. 4 – ``... we consider balls of radius $r$...'' $\to$ ``...we consider balls of radius $r$ centered on $q$...''
\end{itemize}

\resp{Corrected}

\begin{itemize}
\item\em P. 4 – ``... bitrate...'' $\to$ ``...bit rate...''
\end{itemize}

\resp{Fixed}

\begin{itemize}
\item\em P. 5 – Define the base of the logarithms in eq. (5) or else specify that you always take the base of the logarithms to be 2.
\end{itemize}

\resp{Since not all of our logs are base 2, we corrected it in
  (5). Except for very few equations, such as (5), the base of the log
  does not matter since it can be subsumed in the constants. In any case, we have clarified this in Section 1.}

\begin{itemize}
\item\em P. 5 – Eqs. (6) and (7) – I am not sure that the asymptotic notation has been defined, on Chapter 1, this Chapter or under ``Proposed Notation''. I would suggest the authors to do it.
\end{itemize}

 \resp{We have added a notation subsection, which includes the
   definitions for asymptotic notation.}

\begin{itemize}
\item\em P. 7 – I would suggest a reference for eqs. (10) and (11)
\end{itemize}

\resp{The references were at the end of the sentence following those
equations. We moved them earlier to make the association with the
equations clear.}

\begin{itemize}
\item\em P. 12 – ``Importantly, and in contrast to conventional definitions of RIP matrices...'' I would suggest the authors to elaborate further about this remark.
\end{itemize}

 \resp{We have revised the text to elaborate on that
  point. The main issue is that with the conventional scaling of RIP
  matrices, the norm of each measurement vector, i.e., of each row of
  the matrix, decreases as the measurements increase. This means that
  the dynamic range of the measurements decreases and the quantizer
  should be scaled appropriately. Keeping track of this is
  cumbersome. Drawing the matrices from a distribution independent of
  the number of measurements is the simplest way to handle the
  problem, as long as existing results are scaled appropriately when
  used.}

\begin{itemize}
\item\em P. 12 – Please define the set in the very first equation on this page
\end{itemize}

\resp{Done.}

\begin{itemize}
\item\em P. 12 – ``...i.e., $\Theta = Id$...” $\to$ “...i.e., $\Psi = I_n$...'' ?
\end{itemize}

\resp{Fixed.}

\begin{itemize}
\item\em P. 13 – ``...for $\zeta = O(1)$.'' $\to$ ``...for $\zeta$ a constant.''
\end{itemize}

\resp{Fixed.}

\begin{itemize}
\item\em P. 13 – second equation from the bottom – the asymptotic variables underlying the big $O$ notation are a bit unclear. I would suggest the authors to clarify further.
\end{itemize}

\resp{This notation has been replaced by ``$\lesssim$'' symbol defined
  in a ``notation'' section in the Introduction, \emph{i.e.},
  $f \lesssim g$ if there exists a $C>0$ sucht that $f \leq C
  g$.}

\begin{itemize}
\item\em P. 13 – second equation from the bottom – I am not sure whether $x_k$ has already been defined. I would suggest the authors to do it.
\end{itemize}

\resp{This was indeed confusing, $x_k$ refered to the best $k$-term
  approximation of $x$ in the canonical basis. Now we use the notation of Chapter
  1 and prefer to write $\sigma_k(x)_1/\sqrt k$ as in Theorem 1.6.}

\begin{itemize}
\item\em P. 13 – ``...a posteriori...'' $\to$ ``...\emph{a posteriori}...''
\end{itemize}

\resp{Done.}

\begin{itemize}
\item\em P. 14 – Definition 1 – Please clarify what ``...an exponent being a function of the geometries...'' means
\end{itemize}

\resp{The sentence was unclear. We replaced it by ``the exponent
  $\kappa$ depending on the spaces $\mathcal X$ and $\mathcal Y$''.}

\begin{itemize}
\item\em P. 15 – Proposition 1 – ``...such that...'' $\to$ ``...be such that...''
\end{itemize}

\resp{Done.}

\begin{itemize}
\item\em P. 16 – ``This is particularly interesting ...'' It is not entirely clear what is meant by interesting. Is it mathematical convenience, as suggested in the rest of the paragraph? ``...when it ``behaves'' similarly.'' Do you mean behave similarly to a Gaussian random matrix?
\end{itemize}

\resp{The sentence has been more clearly formulated.}

\begin{itemize}
\item\em P. 16 – ``Thresholds and levels of the quantizer...optimally adjusted...'' The operational interpretation of the optimality remark is not entirely clear. Presumably, it is optimal for a fixed signal (say with a certain energy) as the matrix changes randomly. Would then such thresholds still be optimal for arbitrary signals (with arbitrary energies)?
\end{itemize}

\resp{The reviewer is correct. A sentence has been added for
  clarifying this point. In particular, we assume that any other
  signal to be sensed will share the same energy (\emph{e.g.},  with some
  automatic gain adjustment before quantization).}

\begin{itemize}
\item\em P. 17 - I do not think the notion of a distortion optimal quantizer has been introduced earlier. I wonder whether such a notion might fit better in the fundamentals of quantization section.
\end{itemize}

\resp{It is now defined in Sec. 2.2.1 and the reader is
  reminded in Sec. 3.2.}

\begin{itemize}
\item\em P. 17 – The notation $\mathcal{G}’$ associated with the distortion optimal quantizer is ambiguous.
\end{itemize}

\resp{Actually, $\mathcal G' = \tfrac{\ud}{\ud \lambda} \mathcal G$ and $\mathcal
  G$ is defined through its derivative in Sec. 3.2. This is now more
  clearly stated.}

\begin{itemize}
\item\em P. 17 – ``...as it comes by seeing the signal fixed and the Gaussian matrix random..'' – This comment is similar to the previous one – I wonder about the operational relevance of this approach, so the authors might wish to clarify further.
\end{itemize}

\resp{As for the previous remark, the text has been changed
  similarly for clarifying this point.}

\begin{itemize}
\item\em P. 17 – Eq. $(D_p C)$ – does the equality imply that $Q_p (q) = Q_p (Ax)$ ?
\end{itemize}

\resp{Yes, this aspect was missing and is now added.}

\begin{itemize}
\item\em P. 17 – ${\rm diag}(.)$ operator does not seem to have been defined.
\end{itemize}

\resp{Added in the notation section.}

\begin{itemize}
\item\em P. 18 – ``...this choice is not specially the one minimizing...'' – this sentence did not seem clear
\end{itemize}

\resp{It is now clarified as ``it is not sure if this limit case minimizes
the reconstruction error $\|x - \hat{x}_{p, w}\|$ (as a function of $p$) given a certain number of
measurements $m$.''.}

\begin{itemize}
\item\em P. 18 – ``...the one found in uniform QCS...'' – please refer to the appropriate equation
\end{itemize}

\resp{Fixed.}

\begin{itemize}
\item\em P. 20 – Fig. 4 – The authors now seem to use a different variable to denote the saturation level $(G)$
\end{itemize}

\resp{Fixed.}

\begin{itemize}
\item\em P. 20 – ``...with $a_i$ the $i^{\rm th}$ row of A.'' I understand that the authors wish to simplify notation here, but this does not seem to be consistent with the proposed notation.
\end{itemize}

\resp{Fixed.}

\begin{itemize}
\item\em P. 20 – ``These constraints can be imposed on any reconstruction problem...'' I wonder whether the authors might wish to elaborate further, or provide some references.
\end{itemize}

\resp{Reference added.}

\begin{itemize}
\item\em P. 22 – Fig. 5 – This is a very nice picture, but I have found it difficult to read both in back \& white and color. I think that increasing the size of the sub-figures might be appropriate.
\end{itemize}

\resp{Figures have been enlarged.}

\begin{itemize}
\item\em P. 22 – ``Defining $\Sigma_k^*$...'' – I am not sure that $\Sigma_k$ has been defined in this chapter. It has been defined though in Chapter 1, so may be there is no need to re-define.
\end{itemize}

\resp{It has been defined in Chapter 1 and the reader is reminded of
  the definition in Sec. 3.1, as part of Definition 1. In the notation
section we now mention that we use the notation from Chapter 1.}

\begin{itemize}
\item\em P. 22 – Define $e$ in Theorem 3
\end{itemize}

\resp{$e$ is the Euler constant but we have now simplified this
  relation by removing constant and using the notation $\gtrsim$.}

\begin{itemize}
\item\em P. 24 – Eq. (33) – The subscript under sup is not clear.
\end{itemize}

\resp{The subscript has now been placed in the definition of a set
  over which the supremum is evaluated and we now recall the set difference definition below Eq. (33).}

\begin{itemize}
\item\em P. 25 – ``...the BIHT...'' – has the B in the acronym been defined?
\end{itemize}

\resp{It was not defined. We corrected that.}

\begin{itemize}
\item\em P. 27 – ``...compressible signals sensed according (17)...'' $\to$ ``...compressible signals sensed according to (38)...'' ?
\end{itemize}

\resp{Fixed.}

\begin{itemize}
\item\em P. 29 – ``When the signal is approximated using as ... using a ...'' – this sentence does not read well
\end{itemize}

\resp{Fixed}

\begin{itemize}
\item\em P. 31 – The constant $C_{r,\Phi}$ does not appear to have been defined
\end{itemize}

\resp{Fixed}

\begin{itemize}
\item\em P. 32 – Has ${\rm supp}(x)$ already been defined?
\end{itemize}

\resp{This operator belongs to those defined in the book notation
  list.}

\begin{itemize}
\item\em P. 32 – Eq. 49 – Has $\kappa$ been defined? The variable $e$ seems to have been used to denote the Euler’s constant previously (see previous comment), and a vector now. I wonder whether it might be appropriate to use another variable.
\end{itemize}

\resp{Fixed}

\begin{itemize}
\item\em P. 32 – ``...is above the the error level.'' $\to$ ``...is above the error level.''
\end{itemize}

\resp{Fixed}

\begin{itemize}
\item\em P. 32 – Definition 5 – the variable $e$ seems now to be used to denote the Euler’s constant
\end{itemize}

\resp{The symbol $e$ does indeed denote Euler's constant.}

\begin{itemize}
\item\em P. 33 – The authors seem to use $(.)^*$ rather than $(.)^T$, as under the proposed notation, to denote transpose. This also appears throughout this section, \emph{e.g.},  in proof of Theorem 6. Are the authors working with complex matrices, so that they require an Hermitian transpose? I would thus suggest to define it.
\end{itemize}

\resp{These have been changed to $(.)^T$}

\begin{itemize}
\item\em P. 34 – Theorem 6 – $S$ is now used in lieu of $T$ to denote support – Proof of Theorem 6 – it then uses $T$ rather than $S$ to denote support
\end{itemize}

\resp{It is now $T$ in the statement.}

\section{Answers to the 2$^{\rm nd}$ reviewer}

{\em In order to digitize analogous signals $s(t)$ we are confronted with
two tasks, sampling and quantization. The first one represents the
``measurement step'' which yields a discrete representation of the
signal $s(t)$ in terms of countably many coefficients. If $s(t)$ is
bandlimited those coefficients are function samples $s(t_i)$ according
to the Nyquist rate. The ``quantization step'' maps the representation
coefficients to elements in a finite set (bits).}

{\em The novel theory (and practice) of compressed sensing affects the
``measurement step'' and helps to reduce the sampling burden given by
Shannon’s sampling theorem. Under additional sparsity assumptions the
number of measurements to obtain a perfect representation of the
signal can be reduced drastically (for this observation one should
cite e.g. Rauhut, ACHA, 2007).} \newline

\resp{We feel that we sufficiently cover the main approaches in this
  area in the papers we cite. We cite the original CS paper, as well as two
  tutorial papers, one on finite rate of innovation and one on
  subnyquist sampling, each from the corresponding groups which
  developed these methods.}\newline

{\em However, the additional use of
compressed sensing methods can not be considered as improvement to
classical quantization methods in a straightforward way. The
significantly reduced sampling rate might be at the expense of
sensitivity to noise and signal model mismatch. Furthermore, because
of the randomized nature of compressed measurements, commonly used
quantization approaches can not be directly applied.}

{\em Section 2 contains a brief introduction to quantization. In
  Subsection 2.1 the authors describe connections to metric entropy
  and entropy numbers. On top of page 5 they cite the well-known
  result for dyadic entropy numbers of the unit ball $B^n_p$ in
  $\ell^n_2$ if $0 < p < 2$. Here one should rather cite $[60,82]$.}

{\em In fact, the lower bound in the interesting middle case is due to
  K\"uhn [60]. By the way, the upper bound in that case first appeared
  1996 in the book by Edmunds/Triebel ``Function Spaces, Entropy
  Numbers and Differential Operators''.}\newline

\resp{Indeed, [60] (now [63]) did prove that lower bound in the middle
  case. [82] (now [86]), cited in [60] provided upper bounds on the
  entropy numbers that later Edmunds/Triebel and K\"uhn made
  tight. [24] is the first use of those bounds to analyze quantization
  and compressive sensing (which is what we do in this chapter). We
  did add a citation to Edmunds and Triebel ([36]) for
  completeness. Thank you for the suggestion.}\newline

{\em  Moreover, the third case in (5) contains an error. It should be replaced by $2^{−R/n}n^{1/2-1/p}$. I do not see right now if this affects any of the lower bounds given in the paper.}\newline

\resp{This is correct. Simple copy/paste mistake from the middle case. Thank
you for noticing! Corrected.}\newline

{\em At the end of Subsection 2.1 the authors use both, the big-$O$ and
$\Omega$-notion without explaining/defining it.}\newline

\resp{Both reviewers have the same suggestion. We now mainly use asymptotic notation -- $\gtrsim$, $\lesssim$, and $\asymp$ --, which is precisely
defined in the new notation section. In certain cases, where the statement we seek to make is more qualitative (``on the order of'') and it felt to us that it improved the flow, we have left the $O$-notation -- we think that it should be well-known to the targeted audience, so we decided to include only a short sentence rather than a precise definition.
}\newline

{\em At other places (like
in (5)) constants are written out explicitly (but without making clear
the possible dependence on some of the parameters). I suggest to do it
in a more consistent way.}\newline

\resp{We have reviewed our use of asymptotic notation and constants,
  and revised eq. (5) and the rest of the text to make all equations
  more consistent.}\newline

{\em In Paragraph 2.2.2 from formula (10) to (11)
the $k$ is gone. The authors should comment on that issue.}\newline

\resp{Thank you for the observation. In most pre-CS literature the $k$ is
not that relevant and consumed into the constants. However, given the
subsequent development, it is relevant in this case and should have
been included. Corrected.}\newline

{\em Subsection
2.3 is completely useless for readers who are not in the field. It
does not become clear what $\Sigma\Delta$ quantization actually
is. Since this article will be a book chapter the material should be
accessible also for a broader readership. At least there should be a
connection to the later Section 4.}\newline

\resp{We have revised the subsection, hoping that it is easier to understand now. We also note that a mathematical description of $\ensuremath{\Sigma\Delta}\xspace$ is in Section 4.}\newline

{\em In Section 3, Definition 1 a generalized version of the commonly used
RIP is defined. On the first glance this looks a bit unnatural to me
if $X=\ell^n_p$ and $Y =\ell^m_2$ for $p \neq 2$ since for $m = n$
those spaces are not isometrically isomorphic to each other. Some
comments would be desirable why an ``almost isometry'' can make sense
here. On the other hand, Proposition 1 is interesting and shows that
one can prove reasonable statements on the expense of the
``exponential price'' on the number of rows of $A$.}\newline

\resp{A sentence has been added to explain this
  particularity.}\newline

{\em Section 3.4 is a nice summary on the results on 1-bit compressed
sensing. Every reader would believe that this is actually a key issue
in combining quantization and compressed sensing. The nice and
surprising results in Theorems 3 and 4 together with the provided
relevant literature on this topic are definitely useful for readers
who are at least a bit familiar with compressed sensing.}

{\em Let me comment on some formal (notational) issues.}
\begin{itemize}
\item\em In Section 4 the authors use $A^*$ for the transpose of a matrix. If I understood correctly,
the convention is $A^T$.
\end{itemize}

\resp{Fixed}

\begin{itemize}
\item\em Definition 5 in Section 4 is standard and should be somewhere earlier in the book. Probably in a first chapter which introduces CS.
\end{itemize}

\resp{Our understanding is that the definition is currently not included in
Chapter 1, so we include it here. If the editors are aware that it is
included in some earlier chapter (or the decide to include it in Ch.1)
we will be happy to remove it from out chapter.}

\begin{itemize}
\item\em The authors are not consistent in the style of the section headings (capital letters).
\end{itemize}

\resp{Thanks for noting the inconsistencies, we have now capitalized all section headings (except for articles and propositions).}

\begin{itemize}
\item\em Define $O$ and $\Omega$ notation.
\end{itemize}

\resp{ See our response above to reviewer 1.
  }

\begin{itemize}
\item\em Put a dot after formula (6) on page 5.
\end{itemize}

\resp{Done.}

\begin{itemize}
\item\em Which norm is used in the definition of the $r$-ball in (2) ? Probably the Euclidean norm, but then $N$ should be replaced by $n$ to be consistent with the volume formula and (5).
\end{itemize}

\resp{Indeed. Noted the norm and fixed $n$ to have the correct case.}

\begin{itemize}
\item\em Write ``Banach'' instead of ``banach'' in reference [82].
\end{itemize}

\resp{Done.}

\begin{itemize}
 \item\em The style of the reference list is not consistent.
\end{itemize}

\resp{We have gone over our bibiliography and put the references in a uniform style.}

\begin{itemize}
\item\em In some of the theorems is a reference after its number, in
  others a title...
\end{itemize}

\resp{Fixed.}\newline

{\em In general, the article gives a reasonable introduction to the
mentioned field. However, the striking improvement when using
compressed sensing methods (in comparison to classical methods) could
be worked out more convincingly. Moreover, some of the content is
mostly understandable for people already working in the field. But for
those the article gives a useful collection of relevant results and
literature on the topic. What concerns the writing style of the
article (besides the above mentioned points) it is easy to see which
one of the authors was responsible for which part. Since this is going
to be a book chapter one should care about those things as well.}

\resp{We tried to make the notation more consistent and as much as possible
unify some of the conventions we use. We hope you find the revision satisfactory}\newline

{\em I \textbf{recommend} the article for publication as the proposed book chapter after a revision according to the points mentioned above.}\newline

\resp{Thank you for your comments and the thorough review. We appreciate the
feedback.}

\end{document}

\subsection{Noise, Quantization and Tradeoffs}
\label{sec:extensions}

The sections above were focused on noiseless QCS models. These models
only consider the statistical or the geometrical properties of
quantization of CS measurements under high or low resolution
modes. However, any signal acquisition system is subject to noise
corruption before quantization, either on the measurement process or
on the signal itself. Such noise can be incorporated in a more general
model
\begin{equation}
  \label{eq:qcs-pcm-noisy}
  q = \ensuremath{Q}(A (x + \xi_x) + \xi_{\rm s}),
\end{equation}
where $\xi_x\in \mathbb R^n$ and $\xi_{\rm s} \in \mathbb R^m$ corrupt
the signal and the sensing, respectively, before
quantization. Examining the impact of such noise in signal recovery
leads to new interesting questions.

In \cite{zymnis2010compressed} two efficient reconstruction methods are
developed for sparse or compressible signals sensed according
\eqref{eq:qcs-pcm-noisy} under sensing noise only, \emph{i.e.},  $\xi_{x} = 0$. The
two approaches are mainly numerical: one relies on a maximum likelihood
formulation, built on the quantization model and on a known Gaussian noise
distribution, the other follows a least square principle. The two resulting methods are both
regularized by an $\ell_1$-norm accounting for sparse signal prior. A
provably convergent procedure inherited from a fixed point
continuation method is used for reconstructing the signal in the two
possible frameworks. With their approach, the combined effects of
noise and coarse quantization can be jointly handled. Reasonable
reconstruction results are achieved even using 1 or 2 bits per
measurement.

The case $\xi_x \neq 0$, $\xi_{\rm s} = 0$ boils down to an interaction of the
well-understood phenomenon of \emph{noise folding} in CS
\cite{davenport2012pros} and quantization \cite{laska2012regime}. Noise-folding in unquantized CS
says that under a weak assumption of orthogonality between the rows of $A$,
the variance of the component $A \xi_x$ undergoes a multiplication
by $n/m$ compared to the variance $\sigma_{\xi}^2$ of
$\xi_x$. This impacts directly the reconstruction error of
signals. The corresponding MSE is then $n/m$ times higher than the
noise power, or equivalently, the SNR looses 3 dB each time $m$ is
divided by 2 \cite{davenport2012pros}.

An extension of this result to noisy QCS has been provided
in~\cite{laska2012regime}, assuming the sensing matrix $A$ is RIP of
order $k$ and constant $\delta$. In this case, if $\xi_x$ is
standard normally distributed and if the quantizer has resolution $B$,
then, under a random signal model where the signal support $T$ is
chosen uniformly at random in $\{1,\cdots, n\}$ and the amplitudes of
the non-zero coefficients are standard normally distributed,
\begin{equation}
  \label{eq:regime-change-tradeoff}
(1-\delta)\mathbb E \|x - \hat x\|^2 = 2^{-2B+1} \tfrac{k}{m} \mathbb E \|x\|^2
+ 2\big(2^{-2B}
+  1 \big) \tfrac{n}{m}\,\mathbb E \|\xi_x|_T\|^2 + km \kappa,
\end{equation}
where $\hat x = (A_T^\dagger q)_T$ is the oracle-assisted
reconstruction of $x$ knowing the
support $T$ of $x$ for each of its realization, and
$$
\kappa =
\max_{i\neq j} |\mathbb E \ensuremath{Q}( a_i^T(x+\xi_x)) \ensuremath{Q}(
a_j^T(x+\xi_x))|,
$$
measures the worst correlation between distinct
quantized measurements.

In \eqref{eq:regime-change-tradeoff}, the first term
accounts for the quantization error of the signal itself, while the
second term represents both the error due to folded signal noise as
well as the quantization of that noise. Finally, the third term
reflects a distortion due to correlation between quantized
measurement. It is expected to be negligible in CS scenarios,
specially when $B$ increases or if a \emph{dithering} is added to $\ensuremath{Q}$ \cite{gray1998quantization}.

\begin{figure*}[!t]
   \centering
   \begin{tabular}{cc}
     \includegraphics[width=.4\textwidth]{RDII35dBsquishDISCRETE}&
     \includegraphics[width=.4\textwidth]{RDII20dBsquishDISCRETE}\\
       (a) \small{$\mathrm{ISNR} = 35$dB, optimal bit-depth $= 7$}&(b) \small{$\mathrm{ISNR} = 20$dB, optimal bit-depth $= 5$}\\
          \includegraphics[width=.4\textwidth]{RDII10dBsquishDISCRETE}&
   \includegraphics[width=.4\textwidth]{RDII5dBsquishDISCRETE}\\
	(c) \small{$\mathrm{ISNR} = 10$dB, optimal bit-depth $= 2$}&(d)\small{ $\mathrm{ISNR} = 5$dB, optimal bit-depth $= 2$}\\
   \end{tabular}
   \caption{Upper bound on the oracle-assisted reconstruction error as
     a function of bit-depth $B$ and ISNR at
constant rate $R = mB$ \cite{laska2012regime}.
The black dots denote the minimum point on each curve.
   }
   \label{fig:bound}
\end{figure*}

Numerical study of \eqref{eq:regime-change-tradeoff} shows that, at
constant rate $R = mB$, a tradeoff can be expected between a
\emph{measurement compression} (MC) regime, where $m$ is small (but
still high enough to guarantee $A$ to be RIP) and $B$ is high, and a
\emph{quantization compression} (QC) regime, where $m$ is high
compared to the standard CS setting but $B$ is small. Interestingly,
the optimal bit-depth $B$, minimizing the expected reconstruction
error, depends on the input SNR: $\mathrm{ISNR}=20 \log_{10}
\|x\|/\|\xi_x\|$. This is illustrated in Fig.~\ref{fig:bound} where
the evolution of \eqref{eq:regime-change-tradeoff} (discarding the
effect of the third term) is plotted for four different noise
scenarios. The optimal bit depth decays smoothly with the ISNR,
suggesting that the QC regime is preferable at low ISNR while MC is
clearly better at high ISNR. The general behavior of
Fig.~\ref{fig:bound} is also confirmed on Monte Carlo error estimation
of the oracle-assisted reconstruction defined above
\cite{laska2012regime}.
\subsection{Non-Uniform Scalar Quantization}
\label{sec:nonunif-scal-quant}

If the distribution of the measurements is known,
  quantization distortion can be decreased by adopting a non-uniform
  scalar quantizer.
For instance, when $A$ is a random Gaussian matrix
viewing the signal as fixed and the matrix as randomly drawn, the
distribution of the components of $y=A x$ is also Gaussian with a
variance proportional to the signal energy
$\|x\|_2^2$ (and similarly, for other matrix constructions, such as
  ones drawn with random sub-Gaussian entries). Assuming the acquired
  signal energy can be fixed, \emph{e.g.},  using some automatic gain control, the
  known distribution of the measurements can be exploited in the
  design of the quantizer, thanks for example to the Lloyd-Max algorithm
  mentioned in Sec.~\ref{sec:intro_scalar}~\cite{Lloyd1982}. In
  particular, the
  quantization thresholds and levels are then optimally adjusted to this
  distribution.

This section shows that the formalism developed in Sec.~\ref{sec:unif-scal-quant} can indeed
be adapted to non-uniform scalar quantizer. To understand this adaptation, we exploit a common tool in
quantization theory~\cite{gray1998quantization}: any non-uniform
quantizer can be factored as the composition of a ``compression'' of
the real line over $[0, 1]$ followed by a uniform quantization of the
result that is finally re-expanded on $\mathbb R$. Mathematically,
\begin{equation}
  \label{eq:compander}
  \ensuremath{Q}\ =\ \mathcal G^{-1} \circ \ensuremath{Q}_{\ensuremath{\Delta}} \circ \mathcal G,
\end{equation}
where $\mathcal G:\mathbb R \to [0,1]$ is the \emph{comp}ressor and $\mathcal
G^{-1}:[0,1]\to \mathbb R$ is the exp\emph{ander}, giving the name
\emph{compander} as a portemanteau.

In particular, under HRA, the compressor $\mathcal G$ of a distortion
optimal quantizer, \emph{i.e.},  one that minimizes $\mathbb E|X-\ensuremath{Q}(X)|^2$ for a
source modeled as a random variable $X$ with pdf $\ensuremath{\varphi}$, must satisfy
$$
\tfrac{\ud}{\ud \lambda} \mathcal G(\lambda) = \left( \int \ensuremath{\varphi}^{1/3}(t) \ud t\right)^{-1} \ensuremath{\varphi}^{1/3}(\lambda),
$$
and if $\ensuremath{Q}$ is an
optimal $B$-bit quantizer (\emph{e.g.},  obtained by Lloyd-Max method) then $\ensuremath{\Delta}
= 2^{-B}$ in \eqref{eq:compander}. In this case, the Panter and Dite formula estimates the
quantizer distortion as \cite{panter1951quantization}
$$
\mathbb E|X-\ensuremath{Q}(X)|^2 \simeq_B \tfrac{2^{-2 B}}{12} \fnorm{\ensuremath{\varphi}}_{1/3}
=: \sigma^2_{\rm PD},
$$
with $L_s$-norm $\fnorm{\ensuremath{\varphi}}_{s} = (\int |\ensuremath{\varphi}^{s}(t)| \ud t)^{1/s}$ and
where ``$\simeq_B$'' means that the relation tends to an equality when
$B$ is large. The rest of this section assumes that the expected distribution is Gaussian, \emph{i.e.},  if $\ensuremath{\varphi}
\sim \mathcal
N(0,\sigma_0^2)$ and $\fnorm{\ensuremath{\varphi}}_{1/3} = \tinv{2}\sqrt{3}\pi
\sigma^2_0$, as it comes by seeing the signal fixed (with known
  energy) and the Gaussian
matrix random in~CS.

Compander theory generalizes quantization
consistency in the ``compressed'' domain, \emph{i.e.},
$$
|\mathcal G(\lambda) - \mathcal G(\ensuremath{Q}(\lambda))| \leq \ensuremath{\Delta}/2 = 2^{-B-1}.
$$ Therefore, for the right compressor $\mathcal G$, in the noiseless QCS
model \eqref{eq:qcs-pcm}, the signal $x$ provides consistency
constraints to be imposed on any reconstruction candidate $x'$:
\begin{equation}
  \label{eq:quant-const-nonunif}
  \|\mathcal G(A x') - \mathcal G(q)\|_\infty \leq \ensuremath{\Delta}/2 = 2^{-B-1}. \tag{QC}
\end{equation}
This generalizes the uniform quantization consistency \eqref{eq:qc-unfi} introduced
in Sec.~\ref{sec:unif-scal-quant}.

The compander formalism is leveraged in \cite{Jacques2013}, to
generalize the approach described in Sec. \ref{sec:unif-scal-quant} to
non-uniform quantization. In particular, a new set of parametric
constraints are introduced, the $p$-Distortion Consistency (or D$_p$C)
for $p\geq 2$. These have for limit cases the QC above and the
\emph{distortion consistency} constraint (DC) arising from Panter and
Dite formula, namely, the constraint imposing any reconstruction
candidate $x'$ to satisfy \cite{Dai2009}
\begin{equation}
  \|A x' - q\|^2_2 \leq \epsilon^2_{\rm PD} := m \sigma^2_{\rm PD}, \tag{DC}
\end{equation}
with DC asymptotically satisfied by~$x$ when both $B$ and
$m$ are large.

The D$_p$C constraint corresponds to imposing that a candidate
signal $x'$ satisfies
\begin{equation}
\label{eq:dpc}
\|A x' - \ensuremath{Q}_p[q]\|_{p,w}\ =\ \|A x' - \ensuremath{Q}_p[Ax]\|_{p,w}\ \leq\ \epsilon_{p,w}, \tag{D$_p$C}
\end{equation}
where $\|v\|_{p,w} = \|\diag(w) v\|_p$ is the weighted $\ell_p$-norm
of $v \in \mathbb R^m$ with weights $w \in
\mathbb R^m_+$, denoting by $\diag(w)$ the diagonal matrix having $w$ on its diagonal. The mapping $\ensuremath{Q}_p:\mathbb R^m \to \mathbb R^m$ is a
  post-quantization modification of $q$ characterized
componentwise hereafter and such that $\ensuremath{Q}_p[q]=\ensuremath{Q}_p[Ax]$.

Under HRA, a careful design of $\ensuremath{Q}_p$, $w$ and the bounds
$\epsilon_{p,w}$ ensures that D$_2$C amounts to imposing DC on $x'$
and, that as $p\to +\infty$, D$_p$C tends to QC \cite{Jacques2013}.
Briefly, if $q_i$ falls in the quantization bin $\ensuremath{\mathcal{C}}_j$,
$\ensuremath{Q}_p(q_i)$ is defined as the minimizer of
$$
\min_{\lambda\in\ensuremath{\mathcal{C}}_j}\ \int_{\ensuremath{\mathcal{C}}_j} |t - \lambda|^p\ \ensuremath{\varphi}(t)\,\ud t.
$$
Actually, $\ensuremath{Q}_2(q_i) = q_i$ by equivalence with \eqref{eq:LM-condition}, and $\lim_{p\to \infty}\ensuremath{Q}_p(q_i) =
\tinv{2}(t_j+t_{j+1})$. The weights are defined by the
quantizer compressor $\mathcal G$ with $w_i(p)= \tfrac{\ud}{\ud \lambda} \mathcal G\big(\ensuremath{Q}_p[q_i]\big)^{\frac{p-2}{p}}$.
Moreover, under HRA and asymptotically in $m$, an optimal bound $\epsilon_p$ reads
$\epsilon_{p,w}^p = m\,
\tfrac{2^{-Bp}}{(p+1)\,2^{p}}\,\fnorm{\ensuremath{\varphi}}_{1/3}$. For $p=2$, $\epsilon_{2,w}=\epsilon_{\rm PD}$ matches the distortion power estimated by
the Panter and Dite formula, while for $p\to
+\infty$, $\epsilon_{p,w} \to \tinv{2} 2^{-B}$, \emph{i.e.},  half the size of
the uniform quantization bins in the domain compressed by $\mathcal G$.

Similarly to Sec.~\ref{sec:unif-scal-quant}, using
\eqref{eq:dpc} as a fidelity constraint in the signal reconstruction leads to the definition of a Generalized
Basis Pursuit DeNoise program:
\begin{equation}
  \hat{x}_{p, w}\ =\ \arg\,\min_{z\,\in\,\mathbb R^n} \|z\|_1
  \ {\rm
  s.t.}\ \|\ensuremath{Q}_p(q) - A z\|_{p,w} \leq \epsilon_{p,w}. \tag{GBPDN$(\ell_{p,w})$}
\end{equation}

Ideally, we would like to directly set $p=\infty$ in order to enforce
consistency of $\hat{x}_{p, w}$ with $q$. However, as studied in
\cite{Jacques2013}, it is not certain that this limit case
  minimizes the reconstruction error $\|x - \hat{x}_{p, w}\|$ as a
  function of $p$, given a certain number of measurements $m$.

Actually, the stability of GBPDN can be established from the one
of BPDQ (Sec.~\ref{sec:unif-scal-quant}) if we impose $A$ to satisfy the more general RIP$_{\ell^m_{p,w},
  \ell^n_2}$, as formally defined in \eqref{eq:rip-p}. Indeed, for any
weighting vector $w$, we have always $\|\ensuremath{Q}_p(q) - A z\|_{p,w} =
\|q' - A' z\|_{p}$ with $q' = \diag(w) \ensuremath{Q}_p(q)$ and $A' = \diag(w)
A$. Therefore, we know from
Theorem~\ref{prop:l2-l1-instance-optimality-BPDQ} that if $A'$ is
RIP$_p$, or equivalently if $A$ is RIP$_{\ell^m_{p,w},
  \ell^n_2}$, with the additional condition \eqref{eq:cond-on-delta-p} on its RIP constants at different
sparsity levels, then the solution of GBPDN$(\ell_{p,w})$ will be stable in the
sense of \eqref{eq:BPDQ-l2-l1-inst_opt}, \emph{i.e.},
$$
\|\hat x_{p,w} - x\|\ \lesssim\ \tfrac{\epsilon_{p,w}}{\mu_{p,w}} +
\tfrac{\sigma_k(x)_1}{\sqrt k}.
$$

Compared to the unit weights case (as involved by
the RIP$_p$), a random Gaussian matrix $A$ with $a_{ij} \sim_{\rm iid}
\mathcal N(0,1)$ satisfies the RIP$_{\ell^m_{p,w},
  \ell^n_2}(k,\delta_k,\mu_{p,w})$ with high probability provided that $m$ grows like
$O\big( (\theta_p \delta_k^{-2} (k
\log(n/k))^{p/2} \big)$.
The ratio $\theta_p := \|w\|_\infty/(m^{-1/p}\|w\|_p)$ depends on the \emph{conditioning} of $w$. It is equal to 1 for constant
weights (recovering \eqref{eq:SGR-RIP-measur-bound}), while it
increases with the dynamic range of $w$. For the weight $w(p)$ defined previously and with a Gaussian
optimal quantizer, $\theta^{p/2}_p
\simeq_{m,B} \sqrt{p+1}$ asymptotically in $m$ and $B$.

As for the uniform case, a strong (polynomial) oversampling in
$m$ is thus required for satisfying the RIP$_{\ell^m_{p,w},
  \ell^n_2}$ at $p>2$ compared to the minimal number of measurements needed
at $p=2$. However, an asymptotic analysis of
$\epsilon_{p,w}/\mu_{p,w}$ shows that the GBPDN reconstruction error due to quantization for a Gaussian
sensing matrix behaves like \cite{Jacques2013}
$$
\|\hat x_{p,w} - x\| \lesssim \tfrac{2^{-B}}{\sqrt{p+1}} + \tfrac{\sigma_k(x)_1}{\sqrt k},
$$
This error decay is thus similar to the one found in
\eqref{eq:unif-error-decay} for uniform QCS with
now a direct interpretation in terms of the quantizer bit-depth $B$.

Efficient convex optimization methods, like those relying on proximal
algorithms \cite{combettes2011proximal}, can also be used to
numerically solve GBPDN.  In \cite{Jacques2013}, numerical simulations
show that the reconstruction qualities reached in the reconstruction
of sparse signals from their non-uniformly quantized measurements
behave similarly, with respect to $p$ and $m$, to those observed in
Sec.~\ref{sec:unif-scal-quant} for the uniformly quantized CS setting.

We should also remark that beyond QCS, the stability of GBPDN (when
$A$ is RIP$_{p,w}$) can also be used for reconstructing signals
acquired under a (heteroscedastic) noisy sensing model $y = Ax +
\xi$ where $\xi\in \mathbb R^m$ is an additive generalized Gaussian
noise with bounded $\ell_{p,w}$-norm for some specific weight $w\in
\mathbb R^m_+$ \cite{varanasi1989pgg,Jacques2013}.

\section{Scalar Quantization and Compressive Sensing}
\label{sec:scalarCS}

The interplay of scalar quantization and compressed sensing has been
widely explored in the literature. In addition to the lower bounds
discussed in~\ref{sec:perf-bounds-sparse}, there is significant
interest in providing practical quantization schemes and
reconstruction algorithms with strong performance guarantees.

This part explores these results. Our development considers
the following quantized compressed sensing (QCS) model:
\begin{equation}
  \label{eq:qcs-pcm}
  q = \ensuremath{Q}(y) = \ensuremath{Q}(A x),
\end{equation}
where $x \in \mathbb R^n$ and $A \in \mathbb R^{m \times n}$. The sensing
matrix can be, for instance, a random Gaussian sensing matrix $A$ such
that $a_{ij} \sim_{\rm iid} \mathcal N(0,1)$. Note that the scaling of the
  entries of the sensing matrix should be independent of $m$. This
  allows us to fix the design of the scalar quantizer $\ensuremath{Q}$ since
  the dynamic range of the components of $Ax$ is then independent of
  the number of measurements. This has no consequence on some of the
  common requirements the sensing matrix must satisfy, such as the
  Restricted Isometry Property (see Chap. 1), as soon as an
  appropriate rescaling of $A$ is applied. For instance, if $A$ has
  RIP of order $2k$ and if $A \to \lambda A$ for some $\lambda>0$,
  then $A/\lambda$ has RIP of the same order and the error bound
  (1.20) in the stability Theorem 1.6 remains
  unchanged~\cite{Jacques2010}.

The first two parts, Sec.~\ref{sec:unif-scal-quant} and
Sec.~\ref{sec:nonunif-scal-quant}, focus on the \emph{high resolution
  assumption} (HRA) that simplifies the QCS model. Under HRA, the
quantization bin widths---$\ensuremath{\Delta}$ or the distance between two
consecutive thresholds---are small with respect to the dynamic range
of the unquantized input. This allows us to model the quantization
distortion $\ensuremath{Q}(A x) - Ax$ as uniform white noise
\cite{gray1998quantization}. Determining bounds on its power and
moments can better constrain signal reconstruction methods, such as the
basis pursuit denoise (BPDN) program
\cite{Chen98atomic,candes2008rip}, which is commonly used for
reconstructing signals whose CS measurements are corrupted by
Gaussian noise. However, the price to pay is an oversampling in CS
measurements.

Sec.~\ref{sec:unif-scal-quant-with-saturation} considers scalar
quantizers with \emph{saturation}. Saturation induces information loss
in the measurements exceeding the saturation level. However,
democracy---a key property of compressive sensing measurements that
makes every measurement equally informative---provides robustness
against such corruption.

In Sec.~\ref{sec:1bitCS}, very low-resolution quantization is studied
through 1-bit compressed sensing.  In this case, the HRA cannot be
assumed anymore---the quantization bins are the two semi-infinite
halves of the real line---and the analysis of the QCS model relies on
high dimensional geometric arguments.

Finally, Sec.~\ref{sec:extensions} studies how noise, either on the
signal or on the measurements, can impact the QCS model
\eqref{eq:qcs-pcm}, the reconstruction error and the quantizer
trade-offs. In particular, at constant bit budget $\ensuremath{R} = m\ensuremath{B}$, the
total noise power determines the optimal trade-off between quantizer
precision and number of measurements.

\input{scalar_uniform}
\input{scalar_nonuniform}
\input{scalar_uniform_saturation}
\input{1bit}
\input{scalar_extensions}

\subsection{Finite-Range Scalar Quantizer Design}
\label{sec:unif-scal-quant-with-saturation}

So far we have only considered a scalar quantizer model without
saturation. Practical scalar quantizers have a finite range, which
implies a saturation level $\pm\ensuremath{S}$ and, using \ensuremath{B}\ bits per
coefficient, a quantization interval equal to
\begin{align}
  \ensuremath{\Delta}=\ensuremath{S} 2^{-B+1}.
  \label{eq:quant_interval}
\end{align}
In order to determine the optimal saturation rate, the system designed
needs to balance the loss of information due to saturation, as
\ensuremath{S}\ decreases, with the increased quantization error due to an
increasing quantization interval in~\eqref{eq:quant_interval}, as
\ensuremath{S}\ increases. In classical systems, this balance requires setting
the quantization level relatively close to the signal amplitude to
avoid saturation. On the other hand, in compressive sensing systems,
the incoherence of the measurements with the sparsity basis of the
signal makes them more robust to loss of information and enables
higher saturation levels with smaller quantization
intervals.

A key property of compressive measurements, which provides the
robustness to loss of information, is {\em democracy}. Intuitively,
each measurement contributes an equal amount of information to the
reconstruction. If the signal is slightly oversampled, relative to the
rate required for CS reconstruction, then any subset with enough measurements should be sufficient to recover the
signal. The notion of democracy was first introduced
in~\cite{CalDau::2002::The-pros-and-cons,Gun::2000::Harmonic-analysis}
in the context of information carried in each bit of the
representation; the definition below strengthens the concept and
formulates it in the context of compressive
sensing~\cite{DLBB_ECETR09,LasBouDav::2009::Demcracy-in-action}.

\begin{definition}
Let $A\in{\mathbb R}^{m \times n}$, and let $\widetilde{m} \le m$ be
given.  We say that $A$ is $(\widetilde{m},k,\delta_k)$-democratic if,
for all row index sets $\Gamma$ such that $|\Gamma|\ge \widetilde{m}$,
any matrix $\widetilde{A}=((A^T)_\Gamma)^T$, \emph{i.e.},  comprised of a
$\Gamma$-subset of the rows of $A$, satisfies the RIP of order $k$
with constant $\delta_k$.
\end{definition}
This definition takes an adversarial view of democracy: a matrix $A$
is democratic if an adversary can pick any $d=m-\widetilde{m}$ rows to
remove from $A$, and the remaining matrix still satisfies the
RIP. This is a much stronger guarantee than just randomly selecting a
subset of the rows to be removed. Such a guarantee is important in the
case of saturation robustness because the saturated measurements are
the largest ones in magnitude, \emph{i.e.},  potentially the ones most aligned with the
measured signal and, presumably, the ones that capture a significant
amount of information. Still, despite this strict requirement,
randomly generated matrices can be democratic if they have a
sufficient number of rows.

\begin{theorem}[\cite{DLBB_ECETR09}] \label{thm:democracy}
Let $A\in{\mathbb R}^{m \times n}$ with elements $a_{ij}$ drawn
according to $\mathcal{N}(0, \frac{1}{m})$ and let $\widetilde{m} \le m$, $k <
\widetilde{m}$, and $\delta \in (0,1)$ be given.  Define $d = m -
\widetilde{m}$. If
\begin{align}
\label{eq:Mdem}
m = C_{1} (k + d)\log\left(\frac{n+m}{k + d}\right),
\end{align}
then with probability exceeding $1 - 3 e^{-C_2 m}$ we have that $A$ is
$(\widetilde{m},k,\delta/(1-\delta))$-democratic, where $C_1$ is
arbitrary and $C_2 = (\delta/8)^2 - \log(42 e/ \delta)/C_1.$
\end{theorem}
The practical implication of democratic measurements is that
information loss due to saturated measurements can be tolerated.

Saturated measurements are straightforward to detect, since they
quantize to the highest or the lowest level of the quantizer. The
simplest approach is to treat saturated measurements as corrupted, and
reject them from the reconstruction, together with the corresponding
rows of $A$. As long as the number of saturated measurements is not
that large, the RIP still holds and reconstruction is possible using
any sparse reconstruction algorithm.

However, saturated measurements do contain the information that the
measurement is large. In the context of consistent reconstruction,
they can be used as constraints in the reconstruction process. If a
measurement $i$ is positively saturated, then we know that $(Ax)_i \ge \ensuremath{S}-\ensuremath{\Delta}$. Similarly, if it is negatively saturated,
$(Ax)_i \le -\ensuremath{S}+\ensuremath{\Delta}$. These constraints can be imposed on
any reconstruction algorithm to improve performance~\cite{LasBouDav::2009::Demcracy-in-action}.

\begin{figure}[!t]
   \centering
   \begin{tabular}{ccc}
  \includegraphics[width=.32\linewidth]{compare_gauss_2_wS}&
  \includegraphics[width=.32\linewidth]{compare_gaussellp_04_wS}&
  \includegraphics[width=.32\linewidth]{compare_gaussellp_08_wS}\\
  (a) $k=20$ &(b) $x\in w\ell_{0.4}$&(c) $x\in w\ell_{0.8}$\\
  \end{tabular}
   \caption{Saturation performance using $\ell_1$ minimization for (a)
     exactly $k=20$-sparse signals and compressible signals in weak
     $\ell_p$ for (b) $p=0.4$ and (c) $p=0.8$, with $n=1024$, $m=384$,
     and $\ensuremath{B}=4$.  The reconstruction SNR as a function of the
     saturation level is measured on the left y-axis assuming (solid line)
     conventional reconstruction, \emph{i.e.},  ignoring saturation, (dotted
     line) enforcing saturation consistency, and (dashed line)
     rejecting saturated measurements. The dashed-circled line,
     measured on the right y-axis, plots the average saturation rate
     given the saturation level.}
   \label{fig:saturation}
\end{figure}

Fig.~\ref{fig:saturation} demonstrates the effect of each approach. As
demonstrated in the plots, rejecting saturated measurements or
treating them as consistency constraints significantly outperforms
just ignoring saturation. Furthermore, if saturation is properly taken
into account, a distortion optimal finite-range scalar quantizer
should be designed with significant saturation rate, often more than
20\%. While the figures suggest that saturation rejection and
saturation consistency have very similar performance, careful
examination demonstrates, as expected, that consistency provides more
robustness in a larger range of saturation rates and conditions. A
more careful study and detailed discussion can be found
in~\cite{LasBouDav::2009::Demcracy-in-action}. Furthermore, further
gains in the bit-rate can be achieved by coding for the location of
the saturated measurements and transmitting those
separately~\cite{kostina2011value}.
\subsection{Uniform Scalar Quantization}
\label{sec:unif-scal-quant}

First we consider the QCS model \eqref{eq:qcs-pcm} using a uniform
quantizer with resolution $\ensuremath{\Delta}$ and a set of levels \ensuremath{\mathcal{Q}},
$$
q = \ensuremath{Q}(y) = \ensuremath{Q}(A x) \in \ensuremath{\mathcal{Q}}^m,
$$
measuring a signal $x\in \mathbb R^n$ using a sensing matrix $A\in\mathbb
R^{m\times n}$. For simplicity, we assume henceforth that $x$ is
sparse in the canonical basis, \emph{i.e.},  $\Psi = I$.

We consider a quantizer $\ensuremath{Q}$ that has uniform quantization regions, \emph{i.e.},
$t_{j+1}-t_j = \ensuremath{\Delta}$ for all $j$, and, setting $t_j = j \ensuremath{\Delta}$,
quantization levels
$\ensuremath{l}_j = \frac{t_j+t_{j+1}}{2} = (j +\tinv{2})
\ensuremath{\Delta}$ in $\ensuremath{\mathcal{Q}}$.

By definition, the signal $x$ satisfies the following \emph{quantization consistency}
constraint (QC$_u$)
\begin{equation}
  \label{eq:qc-unfi}
  \|\ensuremath{q} - Ax\|_{\infty} \leq \ensuremath{\Delta}/2. \tag{QC$_{\rm u}$}
\end{equation}
From this fact, we can also deduce that
$$
\|A x - q\|_2 \leq \sqrt{m}\,\|A x - q\|_\infty
\leq \sqrt m \ensuremath{\Delta}/2.
$$
This shows that the QCS model can be assimilated to a noisy CS model
\begin{equation}
  \label{eq:noisy-unfi-QCS}
  q = \ensuremath{Q}(A x) = Ax + \xi,
\end{equation}
with a ``noise'' $\xi = \ensuremath{Q}(A x) - Ax$ of bounded $\ell_2$-norm, \emph{i.e.},
$\|\xi\|_2 \leq \sqrt m \ensuremath{\Delta}/2$.

The quantization noise power can be further reduced using the high
resolution assumption. Under this assumption, the coefficients of $y$
may lie anywhere in the quantization region determined by the
coefficients of $q$ and it is natural to model the quantization
distortion $\xi$ as a uniform white noise, \emph{i.e.},
$$
\xi_i \sim_{\rm iid}\ \mathcal
U([-\ensuremath{\Delta}/2,\ensuremath{\Delta}/2]).
$$
Under this model, a simple use of the
Chernoff-Hoeffding bound \cite{hoeffding1963pis} provides, with high probability
$$
\|\xi\|^2_2 \leq \epsilon^2_2 := \tfrac{\ensuremath{\Delta}^2}{12} m + \zeta
\tfrac{\ensuremath{\Delta}^2}{6\sqrt 5} m^{1/2},
$$
for a small constant $\zeta>0$.

The first approach in modeling and understanding QCS exploited this
bound and the development of noise-robust CS approaches to impose a
\emph{distortion consistency constraint} (DC$_u$)~\cite{candes2006ssr}
\begin{equation}
  \label{eq:dc-unfi}
  \|\ensuremath{q} - Ax'\|_2 \leq \epsilon_2, \tag{DC$_{\rm u}$}
\end{equation}
on any candidate signal $x'$ estimating $x$. This was indeed a natural
constraint to consider since most noise-robust compressed sensing
reconstruction methods can incorporate a bounded $\ell_2$-norm
distortion on the measurements. For instance, the BPDN program can
find a solution $\hat x$ of
\begin{equation}
  \label{eq:BPDN}
\hat x\ =\ \arg\,\min_{z} \|z\|_1\ \ {\rm s.t.}\ \ \|q - Az\|_2 \leq \epsilon_2. \tag{BPDN}
\end{equation}
Then, if the sensing matrix $A' = A/\sqrt{m}$ satisfies the RIP with constant
$\delta \leq 1/\sqrt{2}$ on $2k$ sparse signals, it is known \cite{Cai14} that
$$
\|x - \hat x\|_2 \lesssim \tinv{\sqrt m}\,\epsilon_2 + \tinv{\sqrt k}\,\sigma_k(x)_1 \asymp \ensuremath{\Delta} + \tinv{\sqrt k}\,\sigma_k(x)_1,
$$
where $\sigma_k(x)_1$ is the best $k$-term approximation
  defined in (1.2).

This approach has two drawbacks. First, there is no guarantee that the
solution $\hat x$ satisfies the QC$_{\rm u}$ constraint above, \emph{i.e.},
$\|\ensuremath{q} - A\hat x\|_{\infty} \nleq \ensuremath{\Delta}/2$. This shows that some sensing
information has been lost in the reconstruction. Moreover, as
described in Sec.~\ref{sec:scal-quant-overs}, the consistency of the
solution helps in reaching the lower bound
\cite{goyal_1998_lowerbound_qc,powell_consistent,jacques2014error}
\[ (\mathbb E \|x - \hat x\|^2)^{1/2} \gtrsim \tfrac{k}{m}\,\ensuremath{\Delta}
\]
in the oversampled setting.  Second, from a maximum \emph{a posteriori}
standpoint, since every constrained optimisation corresponds to an
unconstrained Lagrangian formulation, imposing a small $\ell_2$-norm
on the residual $q - A \hat x$ can be viewed as enforcing a Gaussian
distribution on $\xi$, which is not the uniform one expected from
the HRA.

To circumvent these two limitations, \cite{Jacques2010} studied the
Basis Pursuit DeQuantizer (BPDQ$_p$) program
\begin{equation}
  \label{eq:BPDQ}
\hat x_p = \arg\,\min_{z}\ \|z\|_1\ \ {\rm s.t.}\ \ \|q - Az\|_p \leq \epsilon_p, \tag{BPDQ$_p$}
\end{equation}
where $\epsilon_p$ must be carefully selected in order for $x$ to be a
feasible point of this new $\ell_p$-constraint. If $\epsilon_p\to \ensuremath{\Delta}$
as $p\to\infty$, the BPDQ$_p$ solution $\hat x_p$ tends to be
consistent with the quantized measurements. But what is the price to
pay, \emph{e.g.},  in terms of number of measurements, for being allowed to
increase $p$ beyond $2$?

To answer this, we need a variant of the restricted isometry property.
\begin{definition}
  Given two normed spaces $\mathcal{X}=(\mathbb R^m,\|\cdot\|_{\mathcal X})$
  and $\mathcal{Y}=(\mathbb R^n,\|\cdot\|_{\mathcal Y})$ (with $m< n$), a matrix
  $A\in\mathbb R^{m\times n}$ has the Restricted Isometry
  Property from $\mathcal{X}$ to $\mathcal{Y}$ at order
  $k\in\mathbb N$, radius $0\leq\delta<1$ and for a normalization
  $\mu>0$, if for all $x\in\Sigma_k := \{ u \in \mathbb R^N: \|u\|_0 \leq k\}$,
\begin{equation}
  \label{eq:rip-p}
  (1-\delta)^{1/\kappa}\,\|x\|_{\mathcal{Y}} \leq \tinv{\mu} \|A x\|_{\mathcal{X}} \leq
  (1+\delta)^{1/\kappa}\,\|x\|_{\mathcal{Y}},
\end{equation}
the exponent $\kappa$ depending on the spaces $\mathcal X$ and $\mathcal Y$. To lighten notation, we write that
$A$ is RIP$_{\mathcal{X},\mathcal{Y}}(k,\delta,\mu)$.
\end{definition}

In this general definition, the common RIP is equivalent to
RIP$_{\ell_2^m,\ell_2^n}(k,\delta,1)$ with $\kappa=2$ (see Chap. 1, Eq.
(1.10)). Moreover, the RIP$_{p,k,\delta'}$ defined in \cite{Berinde2008}
is equivalent to the RIP$_{\ell_p^m,\ell_p^n}(k,\delta,\mu)$ with
$\kappa=1$, $\delta'=2\delta/(1-\delta)$ and $\mu=1/(1-\delta)$. Finally, the
Restricted $p$-Isometry Property proposed in
\cite{chartrand2008restricted} is also equivalent to the
RIP$_{\ell_p^m,\ell_2^n}(k,\delta,1)$ with $\kappa=p$.

To characterize the stability of BPDQ we consider the space $\mathcal X =
\ell_p^m := (\mathbb R^m, \|\cdot\|_p)$ and $\mathcal Y = \ell_2^n := (\mathbb R^m,
\|\cdot\|_2)$ with $\kappa=1$, and we write RIP$_{p}$ as a shorthand
for RIP$_{\ell_p^m,\ell_2^n}$. At first sight, it could seem
  unnatural to define an embedding of $\mathcal X=\ell^n_p$ in $\mathcal Y =\ell^m_2$ for $p \neq 2$,
those spaces being not isometrically isomorphic to each other for $m = n$. However,
the RIP$_p$ rather sustains the possibility of an isometry between
$\mathcal X
\cap A\Sigma_k$ and $\mathcal Y\cap\Sigma_k$. We will see in
Prop.~\ref{prop:grip-gauss} that the existence of such a relation comes with
an exponential growth of $m$ as $p$ increases, a phenomenon that can
be related to Dvoretsky's theorem when specialized to those Banach
spaces \cite{ledoux}.

From this new characterization,
one can prove the following result.
\begin{theorem}[\cite{Jacques2010,Jacques2013}]
  \label{prop:l2-l1-instance-optimality-BPDQ}
  Let $k\geq 0$, $2\leq p <\infty$ and $A\in \mathbb R^{m\times n}$ be a RIP$_{p}(s,\delta_s,\mu_p)$ matrix for
  $s\in\{k,2k,3k\}$ and some normalization constant $\mu_p>0$. If
  \begin{equation}
    \label{eq:cond-on-delta-p}
    \delta_{2k}+\sqrt{(1+\delta_k)(\delta_{2k}+\delta_{3k})(p - 1)} < 1/3,
  \end{equation}
then, for any signal $x\in\mathbb R^n$ observed according to the noisy
sensing model $y=A x + n$ with $\|n\|_{p}\leq
  \epsilon_p$, the unique solution $\hat x_p$  obeys
\begin{equation}
\label{eq:BPDQ-l2-l1-inst_opt}
\|x^* - x\| \ \leq\ 4\,\tinv{\sqrt k}\,\sigma_k(x)_1\ +\ 8\,\epsilon_p/\mu_p,
\end{equation}
where, again, $\sigma_k(x)_1$ denotes the best $k$-term approximation.
\end{theorem}
This theorem follows by generalizing the fundamental result of
Cand\`es in \cite{candes2008rip} to the particular geometry of Banach
spaces $\ell_p^m$. It shows that, if $A$ is RIP$_p$ with particular
requirement on the RIP$_p$ constant, the BPDQ$_p$ program is stable
under both measurement noise corruption and departure from the strict
sparsity model, as measured by $e_0$. In particular, under the same
conditions, given a measurement noise $\xi$ and some upper bounds
$\epsilon_p$ on its $\ell_p$-norm, \eqref{eq:BPDQ-l2-l1-inst_opt}
provides the freedom to find the value of $p$ that minimizes
$\epsilon_p/\mu_p$.

This is exactly how QCS signal recovery works. Following Theorem
\ref{prop:l2-l1-instance-optimality-BPDQ} and its stability result
\eqref{eq:BPDQ-l2-l1-inst_opt}, we jointly determine a RIP$_p$ sensing
matrix with known value $\mu_p$ and a tight error bound $\epsilon_p$
on the $\ell_p$ norm of the residual $q-Ax$ under HRA. The existence
of a RIP$_p$ matrix is guaranteed by the following
result~\cite{Jacques2010,Jacques2013}.
\begin{proposition}[\bf RIP$_{p}$ Matrix Existence]
\label{prop:grip-gauss}
Let a random Gaussian sensing matrix $A\in \mathbb R^{m\times n}$ be such
that $a_{ij} \sim_{\rm iid} \mathcal N(0,1)$, $p\geq 1$ and $0\leq \eta <
1$. Then, $A$ is RIP$_{p}(k,\delta_k,\mu_p)$ with probability higher than
$1-\eta$ when we have jointly $m\geq 2^{p+1}$ and
\begin{align}
\label{eq:SGR-RIP-measur-bound}
m \geq
m_0^{\max(p/2,1)}\quad \text{with}\ m_0 = O(\delta_k^{-2}\,
\big(k\log(\tfrac{n}{k}) + k \log(\delta_k^{-1}) + \log\tfrac{2}{\eta}\big)).
\end{align}
Moreover, $\mu_p = \Theta(m^{1/p} \sqrt{p+1})$.
\end{proposition}
There is thus an exponential price to pay
for a matrix $A$ to be RIP$_p$ as $p$ increases: roughly speaking, for
$p\geq 2$, we
need $m \geq m^{p/2}_0 = O( k^{p/2} \log^{p/2} (n/k))$ measurements for
satisfying this property with non-zero probability.

To estimate a tight value of $\epsilon_p$ in the case of quantization
noise---since, under HRA $\xi_j \sim_{\rm iid} \mathcal
U([-\ensuremath{\Delta}/2,\ensuremath{\Delta}/2])$---we can show that
\begin{equation}
\label{eq:epsilon-p-def}
\|\xi\|_p \leq \epsilon_p\ :=\
\tfrac{\ensuremath{\Delta}}{2\,(p+1)^{1/p}}\,
\big(\,m + \zeta\,(p+1)\,\sqrt{m}\,\big)^{\inv{p}},
\end{equation}
with probability higher than $1-e^{-2\zeta^2}$. Actually, for $\zeta =
2$, $x$ is a feasible solution of the
BPDQ$_p$ fidelity constraint with a probability exceeding $1-e^{-8} > 1-
3.4 \times 10^{-4}$.

Finally, combining the estimation $\epsilon_p$ with the bound on
$\mu_p$, we find, under the conditions of Prop.~\ref{prop:grip-gauss},
\begin{equation}
  \label{eq:unif-error-decay}
  \tfrac{\epsilon_p}{\mu_p} \lesssim \tfrac{\ensuremath{\Delta}}{\sqrt{p+1}}.
\end{equation}
This shows that, in the high \emph{oversampled sensing scenario}
driven by \eqref{eq:SGR-RIP-measur-bound}, and provided the RIP$_p$ constants $\{\delta_k,\delta_{2k},\delta_{3k}\}$ satisfy
\eqref{eq:cond-on-delta-p}, the part of the reconstruction error due to
quantization noise behaves as $O(\ensuremath{\Delta}/\sqrt{p+1})$.
This is also the
error we get if $x$ is exactly $k$-sparse since then $e_0$ vanishes in \eqref{eq:BPDQ-l2-l1-inst_opt}.

If we solve for $p$, we can see that the error decays as $O(\ensuremath{\Delta}/\sqrt{\log m})$
as $m$ increases. There is possibly some room
for improvements since, as explained in
Sec.~\ref{sec:scal-quant-overs}, the lower bound on reconstruction of
sparse signal is $\Omega(\ensuremath{\Delta}/m)$.
Beyond scalar quantization schemes,
Sec.~\ref{sec:SDCS} will also show that much better theoretical error
reduction can be expected using \ensuremath{\Sigma\Delta}\xspace quantization.

\begin{figure*}[t]
  \centering
  \null\hfill
  \subfigure[\label{fig:first-exper-qual}]
  {\raisebox{-3mm}{\includegraphics[width=.35\textwidth]{examp1d_snr_mean}}}
  \hfill
  \subfigure[\label{fig:nrh-1}]
  {\includegraphics[width=.25\textwidth]{yerr-hist-m-over-k40-p02}}
  \hfill
  \subfigure[\label{fig:nrh-2}]
  {\includegraphics[width=.25\textwidth]{yerr-hist-m-over-k40-p10}}
  \hfill\null
  \caption{(a) Quality of BPDQ$_p$ for different $m/k$ and
    $p$. (b) and (c): Histograms of $\ensuremath{\Delta}^{-1}(A \hat{x}-q)_i$ for $p=2$ and
  for $p=10$, respectively. \label{fig:first-exper}}
\end{figure*}

Interestingly, we can, however, observe a numerical gain
in using BPDQ$_p$ for increasing values of $p$ when the signal $x$ is observed by the model
\eqref{eq:noisy-unfi-QCS} and when $m$ increases beyond the minimal
value $m_0$ needed for stabilizing BPDN (\emph{i.e.},  BPDQ$_2$).

This gain is depicted in Fig.~\ref{fig:first-exper}. The plots on the
left correspond to the reconstruction quality, \emph{i.e.},  the value ${\rm
  SNR} = 20 \log(\|x\|/\|x - \hat x_p\|)$ expressed in dB, reached by
BPDQ$_p$ for different values of $p$ and $m/k$. The original signal
$x$ has dimension $n=1024$ and is $k$-sparse in the canonical basis,
with support of size $k=16$ uniformly random and normally distributed
non-zero coefficients. Each point of each curve represents average
quality over 500 trials.  For each sparse signal $x$, $m$ quantized
measurements were recorded using \eqref{eq:noisy-unfi-QCS} with a
random Gaussian sensing matrix $A$ and $\ensuremath{\Delta} = \|A x\|_\infty/40$. The
reconstruction was done by solving BPDQ$_p$ with the Douglas-Rachford
algorithms \cite{Jacques2010}, an efficient convex optimization method
solving constrained programs, such as BPDQ\footnote{The code of BPDQ
  is freely available at \url{http://wiki.epfl.ch/bpdq}.}, using
simpler \emph{proximal operators} \cite{combettes2011proximal}.
Fig.~\ref{fig:first-exper-qual} shows that higher oversampling ratio
$m/k$ allows the use of higher $p$ with significant gain in the
reconstruction quality. However, if $m/k$ is low, \emph{i.e.},  close to
$m/k=10$, the best quality is still reached by BPDN. The quantization
consistency of the reconstruction, \emph{i.e.},  the original motivation for
introducing the BPDQ$_p$ program, can also be tested. This is shown on
Fig.~\ref{fig:nrh-1} and Fig.~\ref{fig:nrh-2} where the histograms of
the components of $\ensuremath{\Delta}^{-1}(A \hat x_p - q)$ are represented for $p=2$
and $p=10$ at $m/k=40$. This histogram for $p=10$ is indeed closer to
a uniform distribution over $[-1/2,1/2]$, while the one at $p=2$ is
mainly Gaussian.
\section{Sigma-Delta Quantization for Compressive Sensing}\label{sec:SDCS}

As mentioned in the introduction, $\ensuremath{\Sigma\Delta}\xspace$ quantization for compressed
sensing fundamentally builds on corresponding schemes for finite
frames. Thus before presenting an analysis specific to compressed
sensing, we first discuss the finite frame case.

\subsection{$\ensuremath{\Sigma\Delta}\xspace$  Quantization for Frames}\label{subsec:frame}
Let $\Phi \in \mathbb{R}^{n\times N}$  with columns $\{ \phi_j\}_{j=1}^N$ be a frame in the sense of (1.32) and consider the frame expansion $$c=\Phi^T x$$ of a signal $x\in {\mathbb R}^n$. The goal is now to quantize  $c$ as a whole such that the quantized representation $q$ allows for approximate recovery of $x$. $\ensuremath{\Sigma\Delta}\xspace$ quantization schemes obtain such a $q$ using a recursive procedure, which we will now explain in detail.

At the core of the schemes is a uniform scalar quantizer $\ensuremath{Q}$, which maps a real number to the closest point in a codebook of the form
\begin{equation}\ensuremath{\mathcal{Q}} = \big\{ (\pm j-1/2)\ensuremath{\Delta}, j \in \{ 1, ... ,  L\}\big\}.\label{eq:SD_codebook}\end{equation}

A $\ensuremath{\Sigma\Delta}\xspace$ scheme applies such a quantizer sequentially to the entries of $c$, taking in each quantization step the errors made in $r$ previous steps into account. The complexity parameter $r$ is referred to as the order of the $\ensuremath{\Sigma\Delta}\xspace$ scheme; it quantifies the trade-off between required storage and achievable accuracy.

A first order $\ensuremath{\Sigma\Delta}\xspace$ quantization scheme, the simplest such algorithm, hence retains the error only for one step.   In the following formalization associated with the so-called \emph{greedy} first order $\ensuremath{\Sigma\Delta}\xspace$ scheme, the error parameter appears as the state variable $u_i$; it measures the total accumulated error up to step $i$.
The quantized frame coefficient vector $q\in \ensuremath{\mathcal{Q}}^N$ is computed by running the iteration
\begin{align}
q_i &= \ensuremath{Q} (u_{i-1} + c_i) \nonumber \\
u_i &= u_{i-1} + c_i - q_i. \label{eq:1st_order}
\end{align}
As initialization, one typically uses $u_0=0$. In matrix-vector notation, the above recurrence relation reads \begin{equation}D u = c - q.\label{eq:mat_form} \end{equation} Here $D\in \mathbb{R}^{N\times N}$ is the finite difference matrix with entries given in terms of the Kronecker delta by $D_{ij} = \delta_{i,j} -\delta_{i+1,j}$, that is,
\begin{equation}
D = \begin{pmatrix}1&0  & 0 & \cdots & 0\\-1 &1 &0 & & 0\\ 0& -1 & 1 & & 0\\ \vdots& & \ddots&\ddots &\vdots
\\ 0& 0 &\cdots &-1 &1
\end{pmatrix}.
\end{equation}

The scheme is explicitly designed such that each $q_j$ partly cancels the error made up to $q_{j-1}$. When the signal is approximated as $\widetilde{\Phi} q$ using a dual frame $\widetilde \Phi \in \mathbb{R}^{n\times N}$  with columns $\{ \widetilde \phi_j\}_{j=1}^N$, this entails that one seeks to compensate an error in the direction of a dual frame vector $\widetilde \phi_{j-1}$ using a distortion in the direction of the next dual frame vector $\widetilde \phi_{j}$. This serves as a motivation to choose a smoothly varying dual frame, \emph{i.e.},  with subsequent dual frame vectors close to each other.

Bounding the reconstruction error using \eqref{eq:mat_form} in terms of the operator norm $\|A\|_{2\rightarrow 2}:= \sup_{\|x\|_2\leq 1} \|Ax\|_2$, one obtains \[ \| x- \widetilde{\Phi}q \|_2 =  \|\widetilde{\Phi}(c-q)\|_2 = \|\widetilde{\Phi}D u \|_2 \leq \|\widetilde{\Phi} D \|_{2\rightarrow 2} \|u\|_2. \]  The smoothness intuition is reflected in the fact that the columns of $\widetilde{\Phi} D$ are given by $\widetilde{\phi}_j - \widetilde{\phi}_{j-1}. $  Thus more precisely, finding a smooth dual frame $\widetilde \Phi$ amounts to minimizing $\|\widetilde \Phi D\|_{2\rightarrow 2}$.

If one is willing to store more than one previous value of the state variable, that is, to consider a higher order $\ensuremath{\Sigma\Delta}\xspace$ scheme, it is possible to profit from higher order smoothness of the dual frame. Such a generalization of \eqref{eq:1st_order} is the greedy $r$-th order $\ensuremath{\Sigma\Delta}\xspace$ scheme, which is associated with the recurrence relation
\begin{equation}D^r u = c - q.\label{eq:mat_form_r} \end{equation}
Here, the iteration to compute the quantized coefficients is explicitly given by
\begin{align}
q_i &\textstyle = \ensuremath{Q} \big(\sum\limits_{j=1}^{r}(-1)^{j-1} {{r}\choose{j}}u_{i-j} + c_i\big) \nonumber \\
u_i &\textstyle = \sum\limits_{j=1}^{r}(-1)^{j-1} {{r}\choose{j}}u_{i-j} + c_i - q_i. \label{eq:r-th_order}
\end{align}
As before, one initializes $u_i=0, \ i\leq 0$. The reconstruction
error is now bounded by
\begin{equation} \| x- \widetilde{\Phi}q \|_2 =  \|\widetilde{\Phi}(c-q)\|_2 = \|\widetilde{\Phi}D^r u \|_2 \leq \|\widetilde{\Phi} D^r \|_{2\rightarrow 2} \|u\|_2. \label{eq:errbound}
\end{equation}
Examining \eqref{eq:errbound}, it is advantageous to choose a dual frame that minimizes $\|\widetilde{\Phi}D^r \|_{2\rightarrow2}$, and a $\ensuremath{\Sigma\Delta}\xspace$ scheme that yields a state-variable sequence with well bounded $\|u\|_2$. This motivates the following definitions.
\begin{definition}
Let $\Phi \in \mathbb{R}^{n\times N}$ be a frame and $r$ be a positive integer. Then the $r$-th order  \emph{Sobolev dual} of $\Phi$ is given by
\begin{equation}\widetilde\Phi^{(r)}:= \arg\min \| \widetilde{\Phi}D^r\|_{2 \rightarrow 2} = (D^{-r} \Phi)^\dagger D^{-r} \label{eq:Sobolev},\end{equation}
where the minimum is taken over all dual frames of $\Phi$.
\end{definition}

\begin{definition}
A $\ensuremath{\Sigma\Delta}\xspace$ scheme with a codebook $\ensuremath{\mathcal{Q}}$ is \emph{stable} if there exist constants $C_1$ and $C_2$ such that whenever $\|c\|_\infty \leq C_1$ we have $\|u\|_\infty \leq C_2.$
\end{definition}

In general, designing and proving the stability of $\ensuremath{\Sigma\Delta}\xspace$ quantization schemes of arbitrary order can be quite difficult if the number of elements in the associated codebook is held fixed. This challenge is especially difficult in the case of 1-bit quantizers and overcoming it is the core of the contributions of \cite{DD03, G03, DGK11}, where stable $\ensuremath{\Sigma\Delta}\xspace$ quantization schemes of arbitrary order are designed. On the other hand, if the number of elements in the codebook \eqref{eq:SD_codebook} is allowed to increase with order, then even the simple greedy $\ensuremath{\Sigma\Delta}\xspace$ schemes \eqref{eq:r-th_order} are stable, as the following proposition shows (see, \emph{e.g.},  \cite{BPA07}).

\begin{proposition}
The greedy $r$-th order $\ensuremath{\Sigma\Delta}\xspace$ scheme \eqref{eq:r-th_order} associated with the $2L$-level scalar quantizer \eqref{eq:SD_codebook} is stable, with $\|u\|_\infty \leq \ensuremath{\Delta}/2$, whenever $\|c\|_\infty \leq  \ensuremath{\Delta} (\ensuremath{L} - 2^{r-1}+ 2^{-1})$.
\end{proposition}
\begin{proof}
The proof is by induction. We begin by rewriting \eqref{eq:r-th_order} in terms of auxiliary state variables  $u_i^{(j)}, j=1,...,r$ and $u_i^{(0)}=c_i -q_i$ as
\begin{align}
q_i &= \ensuremath{Q} \Big(\sum_{j=1}^{r}u^{(j)}_{i-1} + c_i\Big) \nonumber \\
u_i^{(j)} &= u_{i-1}^{(j)}+u_{i}^{(j-1)}, \quad j=1,...,r \label{eq:r-th_order2}
\end{align}
with $u^{(j)}_0=0$ for $j=1,...,r$. Note that with this notation $u_i^{(r)}=u_i$. Now suppose that $|u^{(j)}_{i-1}| \leq 2^{r-j} \ensuremath{\Delta}/2$ for all $j \in \{1,...,r\}$, then $|\sum_{j=1}^r u^{(j)}_{i-1}| \leq (2^r-1) \ensuremath{\Delta}/2$.
Since by the $\ensuremath{\Sigma\Delta}\xspace$ iterations we have $u^{(j)}_{i} = \sum_{k=1}^j u^{(k)}_{i-1} + c_i - q_i$ we deduce that \[ |u_i^{(r)}| = | \sum_{k=1}^r u^{(k)}_{i-1} + c_i - \ensuremath{Q}(\sum_{k=1}^r u^{(k)}_{i-1} + c_i)|  \leq \ensuremath{\Delta}/2\]
provided  $\|c\|_\infty \leq \ensuremath{\Delta} (\ensuremath{L} - 2^{r-1}+ 1/2)$. Moreover, by \eqref{eq:r-th_order2}, $|u^{(j)}_{i}| \leq 2^{r-j}\Delta/2$.
 \end{proof}

 Working with stable $r$-th order $\ensuremath{\Sigma\Delta}\xspace$ schemes and frames with
 smoothness properties, and employing the Sobolev dual for
 reconstruction, it was shown in \cite{BLPY10} that the reconstruction
 error satisfies $\| x - \widetilde{\Phi}q\|_2 \leq C_{r,\Phi}
 N^{-r}$, where the constant $C_{r,\Phi}$ depends only on the quantization scheme and the frame. Such results for $\ensuremath{\Sigma\Delta}\xspace$-quantization show that its error
 decay rate breaks the theoretical $\sim 1/N$ lower bound of scalar
 quantization described in the introduction.  Fig.~\ref{fig:quant_cells} helps to illustrate why such a result is
 possible. It shows the quantization cells associated with two bit
 quantization of $\Phi^T x$, where $x$ in the unit ball of
 $\mathbb{R}^2$, using both first order $\ensuremath{\Sigma\Delta}\xspace$ quantization and scalar
 quantization. For vectors belonging to a given cell, the worst case
 error achieved by an optimal decoder is proportional to the diameter
 of the cell.  The figure shows that the cells resulting from $\ensuremath{\Sigma\Delta}\xspace$
 quantization are smaller than those resulting from scalar
 quantization, indicating the potential for a smaller reconstruction
 error. For a detailed overview of $\ensuremath{\Sigma\Delta}\xspace$ quantization of frame
 expansions see, \emph{e.g.},  \cite{PSY13}.

\begin{figure}[t]
  \centering
   \begin{tabular}{cc}
  \includegraphics[width=.5\linewidth]{SD_2bit_randn15}&
  \includegraphics[width=.5\linewidth]{MSQ_2bit_randn15}
  \end{tabular}
  \vspace{-40pt}
  \caption{The first order $\ensuremath{\Sigma\Delta}\xspace$ (left) and scalar quantization (right) cells associated with 2-bit quantization of $\Phi^T x$ where $x$ is in the unit ball of $\mathbb{R}^2$ and $\Phi$ is a $2\times 15$ Gaussian random matrix.}
  \label{fig:quant_cells}
  \end{figure}

The existing recovery algorithms for $\ensuremath{\Sigma\Delta}\xspace$ quantized compressed sensing measurements rely on a two stage algorithm. In the first stage, the signal support is recovered and in the second stage, the signal coefficients are estimated using the Sobolev dual of the frame associated with the recovered support.

\subsection{Finding the Signal Support}\label{subsec:supp}
Let $q\in \ensuremath{\mathcal{Q}}^m$ be the $r$-th order $\ensuremath{\Sigma\Delta}\xspace$ quantization of the compressed sensing measurements $y=Ax \in \mathbb{R}^m$ associated with the sparse vector $x\in\Sigma_k$ and the measurement matrix $A\in \mathbb{R}^{m\times n}$. In order to preserve the codebook definition \eqref{eq:SD_codebook}, we assume as in Sec.~\ref{sec:scalarCS} that the scaling of the entries of $A$ is independent of $m$.

The goal of the first stage of the reconstruction algorithm is to recover  $T:= supp(x)$. To that end, following \cite{GLPSY13} we will use a (standard) compressed sensing decoder $\ensuremath{\mathcal D}: \mathbb{R}^m \to \mathbb{R}^N$ that has uniform robustness guarantees for matrices with an appropriate RIP constant. For such a decoder and an arbitrary scalar $\kappa$
\begin{equation}\label{eq:CS_guarantee}
x\in\Sigma_k \text{ and } \gamma\in\mathbb R^m: \|\gamma\|_2 \leq \kappa\sqrt{m} \quad \implies \quad \|\ensuremath{\mathcal D}(A x + \gamma) - x \|_2 \leq C\kappa.
\end{equation}
For example, if $\ensuremath{\mathcal D}(Ax+\gamma)$ is the output of an $\ell_1$-minimization algorithm such as Basis Pursuit DeNoising (BPDN), it satisfies \eqref{eq:CS_guarantee} with constant $C:=C(\delta, k)$ when the matrix $A$ (more precisely ${A}/{\sqrt{m}}$) satisfies an appropriate restricted isometry property \cite{candes2008rip}. As the next proposition shows, robust decoders allow recovering the support of a sparse vector when its smallest non-zero entry is above the error level.

\begin{proposition}\label{prop:support_recovery}Let $\ensuremath{\mathcal D}$ be a compressed sensing decoder satisfying \eqref{eq:CS_guarantee} and let $x\in \Sigma_k$ with $T:=supp(x)$. Define $\hat{x}:=\ensuremath{\mathcal D}(A x + \gamma)$.  If $\min_{i\in T} |x_i| > 2C\kappa$ then the largest $k$ coefficients of $\hat{x}$ are supported on $T$.
\end{proposition}
\begin{proof}First, note that
for all $i\in T$,  \eqref{eq:CS_guarantee} yields $|\hat{x}_i - x_i| \leq C\kappa$.  Since $\min_{i\in T} |x_i| > 2C\kappa$, the reverse triangle inequality gives $|\hat{x}_i| > C\kappa$ for all $i$ in $T$. On the other hand, \eqref{eq:CS_guarantee} also ensures that $|\hat{x}_i| \leq C\kappa$ for all $i \in T^c$.

 A sharper version of this argument appears in \cite{GLPSY13} but Proposition \ref{prop:support_recovery} is sufficient for our purposes. In particular, consider an $r$th order greedy $\ensuremath{\Sigma\Delta}\xspace$ quantization associated with  a codebook $\ensuremath{\mathcal{Q}}$ having $2L$ elements.  Applying such a scheme to $Ax$ yields a quantized vector $q$ satisfying $\|q-Ax \|_2 \leq \frac{\Delta}{2} 2^r \sqrt{m}$ provided
\begin{equation}
 L>\|Ax \|_\infty/\Delta + 2^{r-1} - 1/2 \label{eq:number_of_levels}.
\end{equation}
 Thus assuming that $A/\sqrt{m}$ has appropriate RIP constants,  Proposition \ref{prop:support_recovery} shows that using a decoder satisfying \eqref{eq:CS_guarantee}, the support $T$ of $x\in\Sigma_k\subset \mathbb{R}^n$ can be accurately recovered provided $|z_i| > 2^rC~ \Delta$ for all $i \in T$. What remains is to choose the number of levels $L$ in the codebook to satisfy \eqref{eq:number_of_levels}; this in turn requires an estimate of $\|Ax\|_\infty$.

 To that end, we now consider subgaussian measurement matrices, \emph{i.e.},  matrices whose entries are subgaussian random variables as defined below.

 \begin{definition}
Let $\xi$ be a Gaussian random variable drawn according to $\mathcal{N}(0,\sigma^2)$. If a random variable $\eta$ satisfies $P(|\eta|>t) \leq e P(|\xi|>t)$ for all $t$, then we say $\eta$ is subgaussian with parameter $\sigma>0$.
\end{definition}

Examples of subgaussian random variables include Gaussian, Bernoulli, and bounded random variables, as well as their linear combinations. For matrices populated with such subgaussian entries, the following proposition from \cite{KSY13} gives a bound on $\|Ax\|_\infty$ when the non-zero entries of $x$ are restricted to a fixed support $T$ so that $Ax = \Phi^Tx_T$ for a frame $\Phi$ associated with the support.

 \begin{proposition}\label{prop:infty_norm_bound}
 Let $\widehat{\Phi}$ be a $k \times m$ subgaussian matrix with mean zero, unit variance, and parameter $\sigma$, where $k<m$. Let $\Phi= \frac{1}{\sqrt{m}}\widehat{\Phi}$ and fix $\alpha\in (0,1)$. Then, with probability at least $1-e^{-\frac{1}{4} m^{1-\alpha}k^\alpha}$, we have  for all $m> C^{\frac{1}{1-\alpha}} k$ and $x \in \mathbb{R}^k$
\begin{equation}\|\Phi^Tx\|_{\infty} \leq  e^{1/2}\big(\tfrac{m}{k}\big)^{-\frac{\alpha}{2}}\|x\|_2.\label{eq:infty_norm_bound}\end{equation}
Here $C$ is a constant that may depend on $\sigma$, but is independent of $k$ and $\alpha$.
\end{proposition}
Taking a union bound over all the ${n\choose k}$ submatrices of $A$ of size $m\times k$ yields an identical uniform bound on $\|Ax\|_\infty$, which holds for sparse vectors $x$ with high probability, provided $m>C k (\log n)^{\frac{1}{1-\alpha}}$.

Thus an $r$-th order greedy $\ensuremath{\Sigma\Delta}\xspace$ scheme with sufficiently many quantization levels allows the recovery of a sparse signal's support from its compressed sensing measurements. Equipped with this knowledge, we can estimate the signal coefficients using the Sobolev dual of the frame associated with the recovered support.

\end{proof}

\subsection{Recovering the Signal Coefficients}\label{subsec:coeff}

\newcommand{S}{S}

We continue to consider Gaussian or subgaussian measurement matrices, now assuming that the support $T$ of the signal $x$ has been identified. Our goal is to approximate the coefficients $x_i$, $i\in T$. With high probability, the matrix $A/\sqrt{m}$ has the restricted isometry property of order $2k$ and level $\delta_{2k}\leq 1/\sqrt{2}$ provided one takes at least on the order of $k\log(n/k)$ measurements. Then the matrix $A_T/\sqrt{m}$ restricted to the columns indexed by $T$ is close to an isometry and its rows hence form a frame. Consequently, the measurement vector is the associated frame expansion of $x_T$, and $q$ is the corresponding $\ensuremath{\Sigma\Delta}\xspace$ frame quantization.

As shown in Sec.~\ref{subsec:frame}, it is advantageous to reconstruct $x$ from the $r$-th order $\ensuremath{\Sigma\Delta}\xspace$ quantization $q$ of the measurement vector $Ax$ using the Sobolev dual $\widetilde A_T^{(r)}$ of $A_T$, see \eqref{eq:errbound} and \eqref{eq:Sobolev}.  A possible bound for the reconstruction error is then proportional to $\|\widetilde A_T^{(r)}D^r\|_{2\rightarrow 2}$. Thus to show a uniform recovery guarantee, one needs a bound for this quantity which is uniform over all potential support sets $T$. In the initial work \cite{GLPSY13}, dealing with Gaussian compressed sensing matrices, the approach to proving such a bound consisted of explicitly controlling the lowest singular value of $D^{-r}A_T$. Their approach utilized the unitary invariance of the Gaussian measure to identify the distribution of the singular values of the random matrix $D^{-r}A_T$ with those of $S_{D^{-r}}\Psi$, where $S_{D^{-r}}$ is a diagonal matrix whose entries are the singular values of $D^{-r}$, and $\Psi$ is a Gaussian matrix. This,
coupled with bounds on the singular values of $D^{-r}$,
allowed \cite{GLPSY13} to derive bounds that held with probability high enough to survive a union bound over all ${n}\choose{k}$ Gaussian submatrices of $A$. In \cite{KSY13}, this approach was extended to subgaussian matrices.  Herein, to prove such a bound on $\|\widetilde A_T^{(r)}D^r\|_{2\rightarrow 2}$, we follow the simpler, RIP-based approach presented in \cite{FK13}.

To that end, let $E=U_{E} S_{E} V_{E}^T$ be the singular value decomposition (SVD) of any matrix $E$ (for some orthogonal matrices $U_{E}$ and $V_{E}$) where the matrix  $S_{E}$ is diagonal with (ordered) diagonal entries $\sigma_j(E)$. We denote also $\sigma_{\min}(E) := \sigma_1(E)$ the smallest singular value of $E$.
Then the following proposition (see, \emph{e.g.},  \cite{GLPSY13}) holds.
\begin{proposition}
There are positive constants $C_1(r)$ and $C_{2}(r)$, independent of $m$, such that
\begin{equation}\label{D^-r}
C_1(r)(\tfrac{m}{j})^r\leq \sigma_j(D^{-r})\leq C_{2}(r)(\tfrac{m}{j})^{r},\ j=1,\ldots,m.
\end{equation}
\end{proposition}

Denote by $P_\ell$ the $\ell\times m$ matrix that maps a vector to its
first $\ell$ components. Moreover, denote by
$\widetilde\Sigma_k(A,\ensuremath{\mathcal D}) \subset \Sigma_k$ the set of
$k$-sparse signals $x$ whose support can be recovered from $q$ with
the decoder $\ensuremath{\mathcal D}$ as in Proposition
\ref{prop:support_recovery}. The following theorem describes the
reconstruction performance.

 \begin{theorem}[\cite{FK13}]\label{thm:SD_RIP}
Let $A\in \mathbb{R}^{m\times n}$ be a matrix such that for a fixed $\ell\leq m$, the $\ell \times n$ matrix $\frac{1}{\sqrt\ell}P_\ell V^T_{D^{-r}}A$ has restricted isometry constant $\delta_k \leq \delta$. Then the following holds uniformly for all $x\in \widetilde\Sigma_k(A,\ensuremath{\mathcal D})$.

If $x$ has support $T$, $q$ is the $r$-th order $\ensuremath{\Sigma\Delta}\xspace$ quantization of $Ax$, and $\hat{x} := \widetilde{A}_{T}^{(r)}q$, then
$$
\|x-\hat{x}\|_2 \leq \tfrac{\ensuremath{\Delta}}{C(r)\sqrt{(1-\delta)}}(\tfrac{m}{\ell})^{-r+\frac{1}{2}},
$$
where $C(r)>0$ is a constant depending only on r and $\ensuremath{\Delta}$ is the quantization step size.
\end{theorem}
 \begin{proof}
As the SVD of $D^{-r}$ provides $D^{-r}=U_{D^{-r}}S_{D^{-r}}V_{D^{-r}}^T$, the smallest singular value of $D^{-r}A_T$ satisfies
 \begin{align*}
 \sigma_{\min}(D^{-r}{A_T})&=\sigma_{\min}(S_{D^{-r}}V^T_{D^{-r}}{A_T})\\
 &\geq \sigma_{\min}(P_\ell S_{D^{-r}}V^T_{D^{-r}}{A_T})\\
 &= \sigma_{\min}((P_{\ell}S_{D^{-r}}P^T_{\ell} )(P_\ell V^T_{D^{-r}}{A_T}))\\
 &\geq \sigma_\ell(D^{-r}) \sigma_{\min}(P_\ell V^T_{D^{-r}}{A_T}),
 \end{align*}
To bound $\sigma_{\min}(P_\ell V^T_{D^{-r}}{A_T})$ uniformly over all support sets $T$ of size $k$  we simply note that if $\frac{1}{\sqrt{\ell}}P_\ell V^T_{D^{-r}}{\Phi}$ has restricted isometry constant $\delta_k\leq\delta$
 then $\sigma_{\min}(P_\ell V^T_{D^{-r}}{A_T})$ is uniformly bounded from below by
 \begin{equation}\label{subRIP}
 \sqrt{\ell}\sqrt{1-\delta}.
 \end{equation}
 The theorem follows by applying (\ref{eq:errbound}), (\ref{D^-r}), (\ref{subRIP}) as
\begin{equation}
 \tfrac{1}{\sigma_{min}(D^{-r}{A_T})}\|u\|_2\leq  \tfrac{\Delta}{C(r)\sqrt{(1-\delta)}}(\tfrac{m}{\ell})^{-r+\frac{1}{2}}
\end{equation}\qed
\end{proof}

The above theorem can be applied almost directly to Gaussian compressed sensing matrices. If $A$ is a Gaussian matrix with independent zero mean and unit variance entries, then by rotation invariance so is the matrix $P_\ell V^T_{D^{-r}}A$. Regarding the choice of $\ell$, note from Theorem \ref{thm:SD_RIP} that the smaller $\ell$ is, the better the bound. On the other hand $\ell$ has to be large enough for $\frac{1}{\sqrt{\ell}}(P_\ell V^T_{D^{-r}}\Phi)$ to have the restricted isometry constant $\delta_k\leq\delta$. This prompts the choice $\ell \asymp k\log n$, as then $\frac{1}{\sqrt{\ell}}(P_\ell V^T_{D^{-r}}\Phi)$ has the restricted isometry constant $\delta_k<\delta$ with high probability, as discussed in Chapter 1 (see, \emph{e.g.},  Theorem 1.5). In particular, if \[m\ \gtrsim\ k(\log n)^\frac{1}{1-\alpha}, \quad \alpha \in (0,1)\]
and \[ \ell \asymp k\log n\]
 then \[\tfrac{m}{\ell}\asymp \tfrac{m}{k\log n} = (\tfrac{m}{k})^\alpha\cdot \bigg(\tfrac{m}{k(\log n)^{\frac{1}{1-\alpha}}}\bigg)^{1-\alpha}\gtrsim (\tfrac{m}{k})^{\alpha}\]
Applying Theorem \ref{thm:SD_RIP} directly, we obtain
  \begin{align*}
\|x-\hat{x}\|_2&\lesssim\Delta(\tfrac{m}{k})^{-\alpha (r-\frac{1}{2})}.
\end{align*}
This essentially recovers the result in \cite{GLPSY13} and a similar, albeit more technical argument for subgaussian matrices, using either bounds on tail probabilities for quadratic forms \cite{HW71, rv13} or bounds for suprema of chaos processes \cite{krmera12} recovers the analogous result in \cite{KSY13}.

To illustrate the advantage of using $\ensuremath{\Sigma\Delta}\xspace$ schemes for quantizing compressed sensing measurements we conduct a numerical experiment with $k$-sparse signals in $\mathbb{R}^n$, as we vary the number of measurements $m$. We fix $k=10,$ $n=1000,$ and the quantization step-size $\Delta=0.01$. We draw  $m \times n$ Gaussian matrices $A$ for $m\in\{100, 200, 400, 800\}$ and quantize the measurements $Ax$ using scalar quantization and $r$th order $\ensuremath{\Sigma\Delta}\xspace$ schemes with $r=1,2,3$. We then use the two-stage reconstruction method described herein to obtain an approximation $\hat{x}$ of $x$ using its quantized measurements. Repeating this experiment 30 times, we compute the average of the reconstruction error $\|x-\hat{x}\|_2$ for each of the quantization methods and plot them against the oversampling ratio $m/k$ in Fig.~\ref{fig:SD_CS_results}.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.9\linewidth]{experiment_SD_results}
  \vspace{-10pt}
  \caption{Average errors over 30 experiments. The figure shows the reconstruction errors resulting from scalar quantization, using $\ell_1$-minimization for reconstruction (dashed line). Also corresponding to scalar quantization, the figure shows the errors resulting from reconstructing via the two-stage algorithm described herein (solid black line), using the canonical dual of the frame corresponding to the recovered support in the second stage. It also shows the reconstruction errors resulting from $1st$, $2nd$, and $3rd$ order $\ensuremath{\Sigma\Delta}\xspace$ quantization respectively. These errors decay as $(m/k)^{-r}$ for $r=1,2,3$ respectively, slightly outperforming the theoretical predictions presented here.}
  \label{fig:SD_CS_results}
  \end{figure}

In summary, using Gaussian and subgaussian compressed sensing matrices
recovery of sparse signals from their $\ensuremath{\Sigma\Delta}\xspace$ quantized measurements is
possible. More importantly, the reconstruction error decays
polynomially in the number of measurements and thus outperforms the
(at best) linear error decay that can be achieved with scalar
quantization. This improvement comes at the cost of introducing memory
elements, and feedback, into the quantization procedure.

Quantization and Compressive Sensing