Between hard and soft thresholding: optimal iterative thresholding algorithms

Abstract: Iterative thresholding algorithms seek to optimize a differentiable objective function over a sparsity or rank constraint by alternating between gradient steps that reduce the objective, and thresholding steps that enforce the constraint. This work examines the choice of the thresholding operator, and asks whether it is possible to achieve stronger guarantees than what is possible with hard thresholding. We develop the notion of relative concavity of a thresholding operator, a quantity that characterizes the worst-case convergence performance of any thresholding operator on the target optimization problem. Surprisingly, we find that commonly used thresholding operators, such as hard thresholding and soft thresholding, are suboptimal in terms of worst-case convergence guarantees. Instead, a general class of thresholding operators, lying between hard thresholding and soft thresholding, is shown to be optimal with the strongest possible convergence guarantee among all thresholding operators. Examples of this general class includes $\ell_q$ thresholding with appropriate choices of $q$, and a newly defined {\em reciprocal thresholding} operator. We also investigate the implications of the improved optimization guarantee in the statistical setting of sparse linear regression, and show that this new class of thresholding operators attain the optimal rate for computationally efficient estimators, matching the Lasso.

\section{Introduction}\label{sec:intro}

We consider the general problem of sparse optimization, where we seek to optimize a likelihood function or loss function subject to a sparsity constraint,
\[\min_{x\in\mathbb{R}^d, \norm{x}_0\leq s}\mathsf{f}(x).\]
Here $\mathsf{f}:\mathbb{R}^d\rightarrow \mathbb{R}$ is the target function that we would like to minimize, while the constraint $\norm{x}_0\leq s$ requires that the solution vector $x$ has at most $s$ many nonzero entries. Similarly, we may work with a matrix parameter $X\in\mathbb{R}^{n\times m}$ and search for a low-rank solution,
\[\min_{X\in\mathbb{R}^{n\times m},\textnormal{rank}(X)\leq s}\mathsf{f}(X).\]
Optimization problems over a sparsity constraint or a rank constraint are ubiquitous in high-dimensional statistics and machine learning. Sparsity of a vector parameter $x$ represents the idea that we can model the data using a small fraction of the available features, which, for instance, may correspond to covariates in a regression model or to basis expansion terms in a nonparametric function estimation problem. Similarly, a rank constraint on a matrix parameter $X$ might correspond to an underlying factor model with a small number of factors. We will focus on problems where $\mathsf{f}$ is a differentiable function, as is often the case for many likelihood models and other loss functions.

In this work, we will study the iterative thresholding approach, where gradient steps that lower the value of the target function $\mathsf{f}$ are alternated with thresholding steps to enforce the sparsity constraint---for instance, {\em hard thresholding} sets all but the largest $s$ entries to zero, while {\em soft thresholding} shrinks all values towards zero equally until the sparsity constraint is satisfied. (The same ideas apply to a rank constraint, by thresholding or shrinking singular values instead of vector entries. For simplicity, we will primarily discuss the sparse minimization problem, and will return to the low-rank problem later on.)

For sparse minimization of a differentiable target function $\mathsf{f}(x)$, many existing algorithms can be broadly described as iterating steps of the following form:
\begin{equation}\label{eqn:iter_intro}\begin{cases}
\text{Gradient step: $x'_t = x_{t-1} - \eta_t \cdot \nabla \mathsf{f}(x_{t-1})$ for some step size $\eta_t$},\\
\text{Sparsity step: $x_t$ = some sparse (or nearly sparse) approximation to $x'_t$}.
\end{cases}\end{equation}
Our aim in this work is to characterize the type of thresholding operators that are likely to be most successful at converging to a good solution, i.e.~to a value of $\mathsf{f}(x)$ that is as low as possible. Is an iterative thresholding algorithm most likely to succeed if we use hard thresholding, soft thresholding, or yet another form of thresholding to enforce the sparsity constraint?

In this work, we find that the worst-case performance of a thresholding operator, relative to a broad class of target functions $\mathsf{f}$ that we may want to minimize, is fully characterized by a simple measure that we call the {\em relative concavity}. The relative concavity studies the behavior of the sparse thresholding map $x'_t\mapsto x_t$ in the iterative algorithm~\eqref{eqn:iter_intro}, viewed as an approximate projection onto the space of $s$-sparse vectors. Using relative concavity as a tool to evaluate and compare different thresholding operators, we find that commonly used thresholding operators, for example hard thresholding and soft thresholding, are indeed suboptimal. Instead, we characterize a general class of thresholding operators, lying between hard thresholding and soft thresholding, that we show to be optimal. This class includes $\ell_q$ norm thresholding, where $q\in(0,1)$ is chosen adaptively relative to the particular problem; furthermore, choosing $q=2/3$ is ``universal'' in the sense that it is nearly optimal across all sparse thresholding problems. We also develop the {\em reciprocal thresholding} operator, which enjoys the same optimality guarantees as $\ell_q$ thresholding, but with a closed-form equation for the iterative thresholding step. These simple and efficient iterative thresholding methods are then applied to the statistical setting of sparse linear regression problem:
\begin{equation}
y=X\theta_0+z,
\end{equation}
and are shown to match the Lasso in terms of the resulting guarantee on estimating the true mean vector $X\theta_0$.

\section{Background: sparse minimization}

Before defining relative concavity and the reciprocal thresholding operator, we first review some of the recent literature on hard thresholding and related methods, and define the convexity and smoothness properties of the objective function $\mathsf{f}$ that we will assume throughout this work.

\subsection{Restricted strong convexity and restricted smoothness}
In many problems in high-dimensional statistics, we aim to optimize loss functions that may be very poorly conditioned in general, but nonetheless exhibit convergence properties of a well-conditioned function when working only with sparse or approximately sparse vectors. This behavior is captured in the notions of restricted strong convexity and restricted smoothness (see e.g.~\citet{negahban2009unified,loh2013regularized} for background).

A differentiable function $\mathsf{f}:\mathbb{R}^d\rightarrow\mathbb{R}$ satisfies {\em restricted strong convexity}
with parameter $\alpha$ at sparsity level $s$, abbreviated as $(\alpha,s)$-RSC, if
\[\mathsf{f}(y)\geq \mathsf{f}(x) + \inner{\nabla \mathsf{f}(x)}{y-x} + \frac{\alpha}{2}\norm{x-y}^2_2\text{ for all $x,y\in\mathbb{R}^d$ with $\norm{x}_0\leq s,\norm{y}_0\leq s$.}\]
Similarly, $\mathsf{f}$ satisfies {\em restricted smoothness}
with parameter $\beta$ at sparsity level $s$, abbreviated as $(\beta,s)$-RSM, if
\[\mathsf{f}(y)\leq \mathsf{f}(x) + \inner{\nabla \mathsf{f}(x)}{y-x} + \frac{\beta}{2}\norm{x-y}^2_2\text{ for all $x,y\in\mathbb{R}^d$ with $\norm{x}_0\leq s,\norm{y}_0\leq s$.}\]
Our results will focus on $\kappa = \beta/\alpha$, the {\em condition number} of the function $\mathsf{f}$ (at the given sparsity level $s$).

\subsection{Iterative hard thresholding}\label{sec:IHT}
Recent work by \citet{jain2014iterative} studies the {\em iterative hard thresholding algorithm},
which alternates between taking a gradient step, $x - \eta\nabla\mathsf{f}(x)$,
and projecting onto the sparsity constraint. Specifically,
given a target sparsity level $s$ and an initial point $x_0\in\mathbb{R}^d$,
the iterative
step of the algorithm is defined by
\begin{equation}\label{eqn:IHT}
x_t = \thr^{\textnormal{HT}}\big(x_{t-1} - \eta \nabla \mathsf{f}(x_{t-1})\big),
\end{equation}
where $\thr^{\textnormal{HT}}$ is the ``hard thresholding'' operator, which truncates any vector $z\in\mathbb{R}^d$ to its $s$ largest entries,
\[\big(\thr^{\textnormal{HT}}(z)\big)_i = \begin{cases}
z_i, & i\in S, \\ 0, & i\not\in S,
\end{cases}\]
where $S\subset\{1,\dots,d\}$ indexes the $s$ largest-magnitude entries of $z$.\footnote{To be fully precise, in the case of a tie between different entries of $z$, we may need to choose which entries to keep and which to set to zero. This choice will not matter from the point of view of our theoretical analysis, and from this point on, we will assume that we have fixed some map $z\mapsto S$, mapping each vector $z\in\mathbb{R}^d$ to a set $S\subset\{1,\dots,d\}$ corresponding to the indices of the $s$ largest entries, so that $|S|=s$ and $\min_{i\in S}|z_i| \geq \max_{j\not\in S}|z_j|$, for every $z$. For instance, in the case of a tie between $z_i$ and $z_j$ for the position of the $s$th largest-magnitude entry, we might follow the rule that we choose to keep entry $i$ if $i<j$ and to keep entry $j$ otherwise. Since the exact choice of the rule for breaking ties is not relevant for our results here, we will implicitly assume it to be fixed for the remainder of this paper.}

\paragraph{Restricted optimality for iterative hard thresholding} It is well known that, due to the nonconvexity of the sparsity constraint $\norm{x}_0\leq s$, the iterative hard thresholding algorithm cannot be guaranteed to find the global minimum, $\min_{\norm{x}_0\leq s}\mathsf{f}(x)$---at least, not without strong assumptions. In other words, it may be the case that $\lim_{t\rightarrow \infty}\mathsf{f}(x_t)$ is strictly larger than $\min_{\norm{x}_0\leq s}\mathsf{f}(x)$.
However, \citet{jain2014iterative}'s analysis of the iterative hard thresholding algorithm~\eqref{eqn:IHT} proves that IHT achieves a weaker optimization guarantee, converging to a loss value that is at least as small as the best value attained under a more restricted constraint $\norm{x}_0\leq s'$ where $s'<s$. More precisely, \citet[Theorem 1]{jain2014iterative} prove that, for an objective function $\mathsf{f}$ satisfying $(\alpha,s)$-RSC and $(\beta,s)$-RSM,
\begin{equation}\label{eqn:jain_IHT}
\mathsf{f}(x_t) \leq \min_{\norm{y}_0\leq s/(32\kappa^2)}\left\{ \mathsf{f}(y) + \left(1 - \frac{1}{12\kappa}\right)^t\cdot (\mathsf{f}(x_0)-\mathsf{f}(y))\right\},
\end{equation}
where $\kappa=\beta/\alpha$, and where the step size is taken to be $\eta \propto 1/\beta$. In other words, their result proves linear convergence to the bound
\[\lim_{t\rightarrow \infty}\mathsf{f}(x_t) \leq \min_{\norm{y}_0\leq s/(32\kappa^2)}\mathsf{f}(y),\]
meaning that while IHT may not find the global minimum of $\mathsf{f}(x)$ relative to the $s$-sparsity constraint, it is nonetheless guaranteed to perform at least as well as the best $s/(32\kappa^2)$-sparse solution. An analogous result is proved for the low-rank setting, thresholding singular values instead of vector entries.

In this work, we will refer to this type of result as a {\em restricted optimality} guarantee, where the output of an $s$-sparse optimization algorithm is guaranteed to perform well relative to a more restrictive $s'$-sparsity constraint, for some $s'<s$. In particular, we will be interested in the sparsity ratio $s'/s$---the ratio between the sparsity level $s$ used in the algorithm, versus the level $s'$ appearing in the guarantee. Ideally, we would like this ratio to be as close to $1$ as possible, for the strongest possible guarantee.

\subsection{Related literature}

\paragraph{Iterative thresholding}
There exists a vast literature on the properties of iterative thresholding algorithms, especially iterative hard thresholding, regarding the optimization properties and statistical guarantees of these algorithms. Recent results in this area include the work of \citet{blumensath2009iterative,jain2014iterative,chen2015fast,bhatia2015robust,jain2016structured,cai2016optimal,kyrillidis2014matrix}.

Accelerated forms of the iterative hard thresholding algorithm are studied in \citet{kyrillidis2011recipes,blumensath2012accelerated,khanna2017iht}. In particular, \citet{khanna2017iht} finds substantial theoretical and empirical improvement
over the original non-accelerated version of the algorithm. \citet{nguyen2017linear} studies iterative hard thresholding in the context of stochastic gradient descent, where at each step $t$ we only have access to a noisy vector that approximates the true current gradient, $\nabla\mathsf{f}(x_t)$. The works mentioned here also consider thresholding algorithms for the low-rank setting, truncating singular values instead of vector entries. More broadly, \citet{nguyen2017linear}'s work considers approximate thresholding procedures and more general definitions of sparsity.

To the best of our knowledge, the question of optimality among thresholding operators has not been addressed before, and it is the goal of this work to provide a framework to identify the worst-case convergence behavior of {\em all} thresholding operators and to find the ones that enjoy the optimal restricted optimality guarantee.

\paragraph{Penalized and constrained optimization methods}
The sparse optimization problem can alternately be approximated by a penalized minimization problem,
\[\min_{x\in\mathbb{R}^d}\left\{\mathsf{f}(x) + \lambda\mathsf{R}(x)\right\},\]
or a constrained optimization problem,
\[\min_{x\in\mathbb{R}^d}\left\{\mathsf{f}(x) \ : \ \mathsf{R}(x) \leq c\right\},\]
where $\mathsf{R}(x)$ is a sparsity-promoting regularizer, and $\lambda$ and $c$ are tuning parameters controlling the penalization or constraint. Of course, choosing $\mathsf{R}(x) = \norm{x}_0$ would reduce to the original target optimization problem, but these minimizations are generally only feasible to solve if $\mathsf{R}(x)$ is some relaxation of the sparsity constraint/penalty. For example, the Lasso~\citep{tibshirani1996regression} uses a convex regularizer, $\mathsf{R}(x)=\norm{x}_1$, which enjoys many strong guarantees of accurate estimation of the true sparse signal $x$ and of its support. More recently, many nonconvex penalties have been proposed that reduce the shrinkage bias of the Lasso, at the cost of a more challenging optimization problem, such as the SCAD~\citep{fan2001variable} and MCP~\citep{zhang2010nearly} penalties. The $\ell_q$ norm, for $q\in(0,1)$, has also been extensively studied as a compromise between the convex but biased $\ell_1$ norm (as in the Lasso), and the theoretically optimal but computationally infeasible $\ell_0$ norm (i.e.~the sparsity constraint, $\norm{x}_0\leq s$). Results for the $\ell_q$ norm include work by \citet{chartrand2007exact,foucart2009sparsest,kabashima2009typical,lai2011unconstrained}. \citet{zheng2015does}'s recent work studies the $\ell_q$ norm using the framework of approximate message passing to characterize its superior performance relative to the convex $\ell_1$ norm. While the resulting optimization problem is nonconvex for these alternatives to the $\ell_1$ norm, \citet{loh2013regularized} show that restricted strong convexity in the objective function $\mathsf{f}$ is sufficient to outweigh bounded concavity in the penalty, to ensure successful optimization within a small error tolerance.

The penalized or constrained formulations of the sparse minimization problem may initially appear to be fundamentally different from the iterative thresholding approach. However, these penalized or constrained problems are often optimized with proximal gradient descent or projected gradient descent algorithms---specifically, for a penalty, the proximal gradient descent algorithm iterates the steps
\[\begin{cases}
\textnormal{Gradient step: }x'_t = x_{t-1} - \eta_t\cdot\nabla\mathsf{f}(x_{t-1})\text{ for some step size $\eta_t$},\\
\textnormal{Proximal step: }x_t =\arg\min_{x\in\mathbb{R}^d} \left\{\frac{1}{2}\norm{x - x'_t}^2_2 + \eta_t \lambda \mathsf{R}(x)\right\},\end{cases}\]
while for a constraint, projected gradient descent iterates the steps
\[\begin{cases}
\textnormal{Gradient step: }x'_t = x_{t-1} - \eta_t\cdot\nabla\mathsf{f}(x_{t-1})\text{ for some step size $\eta_t$},\\
\textnormal{Projection step: }x_t =\arg\min_{x\in\mathbb{R}^d} \left\{\frac{1}{2}\norm{x - x'_t}^2_2 \ : \ \mathsf{R}(x)\leq c\right\}.\end{cases}\]
Since $\mathsf{R}(x)$ is a sparsity-promoting regularizer, each iteration $x_t$ will therefore be sparse or approximately sparse. In this way, the penalized loss or constrained loss formulations of the sparse minimization problem can be viewed as analogous to the family of iterated thresholding algorithms, where the thresholding step is replaced by penalizing or constraining a regularizer $\mathsf{R}(x)$ that is a relaxation of the sparsity constraint. (We will discuss the regularized problem more in Section~\ref{sec:softthresh}.)

\section{Convergence of iterative thresholding}\label{sec:iterative_thresholding}

In this section, we examine the performance of gradient
descent with iterative thresholding, for various choices of the thresholding
operator $\Psi_s$. Specifically, after initializing at any point $x_0\in\mathbb{R}^d$,
the algorithm proceeds by alternating between taking a gradient descent step,
and applying a thresholding operator:
\begin{equation}\label{eqn:iterative_thresh}
x_t = \Psi_s\big(x_{t-1} - \eta_t \nabla \mathsf{f}(x_{t-1})\big),
\end{equation}
where $\Psi_s: \mathbb{R}^d \rightarrow \{x\in\mathbb{R}^d:\norm{x}_0\leq s\}$ is some thresholding operator that enforces $s$-sparsity at each step.

\paragraph{Step size choice}
Throughout the paper, we will primarily study this generalized iterative thresholding algorithm
under the choice of a universal fixed step size $\eta = 1/\beta$, where
$\beta$ is the restricted smoothness parameter for the function $\mathsf{f}$. When $\beta$ is unknown, we will also consider the following adaptive choice of step size based on exact line search:
\begin{equation}\label{eqn:iterative_thresh_eta_adaptive}
\begin{cases}
\text{Define }\widetilde{x}_t(\eta) = \Psi_s\big(x_{t-1} - \eta \nabla \mathsf{f}(x_{t-1})\big),\\
\text{Choose }\eta_t = \max\left\{\eta\geq 0 : \mathsf{f}(\widetilde{x}_t(\eta))\leq \mathsf{f}(x_{t-1}) + \inner{\widetilde{x}_t(\eta) - x_{t-1}}{\nabla\mathsf{f}(x_{t-1})} + \frac{1}{2\eta}\norm{\widetilde{x}_t(\eta) - x_{t-1}}^2_2\right\},\\
\text{Set }x_t = \widetilde{x}_t(\eta_t).
\end{cases}
\end{equation}
Note that, since $x_{t-1}$ and $\widetilde{x}_t(\eta)$ are both $s$-sparse, the curvature condition
\begin{equation}\label{eqn:stepsize_check}
\mathsf{f}(\widetilde{x}_t(\eta))\leq \mathsf{f}(x_{t-1}) + \inner{\widetilde{x}_t(\eta) - x_{t-1}}{\nabla\mathsf{f}(x_{t-1})} + \frac{1}{2\eta}\norm{\widetilde{x}_t(\eta) - x_{t-1}}^2_2\
\end{equation}
is necessarily satisfied for any $\eta\leq \frac{1}{\beta}$ due to the restricted smoothness property. Therefore we will always have $\eta_t \geq \frac{1}{\beta}$. Intuitively, the rule not only helps us get rid of the need to know $\beta$, but also allows the algorithm to take larger step size for more progress when possible. In practice, we would consider using a backtracking line search, that is, starting from a large step size and iteratively shrinking it until condition~\eqref{eqn:stepsize_check} is satisfied. In this way, condition~\eqref{eqn:stepsize_check} is similar to the classical Armijo rule for backtracking line search. For simplicity of our theoretical result we do not treat inexact linesearch in the following.

\paragraph{Restricted optimality}
Given an iterative algorithm that keeps the sparsity of the iterations at $s$, as discussed in Section~\ref{sec:IHT}, we cannot hope to achieve {\em global optimality} (i.e.~a guarantee that $\mathsf{f}(x_t)$ is nearly as good as the best $s$-sparse solution, $\min_{\norm{x}_0\leq s}\mathsf{f}(x)$), but we can instead prove guarantees of {\em restricted optimality}, that is $\lim_{t\rightarrow\infty}\mathsf{f}(x_t)\leq \min_{\norm{x}_0\leq s'}\mathsf{f}(x)$, for some tighter sparsity constraint $s'\leq s$. We will assess a thresholding operator $\Psi_s$ based on its ability to guarantee restricted optimality relative to a sparsity level $s'$ that is as close to $s$ as possible, i.e.~a sparsity ratio $\rho=s'/s$ that is as close to $1$ as possible.

\subsection{Relative concavity of a thresholding operator}
Let $s\in\{1,\dots,d\}$ be any fixed sparsity level and let $\rho\in[0,1]$.
We define the {\em relative concavity} of an $s$-sparse thresholding
operator $\Psi_s$ relative to sparsity proportion $\rho$ as
\[\gamma_{s,\rho}(\Psi_s) = \sup\left\{\frac{\inner{y-\Psi_s(z)}{z-\Psi_s(z)}}{\norm{y-\Psi_s(z)}^2_2} \ : \ y,z\in\mathbb{R}^d, \  \norm{y}_0\leq \rho s, \ y\neq \Psi_s(z)\right\}.\]
Note that $\frac{\inner{y-\Psi_s(z)}{z-\Psi_s(z)}}{\norm{y-\Psi_s(z)}^2_2}$ is the coefficient of projection when projecting $z-\Psi_s(z)$ onto $y-\Psi_s(z)$, and measures how much these two vectors align. To understand the term ``relative concavity'' in the name,
we note that if $\Psi_s$ were a projection operator to some convex constraint set $\mathcal{C}$, then we would have $\inner{y-\Psi_s(z)}{z-\Psi_s(z)}\leq 0$
for any $y\in\mathcal{C}$, by the properties of convex projections.
For sparse estimation, the constraint $\norm{x}_0\leq s$ is not convex; any positive values of $\inner{y-\Psi_s(z)}{z-\Psi_s(z)}$ with $\norm{y}_0\leq s$ measure the extent to which the thresholding operator $\Psi_s$ behaves {\em differently} from a convex projection. By taking a more restrictive constraint on $y$, namely $\norm{y}_0\leq \rho s$ rather than $\norm{y}_0\leq s$, we reduce this measure of concavity; the relative concavity of $\Psi_s$ will be smaller for lower values of $\rho$.

This notion of relative concavity is closely related to the {\em local concavity coefficients} developed in \citet{barber2017gradient} for the purpose of studying projected gradient descent with an arbitrary nonconvex constraint. We will compare the two later on, after presenting our main theorems.

\subsection{Relative concavity and iterative thresholding}
We now examine how the relative concavity of $\Psi_s$ relates
to the convergence behavior of iterative thresholding with a fixed step size. The main message, casted informally, is this:
\begin{quote}
  Given sparsity levels $s$ and $s'=\rho s$, and an $s$-sparse thresholding operator $\Psi_s$, the  condition $\gamma_{s,\rho}(\Psi_s)\leq \frac{1}{2\kappa}$ is both necessary and sufficient for restricted optimality to hold relative to sparsity level $s'$.
\end{quote}

\paragraph{Stationary points}
Before giving our formal results, we start with a warm-up---supposing that $x$ is a stationary point of the iterative thresholding algorithm with step size $\eta = \frac{1}{\beta}$, what guarantees can we give about $\mathsf{f}(x)$? If $\mathsf{f}$ satisfies $(\alpha,s)$-RSC, then we know that
\[\mathsf{f}(y)\geq \mathsf{f}(x) + \inner{y-x}{\nabla\mathsf{f}(x)} + \frac{\alpha}{2}\norm{x-y}^2_2\]
for any $s$-sparse $y$. Furthermore, writing $z = x - \eta\nabla\mathsf{f}(x)$, we know that $\Psi_s(z)=x$ since $x$ is a stationary point. Therefore,
\begin{equation}\label{eqn:stationary} \inner{y-x}{\nabla\mathsf{f}(x)} =- \beta\inner{y-x}{z-x} \geq -\beta\gamma_{s,\rho}(\Psi_s)\norm{x-y}^2_2\geq -\frac{\alpha}{2}\norm{x-y}^2_2,\end{equation}
as long as $y$ is $\rho s$-sparse and the relative concavity satisfies $\gamma_{s,\rho}(\Psi_s)\leq \frac{1}{2\kappa}$. In other words, this condition on relative concavity is sufficient to ensure that
\[\mathsf{f}(x)\leq \min_{\norm{y}_0\leq \rho s}\mathsf{f}(y)\text{ for any stationary point $x$}.\]
Conversely, if $\gamma_{s,\rho}(\Psi_s)>\frac{1}{2\kappa}$, Theorem~\ref{thm:lowerbd} below will construct a stationary point $x$ that {\em fails} to satisfy $\mathsf{f}(x)\leq\min_{\norm{y}_0\leq\rho s}\mathsf{f}(y)$.

\paragraph{Convergence results}
Next we turn to results for the iterated thresholding algorithm initialized at an arbitrary $s$-sparse point $x_0$ (for example, initialized at zero). Our first theorem accounts for the sufficiency of the condition.
\begin{theorem}\label{thm:upperbd}
Consider any objective function $\mathsf{f}:\mathbb{R}^d\rightarrow \mathbb{R}$, any sparsity levels $s\geq s'$, and any $s$-sparse thresholding operator $\Psi_s$. Assume the objective function $\mathsf{f}$ satisfies $(\alpha,s)$-RSC and $(\beta,s)$-RSM. Let $\rho = s'/s$ and $\kappa = \beta/\alpha$, and assume that
\[\gamma_{s,\rho}(\Psi_s) < \frac{1}{2\kappa}.\]
Then, for any $s$-sparse $x_0\in\mathbb{R}^d$ and any $s'$-sparse $y\in\mathbb{R}^d$, the iterated thresholding algorithm~\eqref{eqn:iterative_thresh} initialized at $x_0$ and run with fixed step size $\eta=1/\beta$ satisfies
\[\min_{t=1,\dots,T} \mathsf{f}(x_t) \leq \mathsf{f}(y) + \left(\frac{1 - 1/\kappa}{1-2\gamma_{s,\rho}(\Psi_s)}\right)^T \cdot \frac{\beta}{2} \norm{x_0 - y}^2_2\]
for each $T\geq 1$. The same result holds for the iterative thresholding algorithm with adaptive step size~\eqref{eqn:iterative_thresh_eta_adaptive}.
\end{theorem}
\noindent In other words, the condition $\gamma_{s,\rho}(\Psi_s)<\frac{1}{2\kappa}$ guarantees restricted optimality on the class of $\kappa$-conditioned objective functions at sparsity proportion $\rho$. Next, we examine the necessity of the bound on $\gamma_{s,\rho}(\Psi_s)$. The following result proves that, if $\gamma_{s,\rho}(\Psi_s)>\frac{1}{2\kappa}$, then there exists an objective function $\mathsf{f}(x)$ on which the restricted optimality guarantee fails, when we run iterative thresholding with fixed step size $\eta = \frac{1}{\beta}$.
\begin{theorem}\label{thm:lowerbd}
Consider any sparsity levels $s\geq s'$, any $s$-sparse thresholding operator $\Psi_s$, and any constants $\beta \geq \alpha >0$. Let $\rho=s'/s$ and $\kappa = \beta/\alpha$, and assume that
\[\gamma_{s,\rho}(\Psi_s)>\frac{1}{2\kappa}.\]
Then there exists an objective function $\mathsf{f}(x)$ that satisfies $(\alpha,s)$-RSC and $(\beta,s)$-RSM, and an $s$-sparse $x_0\in\mathbb{R}^d$ and $s'$-sparse $y\in\mathbb{R}^d$, such that the iterated thresholding algorithm~\eqref{eqn:iterative_thresh} run with step size $\eta=1/\beta$ and initialization point $x_0$ satisfies
\[\lim_{t\rightarrow \infty} \mathsf{f}(x_t) > \mathsf{f}(y).\]
\end{theorem}
\noindent This result is proved by constructing an objective function $\mathsf{f}$ and an $s$-sparse point $x_0$, such that $\mathsf{f}(x_0)>\mathsf{f}(y)$, but $x_0$ is a stationary point of the iterated thresholding algorithm, i.e.~by initializing at $x_0$, we obtain $x_t=x_0$ for all $t\geq 1$. This proves that the iterated thresholding algorithm does not satisfy restricted optimality (at the given sparsity levels), since it is trapped at an $s$-sparse point $x_0$ whose objective value is strictly worse than that of the $s'$-sparse point $y$.

\paragraph{Local vs global guarantees}
We have seen that the condition $\gamma_{s,\rho}(\Psi_s)<\frac{1}{2\kappa}$ on the relative concavity, is sufficient to ensure a restricted optimality result, without any initialization conditions---that is, this is a global result, rather than a result that holds only in some neighborhood of the optimal solution. We can compare this framework to the local concavity coefficients of \citet{barber2017gradient}, where the convergence guarantee is of a local type.

In the present work, to achieve our convergence result via relative concavity, we require that, for any $z\in\mathbb{R}^d$ and for $x=\Psi_s(z)$, we have
$\inner{y-x}{z-x}\leq \gamma_{s,\rho}(\Psi_s)\norm{x-y}^2_2$ for all $\rho s$-sparse  $y$. For a stationary point $x$ of the iterated thresholding algorithm with step size $\eta=\frac{1}{\beta}$, we would have $z = x - \eta\nabla\mathsf{f}(x)$, and so the requirement above can be rewritten as
\begin{equation}\label{eqn:compare_RC_LCC_1}\inner{y-x}{-\nabla\mathsf{f}(x)}\leq \beta\cdot \gamma_{s,\rho}(\Psi_s)\cdot \norm{x-y}^2_2 \text{ for all $y$ with $\norm{y}_0\leq \rho s$},\end{equation}
 and the term $\beta\gamma_{s,\rho}(\Psi_s)$ on the right-hand side is bounded as $\beta\gamma_{s,\rho}(\Psi_s)< \beta\cdot\frac{1}{2\kappa} = \frac{\alpha}{2}$ according to the conditions of Theorem~\ref{thm:upperbd}.

In contrast, \citet{barber2017gradient}'s local concavity coefficient framework requires that, for any $z\in\mathbb{R}^d$ and any $y\in\mathcal{C}$, $\inner{y-x}{z-x} \leq\gamma_x(\mathcal{C}) \cdot \norm{z-x}\cdot \norm{y-x}^2_2$ where $x=P_{\mathcal{C}}(z)$ is the projection of $z$ to the constraint set $\mathcal{C}$.
At a stationary point $x$ of projected gradient descent, we have $z = x - \eta\nabla\mathsf{f}(x)$, and so equivalently,
\begin{equation}\label{eqn:compare_RC_LCC_2}\inner{y-x}{-\nabla\mathsf{f}(x)} \leq\gamma_x(\mathcal{C}) \cdot \norm{\nabla\mathsf{f}(x)}\cdot \norm{y-x}^2_2\text{ for all $y\in\mathcal{C}$ }.\end{equation}
\citet{barber2017gradient}'s main results prove convergence to the global minimum over $\mathcal{C}$, as long as the algorithm is initialized in a neighborhood within which the condition $\gamma_x(\mathcal{C})\norm{\nabla\mathsf{f}(x)}<\frac{\alpha}{2}$ holds uniformly.\footnote{The norm $\norm{\cdot}$ measuring the magnitude of the gradient $\nabla \mathsf{f}(x)$ is not necessarily the $\ell_2$ norm---it is typically chosen to be smaller than the $\ell_2$ norm, for instance, the $\ell_{\infty}$ norm in the case of sparse estimation---but this is not relevant to the comparison here.}

Comparing the relative concavity framework~\eqref{eqn:compare_RC_LCC_1} with \citet{barber2017gradient}'s local concavity coefficient framework~\eqref{eqn:compare_RC_LCC_2}, we see that in both settings, $\inner{y-x}{-\nabla\mathsf{f}(x)}$ is required to strictly less than $\frac{\alpha}{2}\norm{y-x}^2_2$. The difference is that:
\begin{itemize}
\item \citet{barber2017gradient}'s work requires this bound to hold for all $y$ in the constraint set $\mathcal{C}$, but only for $x$ in some neighborhood the global optimum. If the algorithm is initialized in this neighborhood, then global optimality is guaranteed.
\item Our present work requires this bound to hold only for a more restricted set of $y$'s, i.e.~with the restricted sparsity level $\norm{y}_0\leq s'=\rho s$, but for all $x$ in the constraint set of $s$-sparse vectors. Regardless of where the algorithm is initialized, we obtain a restricted optimality guarantee.
\end{itemize}
Overall, by requiring the concavity bound to hold only for a more restricted set of $y$'s, our new result is able to avoid initialization conditions, at the cost of obtaining restricted optimality rather than global optimality as the final guarantee.

\section{Upper and lower bounds on relative concavity}\label{sec:RC}
We have now seen that the relative concavity $\gamma_{s,\rho}(\Psi_s)$
fully characterizes the worst-case performance of the thresholding operator $\Psi_s$
in the gradient descent algorithm, with a convergence guarantee in Theorem~\ref{thm:upperbd} and a matching lower bound in Theorem~\ref{thm:lowerbd} (assuming a fixed step size). In this next section, we turn to the question of investigating the relative concavity in greater detail, in order to determine which thresholding operators are most likely to lead to successful optimization. Along the way, we will focus on the following questions:
\begin{itemize}
  \item What is the relative concavity of commonly used thresholding operators, for example, hard thresholding and soft thresholding?
  \item What is the best (i.e.~lowest) possible relative concavity $\gamma_{s,\rho}(\Psi_s)$ among all thresholding operators $\Psi_s$, and which thresholding operators are optimal?
\end{itemize}
\noindent Throughout this section, for providing upper and lower bounds on $\gamma_{s,\rho}(\Psi_s)$, we will assume without comment that $s,s'\in\{1,\dots,d\}$ are two sparsity levels satisfying $1\leq s'\leq s\leq d$ and $s+s'\leq d$, and we will define $\rho = s'/s$ as usual.

\subsection{Relative concavity of hard and soft thresholding}\label{sec:relativeconcavity_HT_ST}

First, we consider hard thresholding, $\Psi_s = \thr^{\textnormal{HT}}$. The following result
computes the relative concavity for the hard thresholding operator:
\begin{lemma}\label{lem:RC_HT}
The relative concavity of hard thresholding is given by
\[\gamma_{s,\rho}(\thr^{\textnormal{HT}}) = \frac{\sqrt{\rho}}{2}\]
for every sparsity proportion $\rho\in(0,1]$.
\end{lemma}
\noindent In particular, with Lemma~\ref{lem:RC_HT}, the condition $\gamma_{s,\rho}(\thr^{\textnormal{HT}})<\frac{1}{2\kappa}$ becomes $\rho<\frac{1}{\kappa^2}$. In light of Theorems~\ref{thm:upperbd} and~\ref{thm:lowerbd}, we see that for iterative hard thresholding algorithm, $\rho<\frac{1}{\kappa^2}$ is necessary and sufficient to guarantee restricted optimality with sparsity level $s$ and $s'$, tightening the condition
obtained in \citet{jain2014iterative} where they prove restricted optimality with the sparsity proportion $\rho = \frac{1}{32\kappa^2}$.

We might wonder whether the highly discontinuous nature of the hard thresholding function might not be ideal---by smoothing out the discontinuity, could we attain better performance? However, we find that any continuous thresholding operator with respect to the Euclidean distance in $\mathbb{R}^d$ is necessarily worse than hard thresholding:
\begin{lemma}\label{lem:RC_contin}
For any continuous map $\Psi_s: \mathbb{R}^d \rightarrow \{x\in\mathbb{R}^d:\norm{x}_0\leq s\}$, its relative concavity satisfies
\[\gamma_{s,\rho}(\Psi_s)\geq 1\]
for every sparsity proportion $\rho\in(0,1]$.
\end{lemma}
\noindent In particular, since $\kappa\geq 1$, the condition $\gamma_{s,\rho}(\Psi_s)<\frac{1}{2\kappa}$ never holds if $\Psi_s$ is continuous. Comparing to Theorem~\ref{thm:lowerbd}, we see that no continuous operator can guarantee restricted optimality at any sparsity ratio $\rho$, even in the ideal setting where $\mathsf{f}$ is well-conditioned.
This includes soft thresholding at a fixed sparsity level, i.e., the map $\thr^{\textnormal{ST}}$ that shrinks all entries of $z$ equally until the desired sparsity level is reached:
\[\big(\thr^{\textnormal{ST}}(z)\big)_i = \begin{cases}
z_i - \lambda, & z_i>\lambda, \\
 0, & |z_i|\leq \lambda,\\
z_i + \lambda,&z_i< -\lambda,
\end{cases}\text{ \quad taking $\lambda\geq 0$ to be the smallest value s.t.~$\norm{\thr^{\textnormal{ST}}(z)}_0\leq s$.}\]
In practice, it is much more common to implement soft thresholding at a fixed $\lambda$, rather than at a fixed $s$.
We will discuss the fixed-$\lambda$ formulation of soft thresholding later on, in Section~\ref{sec:softthresh}.

\subsection{Optimal value of relative concavity}
In this section we turn to the question of optimality: what is the optimal value of relative concavity among all thresholding operators at a given sparsity proportion $\rho$? We will establish that
\[\inf_{\Psi_s: \mathbb{R}^d \rightarrow \{x\in\mathbb{R}^d:\norm{x}_0\leq s\}} \gamma_{s,\rho}(\Psi_s)=\frac{\rho}{1+\rho}.\]
That is, the lowest relative concavity among all thresholding operators at a given sparsity proportion $\rho$ is exactly $\frac{\rho}{1+\rho}$.
Since this is much smaller than $\frac{\sqrt{\rho}}{2}$ when $\rho$ is small, we see that hard thresholding is suboptimal.

We start with the following lower bound for all thresholding operators:
\begin{lemma}\label{lem:RC_lowerbd}
For any map $\Psi_s: \mathbb{R}^d \rightarrow \{x\in\mathbb{R}^d:\norm{x}_0\leq s\}$ and any sparsity proportion $\rho\in(0,1]$,
the relative concavity is lower-bounded as
\[\gamma_{s,\rho}(\Psi_s)\geq \frac{\rho}{1+\rho}.\]
\end{lemma}

\noindent To show that this lower bound is indeed tight, we will consider $\ell_q$ thresholding and establish upper bound for its relative concavity that matches this lower bound with proper choices of $q$. $\ell_q$ thresholding encourage sparsity without exerting too much shrinkage by constraining the $\ell_q$ norm of the vector after thresholding for some $q\in (0,1)$. To be precise, let
\[P_{\ell_q}(z;t) = \arg\min\left\{\norm{x-z}_2 : \norm{x}_q\leq t\right\}\]
denote projection to the $\ell_q$ ball, where $\norm{x}_q =\left(\sum_i |x_i|^q\right)^{1/q}$ is the $\ell_q$ ``norm'' (in fact a nonconvex function since $q<1$). Then define
\[\thr^{\ell_q}(z) = P_{\ell_q}(z;t(z)),\text{ where }
t(z) = \sup\left\{t : \norm{P_{\ell_q}(z;t)}_0\leq s\right\}.\]
In words, $\thr^{\ell_q}(z)$ projects $z$ to an $\ell_q$ ball whose radius is chosen to be as large as possible while still ensuring $s$-sparsity.\footnote{Note that $P_{\ell_q}(z;t)$ may be non-unique. To be fully precise, we define $\thr^{\ell_q}(z)$ by first fixing some map $z\mapsto S$, the possibly non-unique support of its largest $s$ entries, and then defining $t(z)$ and choosing the possibly non-unique projection $P_{\ell_q}(z;t(z))$ in such a way that the nonzero entries in the projection are exactly on this support.} The following result computes the relative concavity for $\ell_q$ thresholding:
\begin{lemma}\label{lem:RC_LQ}
The relative concavity of $\ell_q$ thresholding $\thr^{\ell_q}$ is equal to
\[\gamma_{s,\rho}(\thr^{\ell_q}) = \frac{\frac{\rho}{\min\{1,(\frac{2-q}{q})^2(1-\rho)\}}}{\frac{4q(1-q)}{(2-q)^2}(1+\sqrt{1+(\frac{2-q}{q})^2\frac{\rho}{\min\{1,(\frac{2-q}{q})^2(1-\rho)\}}})}\]
for every sparsity proportion $\rho\in (0,1)$.
In particular, if we choose
\[q = \frac{2(1-\rho)}{3-\rho},\]
then the resulting thresholding operator attains the lowest possible relative concavity,
\[\gamma_{s,\rho}(\thr^{\ell_q}) = \frac{\rho}{1+\rho},\text{ for } q = \frac{2(1-\rho)}{3-\rho}.\]
In addition, the universal choice $q=2/3$ yields relative concavity equal to,
\[\gamma_{s,\rho}(\thr^{\ell_{2/3}}) =  \frac{\frac{\rho}{\min\{1,4(1-\rho)\}}}{\frac{1}{2} + \frac{1}{2}\sqrt{1+\frac{4\rho}{\min\{1,4(1-\rho)\}}}}\leq \frac{\rho}{\min\{1,4(1-\rho)\}},\text{ for all $\rho\in(0,1)$}.\]
\end{lemma}
\noindent Now we provide some explanation for this result. If we are allowed to choose $q$ depend on $\rho$, then the choice $q=\frac{2(1-\rho)}{3-\rho}$ would lead to a relative concavity of $\frac{\rho}{1+\rho}$, which exactly matches the lower bound in Lemma~\ref{lem:RC_lowerbd}. Of course this specific choice of $q$ is chosen for a specific sparsity proportion $\rho$ and might not work well for other values of the sparsity proportion. To avoid this drawback or the need to tune the parameter $q$, one can have the universal choice $q=2/3$. Due to the expression for $\gamma_{s,\rho}(\thr^{\ell_{2/3}})$, we see that $\gamma_{s,\rho}(\thr^{\ell_{2/3}})\approx \rho$ when $\rho$ is small, thus nearly matching the lower bound $\frac{\rho}{1+\rho}$.

In particular, with the optimal value of relative concavity $\gamma_{s,\rho}=\frac{\rho}{1+\rho}$, the condition $\gamma_{s,\rho}<\frac{1}{2\kappa}$ becomes $\rho<\frac{1}{2\kappa-1}$. In light of  Theorem~\ref{thm:upperbd} and Theorem~\ref{thm:lowerbd}, we see that $\rho<\frac{1}{2\kappa-1}$ is both necessary and sufficient for restricted optimality to hold with sparsity proportion $\rho$.  Compare this with the condition $\rho<\frac{1}{\kappa^2}$ required by hard thresholding, we see that the dependence on condition number is greatly improved!

\subsection{A general class of thresholding operators}\label{sec:generalclass}
Now that we have seen that $\ell_q$ thresholding operators enjoy good properties in terms of relative concavity, we can ask whether there are other thresholding operators of such optimal and near-optimal properties. In this section we address this problem by showing $\ell_q$ thresholding can be characterized as a special case of a larger class of thresholding operators, which all enjoy the same optimality properties in the sense of their relative concavity. Consider any nonincreasing function
\[\sigma: [1,\infty)\rightarrow [0,1],\]
which we call the ``shrinkage function'', which will determine the amount of shrinkage on each entry of a vector $z$ at the thresholding step.
Defining the support $S$ and thresholding level $\tau=\max_{i\not\in S}|z_i|$ as before, we then define the thresholding operator $\Psi_{s;\sigma}$ as
\[\big(\Psi_{s;\sigma}(z)\big)_i = \begin{cases}
z_i- \tau\sigma\big(|z_i|/\tau\big),&i\in S,\\
0,&i\not\in S.\end{cases}\]
In other words, for entry $i\in S$, $\sigma\big(|z_i|/\tau\big)$ determines the {\em relative} amount of shrinkage on this entry. The intuitive meaning of $\sigma$ is illustrated in Figure~\ref{fig:explaindef}. (If $\tau = 0$, i.e.~$z$ is already $s$-sparse, then we would simply take $\Psi_{s;\sigma}(z)=z$; we will ignore this case from this point on.)

\begin{figure}[t]
\centering
\includegraphics[width=0.7\textwidth]{explain_sigma.pdf}
\caption{An illustration of the definition of the relative shrinkage function $\sigma$ (for simplicity we use thresholding level $\tau=1$ in this illustration). Here we use the relative shrinkage function $\sigma(t) = \frac{t-\sqrt{t^2-1}}{2}$, corresponding to the reciprocal thresholding operator defined in Section~\ref{sec:RT}.}
\label{fig:explaindef}
\end{figure}

Note that since $\sigma$ is nondecreasing, the maximum shrinkage occurs when $|z_i|=\tau$ exactly; the amount of shrinkage in this setting is governed by $\sigma(1)$.

We can now examine the relationship of the choice of $\sigma$ to the relative concavity:
\begin{lemma}\label{lem:unifying_RC}
For any nonincreasing shrinkage function $\sigma:[1,\infty)\rightarrow[0,1]$ such that $0<\sigma(1)<1$ and
\begin{equation}\label{eqn:sigmono}
t\mapsto \sigma(t)(t-\sigma(t))\text{ is nondecreasing over $t\geq 1$},\end{equation}
the thresholding operator $\Psi_{s;\sigma}$ has relative concavity
\[\gamma_{s,\rho}(\Psi_{s;\sigma}) = \frac{\frac{\rho}{\min\{1,(1-\rho)/\sigma(1)^2\}}}{2\sigma(1)\big(1-\sigma(1)\big)\left(1 + \sqrt{1 + \frac{\rho/\sigma(1)^2}{\min\{1,(1-\rho)/\sigma(1)^2\}}}\right)}.\]
In particular, the resulting operator attains the lowest possible relative concavity,
\[\gamma_{s,\rho}(\Psi_{s;\sigma}) = \frac{\rho}{1+\rho},\]
if and only if $\sigma(1) = \frac{1-\rho}{2}$.
If instead we take a universal shrinkage level $\sigma(1)= \frac{1}{2}$, then the relative concavity is given by
\[\gamma_{s,\rho}(\Psi_{s;\sigma}) =
\frac{\frac{\rho}{\min\{1,4(1-\rho)\}}}{\frac{1}{2} + \frac{1}{2}\sqrt{1+\frac{4\rho}{\min\{1,4(1-\rho)\}}}}
\leq \frac{\rho}{\min\{1,4(1-\rho)\}}.\]
\end{lemma}
Examining the definition of this general family of thresholding operators, we can see that $\ell_q$ thresholding corresponds to setting
\[\sigma(t)= t-\left(\text{the larger-magnitude root $x$ of the equation $t = x + \frac{q(2-2q)^{1-q}/(2-q)^{2-q}}{x^{1-q}}$}\right),\]
for which we have $\sigma(1) = \frac{q}{2-q}$ and which satisfies~\eqref{eqn:sigmono}. We also have that $\sigma(1) = \frac{1}{2}$ corresponds to the ``universal'' choices $q=2/3$, and $\sigma(1) = \frac{1-\rho}{2}$ (the optimal value) corresponds to the $\rho$-specific choices $q = \frac{2(1-\rho)}{3-\rho}$. As a consequence, the previous result for $\ell_q$ thresholding, Lemma~\ref{lem:RC_LQ}, is simply special case of this more general lemma.

On the other hand, the hard thresholding operator $\thr^{\textnormal{HT}}$ can be obtained by setting $\sigma(t)=0$ for all $t\in[1,\infty)$, but this does not satisfy the assumption $\sigma(1)>0$ required in the lemma. However, if we informally consider fixing $\rho>0$ and taking a limit $\sigma(1)\rightarrow 0$ in the upper bound in the lemma, we see
\[\lim_{\sigma(1)\rightarrow 0}\frac{\frac{\rho}{\min\{1,\frac{1-\rho}{\sigma(1)^2}\}}}{2\sigma(1)(1-\sigma(1))\left(1+\sqrt{1+\frac{\rho}{\sigma(1)^2\min\{1,\frac{1-\rho}{\sigma(1)^2}\}}}\right)}= \frac{\sqrt{\rho}}{2},\]
obtaining the relative concavity of hard thresholding calculated earlier.

\subsection{Reciprocal thresholding and minimal shrinkage}\label{sec:RT}
Practically, for two thresholding operators with the same restricted optimality guarantees, i.e. with the exact same value of relative concavity, we may favor the one that exerts smaller amount of shrinkage. Thus it makes sense to ask among the general class of thresholding operators defined in Section~\ref{sec:generalclass}, which operators exert the minimal amount of shrinkage? Consider all operators of the form $\Psi_{s;\sigma}$, with some fixed value of $\sigma(1)\in (0,1/2]$. For any $\sigma$ satisfying the assumption~\eqref{eqn:sigmono}, for all $t\geq 1$ we have
\[\sigma(t)(t-\sigma(t)) \geq \sigma(1)(1-\sigma(1)).\]
For convenience, we reparametrize this equation by setting $c = 1-2\sigma(1) \in[0,1)$, and so we are considering all nonincreasing functions $\sigma:[1,\infty)\rightarrow[0,1]$ that satisfy $\sigma(1) = \frac{1-c}{2}$ and
\[\sigma(t)(t-\sigma(t)) \geq \sigma(1)(1-\sigma(1)) = \frac{1-c^2}{4}.\]
Thus, we must have
\begin{equation}\label{eqn:sigmalowerbd}
\sigma(t) \geq \frac{t - \sqrt{t^2 - (1-c^2)}}{2}
\end{equation}
for all $t\geq 1$.

This motivates a new family of thresholding operators, {\em reciprocal thresholding with parameter $c$}, which is designed to make the inequality~\eqref{eqn:sigmalowerbd} an equality. To be specific, we define reciprocal thresholding with parameter $c$ to be
\[\thr^{\textnormal{RT},c}=\Psi_{s;\sigma}\text{ with shrinkage function }
\sigma(t)=\frac{t-\sqrt{t^2-(1-c^2)}}{2}.\]
To apply this operator to some vector $z\in\mathbb{R}^d$, we first let $S\subset\{1,\dots,d\}$ be the indices of the largest $s$ entries of $z$ (with our usual caveat about needing to establish some rule for breaking ties) and let $\tau = \max_{i\not\in S}|z_i|$ be the magnitude of the $(s+1)$-st largest entry of $z$. Then $
\thr^{\textnormal{RT},c}(z)$ operates entry-wise as follows:
\begin{equation}\label{eqn:define_RTc}\big(\thr^{\textnormal{RT},c}(z)\big)_i = \begin{cases}
\textnormal{sign}(z_i)\cdot \left(\frac{1}{2}|z_i| + \frac{1}{2}\sqrt{|z_i|^2 - \tau^2(1-c^2)}\right),&\text{ if $i\in S$,}\\
0,&\text{ if $i\not\in S$.}
\end{cases}\end{equation}
Here the  thresholded value  $\big(\thr^{\textnormal{RT},c}(z)\big)_i$ is equal to the larger-magnitude root $t$ of the equation
\begin{equation}\label{eqn:explain_RTc} z_i = t + \frac{\tau^2 \cdot \frac{1-c^2}{4}}{t},\end{equation}
hence the name ``reciprocal thresholding''.

As before, to avoid the need for selecting $c$ adaptively, we might want to consider some fixed choices. At one extreme, taking $c=1$ yields $\thr^{\textnormal{RT},1} = \thr^{\textnormal{HT}}$, the hard thresholding operator. At the other extreme, taking $c=0$ defines the ``universal'' {\em reciprocal thresholding} operator:
\[\thr^{\textnormal{RT}}=\thr^{\textnormal{RT},0}.\]
For any $z\in\mathbb{R}^d$, $\thr^{\textnormal{RT}}$ operate entry-wise as:
\begin{equation}\label{eqn:define_RT}\big(\thr^{\textnormal{RT}}(z)\big)_i = \begin{cases}
\textnormal{sign}(z_i)\cdot \left(\frac{1}{2}|z_i| + \frac{1}{2}\sqrt{|z_i|^2 - \tau^2}\right),&\text{ if $i\in S$,}\\
0,&\text{ if $i\not\in S$.}
\end{cases}\end{equation}

The following lemma calculates the relative concavity of $\thr^{\textnormal{RT},c}$ and $\thr^{\textnormal{RT}}$ as a direct consequence of Lemma~\ref{lem:unifying_RC}.
\begin{lemma}\label{lem:RC_RT}
For any sparsity proportion $\rho\in(0,1]$, the thresholding operator $\thr^{\textnormal{RT},c}$ with parameter $c=\rho$ has relative concavity equal to
\[\gamma_{s,\rho}(\Psi_s^{\textnormal{RT},\rho})= \frac{\rho}{1+\rho}.\]
The reciprocal thresholding operator $\thr^{\textnormal{RT}}$ has relative concavity equal to
\[\gamma_{s,\rho}(\thr^{\textnormal{RT}})=
\frac{\frac{\rho}{\min\{1,4(1-\rho)\}}}{\frac{1}{2} + \frac{1}{2}\sqrt{1+\frac{4\rho}{\min\{1,4(1-\rho)\}}}} \leq \frac{\rho}{\min\{1,4(1-\rho)\}}\]
for every sparsity proportion $\rho\in(0,1)$.
\end{lemma}
\noindent Thus, $\thr^{\textnormal{RT},c}$ with $c=\rho$ is exactly optimal among all thresholding operators relative to the sparsity proportion $\rho$ (as is $\thr^{\ell_q}$ with $q=\frac{2(1-\rho)}{3-\rho}$), while $\thr^{\textnormal{RT}}$ is near optimal when $\rho$ is small (as is $\thr^{\ell_{2/3}}$).

\subsection{An illustrative comparison}
Through the development in this section, we see that there are three important benchmarks for relative concavity: the bound $\sqrt{\rho}/2$ attained by hard thresholding $\thr^{\textnormal{HT}}$, the bound $\frac{\frac{\rho}{\min\{1,4(1-\rho)\}}}{\frac{1}{2} + \frac{1}{2}\sqrt{1+\frac{4\rho}{\min\{1,4(1-\rho)\}}}}$ attained by reciprocal thresholding $\thr^{\textnormal{RT}}$ and $\ell_{2/3}$ thresholding $\thr^{\ell_{2/3}}$, and the optimal value $\frac{\rho}{1+\rho}$. In this section we provide a comparison between these three values.

The left-hand plot of Figure~\ref{fig:compare_bounds} displays the three values of relative concavity as functions of the sparsity proportion $\rho$. We see that at small values $\rho\approx 0$, the relative concavity of reciprocal thresholding and $\ell_{2/3}$ thresholding is nearly identical to the optimal bound $\frac{\rho}{1+\rho}$, and is substantially better than the relative concavity for hard thresholding, given by $\sqrt{\rho}/2$. At larger values of $\rho$, the relative concavity for hard thresholding is instead lower.

\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{rho_vs_kappa_gamma.pdf}
\caption{A comparison of three values of relative concavity: the optimal relative concavity (attained by, for instance, $\thr^{\textnormal{RT},c}$ with $c=\rho$, or by $\thr^{\ell_q}$ with $q=\frac{2(1-\rho)}{3-\rho}$); the relative concavity obtained by the ``universal'' operators, including $\thr^{\textnormal{RT}}$ and by $\ell_{2/3}$ thresholding; and the relative concavity of hard thresholding. The left plot shows the relative concavity as a function of the sparsity proportion $\rho$. The right plot shows the largest possible condition number $\kappa$ of the objective function $\mathsf{f}$ for which a restricted optimality guarantee can be attained (Theorems~\ref{thm:upperbd} and~\ref{thm:lowerbd} show that $\gamma_{s,\rho}(\Psi_s) \leq \frac{1}{2\kappa}$ is necessary and sufficient for a restricted optimality guarantee).}
\label{fig:compare_bounds}
\end{figure}

To view this comparison in another light, given any fixed thresholding operator $\Psi_s$ with certain relative concavity, and given an objective function $\mathsf{f}$ with condition number $\kappa$, for what sparsity ratio $\rho = s'/s$ is the iterative thresholding algorithm guaranteed to achieve restricted optimality? Using the condition $\gamma_{s,\rho}(\Psi_s)\leq \frac{1}{2\kappa}$, for each relative concavity $\gamma_{s,\rho}$ we can solve for the largest possible $\kappa$ for which restricted optimality is assured, as a function of $\rho$.

 This is illustrated in the right-hand plot of Figure~\ref{fig:compare_bounds}, where we see that the reciprocal thresholding operator $\thr^{\textnormal{RT}}$ and the $\ell_{2/3}$ thresholding operator achieve a nearly-optimal sparsity ratio $\rho$ when the condition number $\kappa$ is large and $\rho$ is correspondingly close to zero, while hard thresholding $\thr^{\textnormal{HT}}$ is closer to optimal for $\kappa$ and $\rho$ close to $1$. Thus, we can conclude that reciprocal thresholding and $\ell_{2/3}$ thresholding offer stronger theoretical guarantees when $\kappa>2$, while hard thresholding may be better for very well-conditioned problems where $1\leq \kappa <2$. (Empirically, we have observed that it is often the case that the three perform nearly identically in ``generic'' problems, and only show substantial differences in problems constructed to mimic our lower bound result, Theorem~\ref{thm:lowerbd}, for example, in linear regression problems where a small subset of the features are generated to have covariance structure similar to the construction in Theorem~\ref{thm:lowerbd}.)

\subsection{A closer look at soft thresholding}\label{sec:softthresh}
In many applications, it is common to use a sparsity-inducing penalty rather than an explicit sparsity constraint. For example,
we may solve
\[\widehat{x} = \arg\min_{x\in\mathbb{R}^d}\big\{\mathsf{f}(x) + \lambda \norm{x}_1\big\},\]
which is known as the Lasso~\citep{tibshirani1996regression} in the context of regression problems.
More generally, we can consider
\begin{equation}\label{eqn:xhatlambda}\widehat{x}_{\lambda} = \arg\min_{x\in\mathbb{R}^d}\big\{\mathsf{f}(x) + \lambda \mathsf{R}(x)\big\},\end{equation}
where $\mathsf{R}:\mathbb{R}^d\rightarrow\mathbb{R}$ is any proper convex function acting as a regularizer.
This class of problems can be solved iteratively with a proximal gradient method,
\begin{equation}\label{eqn:iterative_prox}
x_t = \textnormal{Prox}_{\lambda\eta\mathsf{R}}\big(x_{t-1} - \eta \nabla \mathsf{f}(x_{t-1})\big),
\end{equation}
where for any $t\geq 0$, the proximal map is defined as
\[\textnormal{Prox}_{t\mathsf{R}}(z) = \arg\min_{x\in\mathbb{R}^d}\left\{\frac{1}{2}\norm{x-z}^2_2 + t\mathsf{R}(x)\right\}.\]
Note that convexity of $\mathsf{R}(x)$ ensures continuity of the proximal map. More properties of the proximal map and the proximal method can be found in \citep{parikh2014proximal}.
Examining the iterations of proximal gradient descent~\eqref{eqn:iterative_prox}, we see that it is very similar
to the iterative thresholding method~\eqref{eqn:iterative_thresh} for a fixed sparsity level $s$ (using some particular
thresholding operator $\Psi_s$); we simply replace the thresholding operator $\Psi_s$ with the proximal map $\textnormal{Prox}_{\lambda\eta\mathsf{R}}$.

In particular, if we consider $\mathsf{R}(x) =\norm{x}_1$, the resulting proximal map is known as ``soft thresholding'', and can be
computed with elementwise shrinkage:
\[\big(\textnormal{Prox}_{t\norm{\cdot}_1}(z) \big)_i =\begin{cases}
z_i - t, & z_i > t, \\
0, & |z_i|\leq t,\\
z_i + t, & z_i< -t.\end{cases}\]
Now, recall that in Section~\ref{sec:relativeconcavity_HT_ST}, we considered a ``soft thresholding'' operator at a {\em fixed} sparsity
level $s$, which we can now rewrite as
\[\thr^{\textnormal{ST}}(z) = \textnormal{Prox}_{t\norm{\cdot}_1}(z) \textnormal{\quad where $t\geq 0$ is the smallest value s.t.~$\norm{ \textnormal{Prox}_{t\norm{\cdot}_1}(z) }_0 \leq s$.}\]
We might ask whether the suboptimal worst-case performance of iterative thresholding with the operator $\thr^{\textnormal{ST}}$, as established
by Lemma~\ref{lem:RC_contin} and Theorem~\ref{thm:lowerbd},
is due to the unusual definition of $\thr^{\textnormal{ST}}$, using a fixed sparsity level $s$, rather than the usual form of soft thresholding where
we would iterate~\eqref{eqn:iterative_prox} at a fixed value of $\lambda$ in order to minimize  $\mathsf{f}(x) + \lambda \norm{x}_1$.

In fact, we will now see that this is not the case---even if we use a fixed $\lambda$ rather than a fixed sparsity level $s$,
we can still find worst-case examples where restricted optimality is not achieved.
\begin{theorem}\label{thm:prox_worstcase}
Let $d\geq 2$, let $\beta\geq \alpha >0$, and let $\mathsf{R}:\mathbb{R}^d\rightarrow\mathbb{R}$ be a proper convex function
that satisfies the following assumptions:
\begin{align}
\label{eqn:prox_assump1}&\text{For any $z\in \mathbb{R}^d$ and any $t'>t\geq 0$, if $\norm{\textnormal{Prox}_{t\mathsf{R}}(z)}_{0}<d$ then $\norm{\textnormal{Prox}_{t'\mathsf{R}}(z) }_{0}<d$.}\\
\label{eqn:prox_assump2}&\text{There exist $v,w \in \mathbb{R}^d$ that are both dense, i.e., $\norm{v}_0=\norm{w}_0=d$,  with $w \in \partial \mathsf{R}(v)$.}
\end{align}Then there exists an objective function $\mathsf{f}(x)$ that satisfies $(\alpha,d)$-RSC and $(\beta,d)$-RSM, and a 1-sparse vector $y\in\mathbb{R}^d$, such that
defining $\widehat{x}_\lambda$ as in~\eqref{eqn:xhatlambda},
\[\text{For all $\lambda\geq 0$, either $\norm{\widehat{x}_\lambda}_0=d$ or $\mathsf{f}(\widehat{x}_\lambda) > \mathsf{f}(y)$.}\]
\end{theorem}
\noindent In other words, this result means there is no value of $\lambda$ that produces a solution that is both sparse (at any sparsity level $<d$) and
has an objective function value at least as good as the best 1-sparse solution $y$.

We remark that
our conditions~\eqref{eqn:prox_assump1} and~\eqref{eqn:prox_assump2} on the regularizer $\mathsf{R}$ are satisfied by many common regularizers---for
 example, the $\ell_1$ norm (Lasso), any $\ell_p$ norm for $1\leq p\leq \infty$, the elastic net (a combination
of the $\ell_1$ and $\ell_2$ norms), the weighted $\ell_1$ norm, and many others.
To help interpret the first condition~\eqref{eqn:prox_assump1}, this essentially requires that a sparse solution $\widehat{x}_\lambda$ will stay sparse if we increase
the penalty parameter $\lambda$, as we would expect for any sparsity-promoting regularizer.

This theorem implies that, just as continuous thresholding operators $\Psi_s$ at a fixed level $s$
can fail to attain restricted optimality in a worst-case scenario, the same holds for regularization with convex penalties (such as soft
thresholding
with the $\ell_1$ norm). An open question remains here, namely, is there a measure in the style of relative concavity,
which can characterize the worst-case performance of penalty functions $\mathsf{R}$ (covering both convex and nonconvex penalty functions,
just as relative concavity treats both continuous and non-continuous thresholding operators)?

\section{Iterative thresholding for low-rank matrices}\label{sec:matrix}
We next extend our analysis of iterative thresholding methods to the setting of a low-rank constraint. In fact, our results carry over fully into this setting. Given a rank constraint, $\textnormal{rank}(X)\leq s$, the hard thresholding operator is defined as
\[\mthr^{\textnormal{HT}}: X\mapsto U\cdot\textnormal{diag}(\thr^{\textnormal{HT}}(d))\cdot V^\top,\]
where $X=U\cdot\textnormal{diag}(d)\cdot V^\top$ is the singular value decomposition of $X$.\footnote{In the case of repeated singular values, the singular value decomposition will not be unique, and we assume that we have some mechanism for specifying a specific singular value decomposition. This is analogous to the sparse vector problem, where if the $s$th largest entry in $z$ is not unique, we need to assume some mechanism for breaking ties and choosing the support of the thresholded vector.} That is, hard thresholding is performed on the singular values of the matrix $X$, rather than on its entries. Of course, we can extend this to any thresholding operator---given any $\Psi_s:\mathbb{R}^{\min\{n,m\}}\rightarrow\{x\in\mathbb{R}^{\min\{n,m\}}:\norm{x}_0\leq s\}$, we can ``lift'' this thresholding operator to the matrix setting by defining
\begin{equation}\label{eqn:mthr_lift}\widetilde\Psi_s:X\mapsto U\cdot \textnormal{diag}(\Psi_s(d))\cdot V^\top.\end{equation}
Of course, its possible to construct a rank-$s$ thresholding operator $\widetilde\Psi_s$ that is not of the form given in~\eqref{eqn:mthr_lift}, for example, if $\widetilde\Psi_s$ does not preserve the left and right singular vectors of $Z$.

We next extend our convergence results, Theorems~\ref{thm:upperbd} and~\ref{thm:lowerbd}, to the low-rank setting. In order to do so, we need to define the matrix version of relative concavity---this definition is analogous to the vector case, with rank constraints in place of sparsity constraints:
\[\widetilde\gamma_{s,\rho}(\widetilde\Psi_s) = \sup\left\{\frac{\inner{Y-\widetilde\Psi_s(Z)}{Z-\widetilde\Psi_s(Z)}}{\fronorm{Y-\widetilde\Psi_s(Z)}^2} \ : Y,Z\in\mathbb{R}^{n\times m}, \textnormal{rank}(Y)\leq \rho s, Y\neq \widetilde\Psi_s(Z)\right\}.\]
As for the vector case, relative concavity is necessary and sufficient for guaranteeing restricted optimality---in fact, the proofs of these are completely identical to the vector case. For completeness, we state the results here, for the matrix version of the iterated thresholding algorithm:
\begin{equation}\label{eqn:iterative_thresh_matrix}
X_t = \widetilde\Psi_s\big(X_{t-1} - \eta_t \nabla \mathsf{f}(X_{t-1})\big),
\end{equation}
with either fixed step size $\eta_t=1/\beta$ or adaptive step size defined as in~\eqref{eqn:iterative_thresh_eta_adaptive}.
\begin{theorem}\label{thm:matrix_upperbd}
Consider any objective function $\mathsf{f}:\mathbb{R}^{n\times m}\rightarrow \mathbb{R}$, any ranks $s\geq s'$, and any rank-$s$ thresholding operator $\widetilde\Psi_s$. Assume the objective function $\mathsf{f}$ satisfies $(\alpha,s)$-RSC and $(\beta,s)$-RSM relative to the rank constraint.\footnote{In the low-rank setting, the RSC and RSM conditions are defined with rank in place of sparsity---specifically, we are assuming that $\frac{\alpha}{2}\fronorm{X-Y}^2 \leq \mathsf{f}(Y)- \mathsf{f}(X) - \inner{\nabla \mathsf{f}(X)}{Y-X} \leq \frac{\beta}{2}\fronorm{X-Y}^2$ whenever $\textnormal{rank}(X)\leq s,\textnormal{rank}(Y)\leq s$.}
Let $\rho = s'/s$ and $\kappa = \beta/\alpha$, and assume that
$\widetilde\gamma_{s,\rho}(\widetilde\Psi_s) < \frac{1}{2\kappa}$.
Then, for any $X_0,Y\in\mathbb{R}^{n\times m}$ with $\textnormal{rank}(X_0)\leq s$ and $\textnormal{rank}(Y)\leq s'$, the iterated thresholding algorithm~\eqref{eqn:iterative_thresh_matrix} run with step size $\eta=1/\beta$ and initialization point $X_0$ satisfies
\[\min_{t=1,\dots,T} \mathsf{f}(X_t) \leq \mathsf{f}(Y) + \left(\frac{1 - 1/\kappa}{1-2\widetilde\gamma_{s,\rho}(\widetilde\Psi_s)}\right)^T \cdot \frac{\beta}{2} \fronorm{X_0 - Y}^2\]
for each $T\geq 1$.
\end{theorem}
\begin{theorem}\label{thm:matrix_lowerbd}
Consider any ranks $s\geq s'$, any rank-$s$ thresholding operator $\widetilde\Psi_s$, and any constants $\beta \geq \alpha >0$. Let $\rho=s'/s$ and $\kappa = \beta/\alpha$, and assume that
$\widetilde\gamma_{s,\rho}(\widetilde\Psi_s)>\frac{1}{2\kappa}$.
Then there exists an objective function $\mathsf{f}(X)$ that satisfies $(\alpha,s)$-RSC and $(\beta,s)$-RSM relative to the rank constraint, and matrices $X_0,Y\in\mathbb{R}^{n\times m}$ with $\textnormal{rank}(X_0)\leq s$ and $\textnormal{rank}(Y)\leq s'$, such that the iterated thresholding algorithm~\eqref{eqn:iterative_thresh_matrix} run with step size $\eta=1/\beta$ and initialization point $X_0$ satisfies
\[\lim_{t\rightarrow \infty} \mathsf{f}(X_t) > \mathsf{f}(Y).\]
\end{theorem}
\noindent In other words, just as for the sparse optimization problem, the relationship between relative concavity and condition number gives a necessary and sufficient condition for guaranteed convergence. We note that these results apply to {\em any} rank-$s$ thresholding operator $\widetilde\Psi_s$, whether or not it can be constructed by ``lifting'' a $s$-sparse thresholding operator as in~\eqref{eqn:mthr_lift}.

Next, how can we calculate relative concavity of a thresholding operator in the matrix setting? For simplicity, from this point on we assume that we are working with ranks $s\geq s'\geq 1$ with $s+s'\leq \min\{n,m\}$. For this question, we will again see that results from the sparse setting transfer to the low-rank setting. First, we have the same lower bound uniformly over all operators:
\begin{lemma}\label{lem:RC_lowerbd_matrix}
For any map $\widetilde\Psi_s: \mathbb{R}^{n\times m} \rightarrow \{X\in\mathbb{R}^{n\times m}:\textnormal{rank}(X)\leq s\}$ and any sparsity proportion $\rho\in(0,1]$,
the relative concavity is lower-bounded as
\[\widetilde\gamma_{s,\rho}(\widetilde\Psi_s)\geq \frac{\rho}{1+\rho}.\]
\end{lemma}
\noindent Furthermore, if we restrict our attention to ``lifted'' thresholding operators of the form~\eqref{eqn:mthr_lift}, the relative concavity of $\Psi_s$ is inherited by the lifted operator $\widetilde\Psi_s$---as long as we restrict ourselves to $s$-sparse thresholding operators $\Psi_s$ that satisfy a natural sign condition:
\begin{equation}\label{eqn:thr_sign}
\text{For any $z\in\mathbb{R}^d$ and any $a\in\{\pm 1\}^d$, $\Psi_s\big(\textnormal{diag}(a)\cdot z\big) = \textnormal{diag}(a)\cdot \Psi_s(z)$.}
\end{equation}
This effectively means that $\Psi_s(z)$ preserves the signs of $z$, but the signs of $z$ do not affect the amount of shrinkage in the thresholded vector $\Psi_s(z)$. For example, this requires that $\Psi_s(-z) = - \Psi_s(z)$. Under this assumption, the relative concavity of $\Psi_s$ carries over into the matrix setting.
\begin{lemma}\label{lem:matrix_RC}
Let $\Psi_s$ be a $s$-sparse thresholding operator satisfying the sign condition~\eqref{eqn:thr_sign}, and let $\widetilde\Psi_s$ be the lifted thresholding operator defined in~\eqref{eqn:mthr_lift}. Then for every sparsity proportion $\rho\in(0,1]$,
\[\widetilde\gamma_{s,\rho}(\widetilde\Psi_s) = \gamma_{s,\rho}(\Psi_s).\]
\end{lemma}
\noindent It is obvious that all the thresholding operators we have considered satisfy the sign condition~\eqref{eqn:thr_sign}. Thus, all the results of relative concavity that we have proved in the sparse setting, carry over directly to the low-rank setting. In particular, as for the sparse setting, the hard thresholding operator has relative concavity
\[\widetilde\gamma_{s,\rho}(\mthr^{\textnormal{HT}}) = \frac{\sqrt{\rho}}{2},\]
while any thresholding operator $\widetilde\Psi_{s;\sigma}$ constructed with some shrinkage function $\sigma$ satisfying $\sigma(1)=1/2$  and the conditions of Lemma~\ref{lem:unifying_RC}, such as the reciprocal thresholding operator, $\mthr^{\textnormal{RT}}$, or $\ell_q$ thresholding with $q=2/3$, $\widetilde\Psi_s^{\ell_{2/3}}$, satisfy
\[\widetilde\gamma_{s,\rho}(\widetilde\Psi_{s;\sigma})=\widetilde\gamma_{s,\rho}(\widetilde\Psi_s^{\ell_{2/3}})=\widetilde\gamma_{s,\rho}(\mthr^{\textnormal{RT}}) \leq \frac{\rho}{\min\{1,4(1-\rho)\}}.\]
If the desired rank proportion $\rho=s'/s$ is fixed in advance, then as before, choosing reciprocal thresholding with parameter $c=\rho$, or $\ell_q$ thresholding with $q = \frac{2(1-\rho)}{3-\rho}$, we again obtain the optimal relative concavity of $\frac{\rho}{1+\rho}$. As before, we can conclude that reciprocal thresholding and $\ell_{2/3}$ each offer lower relative concavity than hard thresholding whenever $\rho$ is small---and, correspondingly, are a safer choice for objective functions $\mathsf{f}$ whose condition number is not close to $1$.

\section{Sparse linear regression}\label{sec:linear_regression}
Now that we have discussed the deterministic optimization setting in depth, it is natural to ask what is the implication of these guarantee for a statistically random setting. In this section, we apply our developed machinery to the concrete statistical setting of sparse linear regression. We work with the Gaussian linear model
\begin{equation}\label{eqn:linearmodel}
y=X\theta_0+z
\end{equation}
where $X\in \mathbb{R}^{n\times p}$ is a fixed design matrix, $\theta_0\in \mathbb{R}^p$ is the true coefficient vector assumed to be fixed and $s_0$-sparse, and $z\sim N(0,\sigma^2\mathbf{I}_n)$ is the noise vector, with fixed unknown noise level $\sigma^2>0$. In this section we will mainly be interested in prediction error, i.e. how well we can estimate the true mean vector $X\theta_0$. One way of capturing the conditioning of the design matrix is by the following definition: at some given sparsity level $s$, we define a set of design matrices $\mathcal{X}(\alpha,\beta,s)$ as
\begin{equation}\label{eqn:Xset}
\mathcal{X}(\alpha,\beta,s)=\left\{X\in\mathbb{R}^{n\times p}:\textnormal{ the map $\theta\mapsto \theta^\top \left(\frac{X^\top X}{2n}\right)\theta$ satisfies $(\alpha,s)$-RSC and $(\beta,s)$-RSM}\right\}.
\end{equation} As usual, we will be interested in the condition number $\kappa = \beta/\alpha$.  A similar definition is the {\em restricted eigenvalue} condition on the design matrix $X$, which constrains $X$ to the following set
\begin{multline}\label{eqn:Xset_RE}
\mathcal{X}_{\textnormal{RE}}(\kappa,s_0)=\bigg\{X\in\mathbb{R}^{n\times p}:\textnormal{ $\max_{j=1,\dots,p}\frac{\norm{X_j}_2}{\sqrt{n}}\leq 1$, and $\theta^\top \left(\frac{X^\top X}{2n}\right)\theta\geq \frac{1}{2\kappa}\norm{\theta}^2_2$}\\\text{ for all $\theta\in\mathbb{R}^d$ with $\norm{\theta}_1\leq 4\max_{|S|=s_0}\norm{\theta_S}_1 $}\bigg\}.
\end{multline} To gain some intuition for when these conditions may hold, for a design matrix $X$ whose rows are i.i.d.~draws from a normal distribution $N(0,\Sigma)$, \citet[Theorem 1]{raskutti2010restricted} show that the population-level eigenvalues of the covariance $\Sigma$ are approximately preserved in the design matrix, at any sparsity level $s\ll \frac{n}{\log(p)}$.

\paragraph{Computational lower bound}
In terms of prediction error, the optimal method, $\ell_0$ constrained least squares method, is not computable. Thus from the lower bound side, it is of interest to ask what is the lowest prediction error achievable in the class of computational feasible estimator. Recently, \citet{zhang2014lower} provide a partial answer to this question, restricting to the class of $s_0$ sparse estimator. Their main result (see Theorem $1$ in \citet{zhang2014lower}) states the following (informally):
\begin{quote}
  Under the assumption that $\emph{NP}\nsubseteq \emph{P}\setminus \emph{poly}$, for any $\delta\in (0,1)$, under some assumption on $n,d,s_0$ and for any $\kappa$ in a wide range, there exists a design matrix $X\in \mathcal{X}_{\textnormal{RE}}(\kappa,s_0)$ such that for any computational efficient methods, the maximum prediction error (over all $s_0$ sparse $\theta_0$) is lower bounded by (up to some constant) $\kappa \cdot \frac{\sigma^2 s_0^{1-\delta}\log(d)}{n}$.
\end{quote}
Thus if we restrict ourselves to all computationally feasible $s_0$ sparse estimator, then the best achievable squared prediction error is of order $\kappa \cdot \frac{\sigma^2 s_0\log(d)}{n}$.

\paragraph{Upper bounds for iterative thresholding methods}
In this section we establish prediction error bounds for iterative thresholding algorithm. First we provide some intuition on how to connect restricted optimality guarantee with statistical performance. It is well known that the global optimum of $\ell_0$-constrained least squared loss, i.e.
\[\widehat{\theta}\in\arg\min_{\norm{\theta}_0\leq s_0}\norm{y-X\theta}^2_2,\]
achieves a squared prediction error scaling as $\frac{\sigma^2s_0\log(d)}{n}$. For iterative thresholding algorithms, since we only have restricted optimality rather than global optimality, we are forced to work over a constraint at a larger sparsity $s\geq s_0$ to guarantee $\norm{y-X
\widehat\theta}^2_2\leq\min_{\norm{\theta}_0\leq s_0}\norm{y-X\theta}^2_2$.
The statistical price one has to pay for this computational strategy is the inflation in noise level corresponding to the inflation in sparsity---that is, we have error on $s$ many nonzero coefficients, rather than $s_0$ many---so the final upper bound for prediction error would scale as $\frac{\sigma^2s\log(d)}{n}$ instead of $\frac{\sigma^2s_0\log(d)}{n}$, where $s$ is chosen to be the smallest sparsity level that guaratees restricted optimality relative to the lower sparsity level $s'=s_0$. Now recall from Section~\ref{sec:RC} that, while hard thresholding offers restricted optimality guarantees at sparsity levels $s\sim \kappa^2s_0$, the optimal and near-optimal thresholding operators (for example reciprocal thresholding and $\ell_{2/3}$ thresholding) improves this scaling to $s\sim \kappa s_0$. This allows us to improve the upper bound for squared prediction error from scaling as $\kappa^2$ to $\kappa$, when we switch our method from iterative hard thresholding, to iterative thresholding with an operator $\Psi_s$ that enjoys a lower relative concavity. Indeed in \citet{jain2014iterative}, it is shown that iterative hard thresholding achieves a prediction error upper bounded by $\kappa^2\cdot\frac{\sigma^2s_0\log(d)}{n}$. In view of our lower bound result Theorem~\ref{thm:lowerbd}, which states that the restircted optimality guarantee is tight, we postulate that the corresponding prediction error bound is also tight for iterative hard thresholding method.

Now we formulate this rigorously. Consider the iterative thresholding algorithm with some thresholding operator $\Psi_s$ applied to the objective function $\mathsf{f}(\theta) = \frac{1}{2n}\norm{y-X\theta}^2_2$, whose iteration takes the form
\begin{equation}\label{eqn:RT_linearreg}\widehat\theta_t = \Psi_s\big(\widehat\theta_{t-1} + \eta_t \cdot\frac{1}{n} X^\top (y - X\widehat\theta_{t-1})\big).\end{equation}
As usual, for the step size we may choose $\eta_t = 1/\beta$ if $\beta$ is known, or we may choose $\eta_t$ adaptively as in~\eqref{eqn:iterative_thresh_eta_adaptive}. We will work with any thresholding operator $\Psi_s$ satisfying
\begin{equation}\label{eqn:RC_rho_for_regr}\gamma_{s,\rho}(\Psi_s)\leq \rho\text{ for all $\rho\in(0,1/2)$}.\end{equation}
From Section~\ref{sec:RC}, we see that on the one hand, this condition rules out hard thresholding and any continuous thresholding operator; on the other hand, it is satisfied by the reciprocal thresholding operator, $\thr^{\textnormal{RT}}$, by $\ell_q$ thresholding with $q=2/3$, $\thr^{\ell_{2/3}}$, and by any shrinkage operator $\Psi_{s;\sigma}$ where $\sigma(1)=1/2$ and $\sigma$ satisfies the conditions of Lemma~\ref{lem:unifying_RC}. We now present our result for this setting:

\begin{theorem}\label{thm:RT_linearreg}
Suppose that $y = X\theta_0 + N(0,\sigma^2\mathbf{I}_n)$, where $\theta_0$ is $s_0$-sparse, and where $X\in\mathcal{X}(\alpha,\beta,s)$, where $s = C\kappa s_0$ for some $C>2$. Suppose that $\Psi_s$ is any $s$-sparse thresholding operator satisfying~\eqref{eqn:RC_rho_for_regr}.

Let $\widehat\theta_t$ be the estimate produced at step $t$ of the iterative thresholding algorithm~\eqref{eqn:RT_linearreg} initialized at some $s$-sparse $\widehat\theta_0\in\mathbb{R}^d$. Let $\tilde\theta_t \in\arg\min_{\theta\in\{\widehat\theta_1,\dots,\widehat\theta_t\}}\frac{1}{2n}\norm{y-X\theta}^2_2$, that is, $\tilde\theta_t$ is the best estimate seen before time $t$, relative to the loss function $\mathsf{f}(\theta) = \frac{1}{2n}\norm{y-X\theta}^2_2$.

Then, for any $\delta>0$ and any $t\geq 1$,
\[\frac{1}{n}\norm{X(\tilde{\theta}_t-\theta_0)}^2_2 \leq \kappa\cdot \frac{28C \sigma^2 s_0\log(d) }{n} +\frac{12\sigma^2 \log(1/\delta)}{n} +  \left(\frac{1-1/\kappa}{1-2/C\kappa}\right)^t \cdot2\beta \norm{\widehat\theta_0 -\theta_0}^2_2,\]
with probability at least $1-\delta$.
\end{theorem}
\noindent  Since $t$ can be taken to be large (each iteration is very cheap), the dominant term is the first one, so we essentially have
\[\frac{1}{n}\norm{X(\tilde\theta_t - \theta_0)}^2_2 \lesssim \kappa\cdot \frac{\sigma^2 s_0\log(d) }{n}.\]
Comparing with the upper bound for iterative hard thresholding, we see that we now attains the ideal $\kappa$, rather than $\kappa^2$, scaling.

\paragraph{Comparison with Lasso}
The Lasso estimate of $\theta_0$, given by the convex optimization problem \[\widehat{\theta}\in\arg\min_{\theta\in\mathbb{R}^d}\left\{\frac{1}{2}\norm{y-X\theta}^2_2+\lambda\norm{\theta}_1\right\},\]
is proved in \citet{bickel2009simultaneous} to achieve a squared prediction error bounded as
\begin{equation}\label{eqn:lasso_upperbd}\frac{1}{n}\norm{X(\widehat{\theta}-\theta_0)}^2_2 \lesssim \kappa \cdot \frac{\sigma^2 s_0\log(d)}{n}\end{equation}
with a penalty parameter value $\lambda\sim \sigma\sqrt{\frac{\log(d)}{n}}$, under the assumption that $X\in \mathcal{X}_{\textnormal{RE}}(\kappa,s_0)$. Compared with Lasso, due to Theorem~\ref{thm:RT_linearreg}, iterative thresholding algorithms with proper thresholding operators, for example the simple and efficient reciprocal thresholding, achieve the same squared prediction error bound. Moreover, both Lasso and iterative reciprocal thresholding method are guaranteed to give an estimator that is $\mathcal{O}(\kappa s_0)$ sparse (this sparsity level for Lasso is proved in \citet[Eqn. (7.9)]{bickel2009simultaneous}), and thus nearly match the computational lower bound with a gap in sparsity. An open question for future work is whether the larger sparsity level, i.e.~$\mathcal{O}(\kappa s_0)$ rather than $s_0$, is unavoidable to achieve the squared prediction error $\kappa \cdot \frac{\sigma^2 s_0\log(d)}{n}$, or whether there may be an $\mathcal{O}(s_0)$-sparse and computationally efficient estimator that achieves this bound.

\section{Discussion}

Relative concavity offers a framework for comparing theoretical properties of thresholding operators. Under this framework, we find a general class of optimal and near-optimal thresholding operators, among which is the new reciprocal thresholding operator, an alternative to hard and soft thresholding with tighter theoretical guarantees that is able to achieve better dependence on condition number for sparse and low-rank optimization problems.

Nonetheless, many open questions remain for these problems. For example, our upper and lower bounds on $\lim_{t\rightarrow \infty}\mathsf{f}(x_t)$ are proved relative to a broad class of functions satisfying (restricted) convexity and smoothness properties, with no underlying statistical model. In a statistical framework, we may be able to make additional assumptions, for instance, assuming that $\nabla\mathsf{f}(y)$ is small at some highly sparse $y$ (e.g.~if $y$ is the true model parameter vector, while $\mathsf{f}$ is the negative log-likelihood on the observed data)---is the relative concavity still necessary and sufficient for optimization guarantees, or would we observe different behavior of the various thresholding operators in this statistical setting?

Relatedly, the relative concavity characterizes the restricted optimality guarantee of a thresholding operator on the worst-case objective function.
In practice we may be more interested in the average-case convergence behavior of a thresholding operator, if the objective function $\mathsf{f}$ arises from
some underlying statistical model or random process. Furthermore, how does the choice of the thresholding operator interact with modifications of the gradient descent algorithm, such as decreasing step size, choosing the step size via backtracking or another adaptive method, acceleration of the gradient descent step, replacing gradients with stochastic gradients, or using second-order information? We hope to address these directions in future work.

\subsection*{Acknowledgements}
R.F.B.~was partially supported by the National Science Foundation via grant DMS-1654076, and by an Alfred
P.~Sloan fellowship. The authors are grateful
to Chao Gao for helpful discussions and feedback on this work.

\bibliographystyle{plainnat}

Methodology (stat.ME); Statistics Theory (math.ST); Computation (stat.CO); Machine Learning (stat.ML)