Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problem

Abstract: Deep learning (DL) has had unprecedented success and is now entering scientific computing with full force. However, current DL methods typically suffer from instability, even when universal approximation properties guarantee the existence of stable neural networks (NNs). We address this paradox by demonstrating basic well-conditioned problems in scientific computing where one can prove the existence of NNs with great approximation qualities, however, there does not exist any algorithm, even randomised, that can train (or compute) such a NN. For any positive integers $K > 2$ and $L$, there are cases where simultaneously: (a) no randomised training algorithm can compute a NN correct to $K$ digits with probability greater than $1/2$, (b) there exists a deterministic training algorithm that computes a NN with $K-1$ correct digits, but any such (even randomised) algorithm needs arbitrarily many training data, (c) there exists a deterministic training algorithm that computes a NN with $K-2$ correct digits using no more than $L$ training samples. These results imply a classification theory describing conditions under which (stable) NNs with a given accuracy can be computed by an algorithm. We begin this theory by establishing sufficient conditions for the existence of algorithms that compute stable NNs in inverse problems. We introduce Fast Iterative REstarted NETworks (FIRENETs), which we both prove and numerically verify are stable. Moreover, we prove that only $\mathcal{O}(|\log(\epsilon)|)$ layers are needed for an $\epsilon$-accurate solution to the inverse problem.

\section{Introduction}\label{sec_intro_lab}
Deep learning (DL) has demonstrated unparalleled accomplishments in fields ranging from image classification and computer vision \cite{krizhevsky2012imagenet,he2016deep,girshick2014rich}, voice recognition and automated diagnosis in medicine \cite{hinton2012deep,ma2015deep,dahl2011context}, to inverse problems and image reconstruction \cite{jin17,mccann2017convolutional,hammernik2018learning,arridge2019solving} (this list of areas is by no means exhaustive, see \cite{LeCun_Nature}, for example). However, at the same time, there is now overwhelming empirical evidence that DL leads to unstable methods, a phenomenon that seems universal and present in all the applications listed above \cite{SzZ-14,moosavi2016deepfool,akhtar2018threat,carlini2018audio,huang2018some,antun2020instabilities, finlayson2019adversarial} and in most of the new artificial intelligence (AI) technologies. These instabilities are often detected by what has been commonly known in the literature as ``adversarial attacks''. Moreover, the instabilities can be present even in random cases and not just worst-case scenarios \cite{gottschling2020troublesome} (see also ``stealth attacks'' introduced in \cite{tyukin2020adversarial} which concern perturbations to the AI system itself). There is a growing awareness of this problem in high-stake applications and society as a whole \cite{finlayson2019adversarial,baker2019workshop,hamonrobustness}, and instability seems to be the Achilles' heel of modern AI and DL -- see Figure \ref{fig:automap_pert} for an example (which is to be contrasted with Figure \ref{fig:automap_pert3}). Classical approximation theoretical results prove, however, that a continuous function can be approximated arbitrarily well by a neural network (NN). Thus, a stable problem described by a stable function can always be solved stably with a NN. This leads to the basic question:
\begin{displayquote}
	\normalsize
	{\it Why does DL lead to universally unstable methods even when one can prove that stable and accurate neural networks exist?}
\end{displayquote}

It is important to note that an existence proof of NNs does not immediately imply that they can be constructed by an algorithm. In particular, we are faced with the following fundamental problem:
\begin{displayquote}
	\normalsize
	({\bf Question I: Can neural networks that provably exist be trained/computed?}) {\it
	Let $\Omega_{\mathcal{T}}$ be a collection of training data $\iota = \{x_j\}$, and suppose that for each $\iota \in \Omega_{\mathcal{T}}$ one can prove that there exists a NN $\Phi_{\iota}$ (with certain properties). In particular, there is a mapping $\mathcal{K}\colon  \Omega_{\mathcal{T}} \rightarrow \mathcal{N}$ (a particular collection of NNs).  Does there exist an algorithm, taking $\iota \in \Omega_{\mathcal{T}}$ as input, that can compute an approximation to $\mathcal{K}(\iota) = \Phi_{\iota}$ for all $\iota \in \Omega_{\mathcal{T}}$, and to what accuracy?}
\end{displayquote}

\noindent Question I is the key problem that we address (see \S \ref{sec:summary_main} for a detailed summary), and our barriers, as well as positive results, help shed light on why the desired NNs that exist can or cannot be computed.

\begin{remark}[{\bf Stability/accuracy trade-off and existence of algorithms}] It is, of course, not difficult to compute stable NNs: the zero network is stable, however, not particularly useful. Hence, the big problem is to compute accurate and stable NNs. Scientific computing is based on two pillars: stability and accuracy, however, there is often a trade-off between the two, see for example \cite{platte2011impossibility}. Indeed, there may be barriers preventing the existence of accurate and stable algorithms. In particular, there is a trade-off between accuracy and stability, and some accuracy may have to be sacrificed in order to secure stability.\hfill$\boxtimes$
\end{remark}

 \begin{figure}
    \setlength{\tabcolsep}{2pt}
    \resizebox{1\textwidth}{!}{
    \begin{tabular}{@{}>{\centering}m{0.25\textwidth}>{\centering}m{0.25\textwidth}>{\centering}m{0.25\textwidth}>{\centering\arraybackslash}m{0.25\textwidth}@{}}
 Original  $x$ & $ |x + r_1|$ & $ |x + r_2|$ & $ |x + r_3|$\\
     \includegraphics[width=0.25\textwidth]{runner_19_r_idx_0_noisy.png}
    &\includegraphics[width=0.25\textwidth]{runner_19_r_idx_1_noisy.png}
    &\includegraphics[width=0.25\textwidth]{runner_19_r_idx_2_noisy.png}
    &\includegraphics[width=0.25\textwidth]{runner_19_r_idx_4_noisy.png}\\
     $ \Psi(A(x))$ &  $\Psi(A(x+r_1))$ &  $\Psi(A(x + r_2))$  & $\Psi(A(x + r_3))$\\
     \includegraphics[width=0.25\textwidth]{runner_19_r_idx_0_noisy_rec.png}
    &\includegraphics[width=0.25\textwidth]{runner_19_r_idx_1_noisy_rec.png}
    &\includegraphics[width=0.25\textwidth]{runner_19_r_idx_2_noisy_rec.png}
    &\includegraphics[width=0.25\textwidth]{runner_19_r_idx_4_noisy_rec.png}\\
\end{tabular}
    }
\caption{({\bf Unstable neural network in image reconstruction}). The neural network AUTOMAP ({\it Nature} (2018) \cite{Bo-18}) represents the tip if the iceberg of DL in inverse problems. The paper promises that one can ``... observe superior immunity to noise...''. Moreover, the follow-up announcement ({\it Nature Methods} ``{\it AI transforms image reconstruction}," \cite{Str-18}) proclaims: ``A deep-learning-based approach improves speed, accuracy and robustness of biomedical image reconstruction". However, the figure shows $|x+r_j|$, where $x$ is the original image and the $r_j$s are perturbations meant to simulate worst-case effect, as well as the that AUTOMAP reconstruction $\Psi(A(x+r_j))$ from the subsampled Fourier MRI data $A(x+r_j)$ (here $A \in \mathbb{C}^{m \times N}$ is a subsampled Fourier transform, see \S \ref{nohonionavfo} for details) concluding that this network is completely unstable. Note that the condition number $\mathrm{cond}(AA^*) = 1$, so the instabilities are not caused by poor condition. As demonstrated in \cite{antun2020instabilities}, this is a universal phenomenon in DL for inverse problems. Experimental details are given in \S \ref{nohonionavfo}.}\label{fig:automap_pert} \vspace{-2mm}
\end{figure}

\begin{figure*}
    \centering
    \setlength{\tabcolsep}{2pt}
    \resizebox{1\textwidth}{!}{
    \begin{tabular}{@{}>{\centering}m{0.25\textwidth}>{\centering}m{0.25\textwidth}>{\centering}m{0.25\textwidth}>{\centering\arraybackslash}m{0.25\textwidth}@{}}
     Original  $x$ &  $| x+v_1 |$ &  $ |x + v_2| $  & $ |x + v_3| $\\
     \includegraphics[width=0.25\textwidth]{im_rID_19_im_nbr_3_experi_001_orig_noiseless.png}
    &\includegraphics[width=0.25\textwidth]{im_rID_19_im_nbr_3_experi_001_pert_nbr_1_orig_p_noise.png}
    &\includegraphics[width=0.25\textwidth]{im_rID_19_im_nbr_3_experi_001_pert_nbr_2_orig_p_noise.png}
    &\includegraphics[width=0.25\textwidth]{im_rID_19_im_nbr_3_experi_001_pert_nbr_4_orig_p_noise.png}\\
     $ \Phi(A(x))$ &  $\Phi(A(x+v_1))$ &  $\Phi(A(x + v_2))$  & $\Phi(A(x + v_3))$\\
     \includegraphics[width=0.25\textwidth]{im_rID_19_im_nbr_3_experi_001_rec_noiseless.png}
    &\includegraphics[width=0.25\textwidth]{im_rID_19_im_nbr_3_experi_001_pert_nbr_1_rec.png}
    &\includegraphics[width=0.25\textwidth]{im_rID_19_im_nbr_3_experi_001_pert_nbr_2_rec.png}
    &\includegraphics[width=0.25\textwidth]{im_rID_19_im_nbr_3_experi_001_pert_nbr_4_rec.png}\\
    \end{tabular}
    }
    \caption{(\textbf{The FIRENET is stable to worst-case perturbations}). Using the same method as in Figure \ref{fig:automap_pert}, we compute perturbations $v_j$ in the image domain, to simulate worst-case effect for the FIRENET $\Phi \colon \mathbb{C}^m \to \mathbb{C}^{N}$. Here $x$ and $A \in \mathbb{C}^{m\times N}$ are the same image and sampling matrix as in Figure \ref{fig:automap_pert}. Moreover, for each $j=1,2,3$ we have ensured that $\|v_j\|_{l^2} \geq \|r_j\|_{l^2}$, where the $r_j$'s are the perturbations from Figure \ref{fig:automap_pert} (we have denoted the perturbations for FIRENET by $v_j$ to emphasise the fact that these adversarial perturbations are sought for the new NNs and have nothing to do with the perturbations in Figure \ref{fig:automap_pert}). In the top row, we see the perturbed images $|x + v_j|$, $j=0,1,2,3$ (assuming $v_0=0$), and in the bottom row, we see the network's reconstruction from the perturbed measurements $A(x + v_j)$. Experimental details are given in \S \ref{nohonionavfo}.}\label{fig:automap_pert3}
\end{figure*}

\subsection{Hilbert's program, Smale's 18th problem and the foundations of AI}
The strong optimism regarding the abilities of AI and DL is summarised in The New Yorker's (April 2017 issue) quote of G. Hinton: ``\textit{They should stop training radiologists now.}'' One can argue that this optimism is comparable to the optimism about mathematics in the early 20th century, led by D. Hilbert. Hilbert believed that mathematics could prove or disprove any statement and, moreover, that there were no restrictions on which problems could be solved by algorithms. The latter being emphasised in Hilbert's 10th problem \cite{matiisevich1993hilbert}: ``Find an algorithm to determine whether a given polynomial Diophantine equation with integer coefficients has an integer solution.'' Hilbert did not consider the case that such an algorithm may not exist (even though, indeed, no such algorithm exists), suggesting a substantial optimism on what mathematics and algorithms can solve. However, Hilbert was also well aware that such foundations were not established in mathematics and initiated a vast program leading to the beginning of modern logic and subsequently modern computer science.

G\"{o}del \cite{godel1931formal} and Turing \cite{turing1937computable} turned Hilbert's optimism upside down by their foundational contributions establishing impossibility results on what mathematics and digital computers can achieve \cite{poonen2014undecidable}. We argue that a program on the foundations on AI, similar to Hilbert's program, is needed, where impossibility results are provided in order to establish the boundaries of DL and AI.

Note that such a program is already suggested in Smale's 18th problem, from the list of mathematical problems for the 21st century \cite{21century_Smale}, which echoes Turing's paper from 1950 \cite{Turing_1950} on the question: what is AI? Turing asks if a computer can think, and suggests the imitation game as a test for his question about AI. Smale takes the question even further and asks in his 18th problem:
\begin{displayquote}
\normalsize
{\it ``What are the limits of intelligence, both artificial and human?''}
\\
\rightline{ --- Smale's 18th problem (from the list of mathematical problems for the 21st century \cite{21century_Smale}) \hspace{15mm}}
\end{displayquote}
The question is followed by a discussion on the problem that ends as follows: {\it ``Learning is a part of human intelligent activity. The corresponding mathematics is suggested by the theory of repeated games, neural nets and genetic algorithms.''}
Given the recent unprecedented developments in DL and NNs \cite{LeCun_Nature}, and the impact these developments may have on AI, it is timely to consider Smale's 18th problem. We interpret the words ``artificial intelligence'' as the current state-of-the-art AI for which DL is essential. Our results provide foundations for Smale's 18th problem as they imply a potentially vast classification theory for determining the limits of what DL can achieve. Importantly, this classification theory cannot be determined by the extensive collection of non-constructive existence theorems (\`a la universal approximation theorems) for NNs that have flourished over the last decades.

\subsection{Summary of the main results}\label{sec:summary_main}
Our main results demonstrate that there are fundamental barriers preventing NNs, despite their existence, from being computed by algorithms. This helps shed light on the intricate question on why current algorithms in DL produce unstable networks despite the fact that stable NNs often exist in the particular application. Indeed, our results demonstrate that there is a rich and unknown classification theory on which types of stable NNs can be computed by algorithms. The techniques for proving the barriers below stem from the Solvability Complexity Index (SCI) hierarchy -- that has recently been used to settle longstanding questions in scientific computing \cite{ben2015can,Hansen_JAMS,colbrook2019compute,colbrook2019infinite,Jonathan_res,benartzi2020computing,opt_big,colbrook2020foundations} -- and that generalises the fundamental problems of S. Smale on existence of algorithms \cite{smale_question,Smale2, Smale_Acta_Numerica, BSS_Machine, BCSS} and the work by C. McMullen \cite{McMullen1,mcmullen1988braiding,mcmullen_int} and P. Doyle \& C. McMullen \cite{Doyle_McMullen}.
\begin{itemize}[leftmargin=1mm]
\item[(I)] {\bf(Neural networks may exist, but cannot be computed, even for well-conditioned problems).} The answer to the above Question I is, in general, `no', even for well-conditioned problems. Mappings that take training data to NNs may exist, however, no algorithm that computes the NN from the training data exists. This statement is made precise in Theorems \ref{thrm:exists} and \ref{impossibility_theorem}, and is valid for any model of computation.
\item[(II)] {\bf(Randomised algorithms do not help in solving the issue).} The answer to Question I is still `no' for any randomised algorithm. That is, as Theorem \ref{impossibility_theorem} reveals, replacing a deterministic algorithm with a randomised algorithm will not yield the desired error with probability better than coin-flipping.

\item[(III)] {\bf(Algorithms may compute neural networks to $K-1$ digits of accuracy, but not $K$).} As Theorem \ref{impossibility_theorem} reveals, the answer to Question I above depends on the desired accuracy. For any integer $K>1$ there exist classes of problems for which there is a mapping taking the training data to the NN, however, there will only exist an algorithm that can compute an approximation to the NN to $K-1$ correct digits, and no algorithm -- even randomised -- can compute an approximation to $K$ correct digits.

\item[(IV)] {\bf(Algorithms may exist, but any algorithm will require arbitrary large training data).}
Theorem \ref{impossibility_theorem} also shows that there are classes of problems where one can compute an approximation to the desired NN with $K-1$ correct digits, however, for any $M \in \mathbb{N}$ and any algorithm, there will be a problem so that the algorithm requires more than $M$ training data to provide an approximation with at least $K-1$ correct digits. However, for the same class of problems, there exists an algorithm providing $K-2$ correct digits using only one training datum. Moreover, condition numbers (see \S \ref{computational_setup}) of the problems in Theorem \ref{impossibility_theorem} are all bounded by 1.

\item[(V)] {\bf(Algorithms computing stable and accurate neural networks exist only in specific cases).}
(I)--(IV) demonstrate that it is only in specific cases that there exists an algorithm that can compute the desired NN, despite the fact that one can prove that the desired NN exists. This implies that there is a theory classifying which NNs can be computed by algorithms and which conditions are sufficient and necessary for the existence of algorithms. We initiate this classification theory below in Theorems \ref{key_CS_prop} and \ref{NN_FW_recov_thm}.

\item[(VI)] {\bf(Unrolling optimisation algorithms as neural networks will, in general, not converge).}
The concept of unrolling optimisation algorithms as a sequence of NNs where the $n$-th iteration can be written as the $n$-th NN in the sequence, often with $n$ layers, has been a highly popular way of creating NNs (see \S \ref{sec:prev_work}). Due to the specific choices of mappings in Theorems \ref{thrm:exists} and \ref{impossibility_theorem} (see \S \ref{niahbignion}), our impossibility results imply that such sequences of NNs will in general never converge. Indeed, potentially surprisingly, the objective function values of iterates will converge, yet the unrolled NNs output will in general never converge to the solution set of the optimisation problem.

\item[(VII)] {\bf(Under specific conditions, unrolling yields stable and accurate neural networks).} Theorems \ref{key_CS_prop} and \ref{NN_FW_recov_thm} show that
under specific conditions that typically are present in, for example, Magnetic Resonance Imaging (MRI), the technique of unrolling optimisation algorithms will yield stable and accurate NNs for certain inverse problems, hence providing an algorithm for constructing these NNs. Such NNs outperform state-of-the-art unstable (displayed in Figure \ref{fig:automap_pert}) trained NNs based on DL, such as AUTOMAP, and they can withstand adversarial attacks, see Figure \ref{fig:automap_pert3}.
When these specific conditions are present, the unrolling NNs can even be used to stabilise any unstable NN, see Figure \ref{fig:stabilisation} for an example.

\item[(VIII)] {\bf(Specific conditions yield unrolled neural networks with exponential convergence in the number of layers and fast transforms to implement the linear maps).}
Given specific conditions, the unrolling procedure can be manipulated through a careful restart scheme to yield exponential convergence: one can obtain an error of $\epsilon$ by using a NN with order $|\log(\epsilon)|$ layers. Moreover, the linear maps in the layers of the NN can be implemented using fast transforms in many applications (e.g. those in \S \ref{examples_in_CS}). Hence, one obtains fast algorithms to produce stable and accurate NNs that can be executed very efficiently. We demonstrate this in \S \ref{CS_comp_spec_examples}, where only on the order of 10 layers are needed to obtain accurate recovery of an undersampled noisy image.

\item[(IX)] {\bf(Stability/accuracy trade-off and avoiding overperformance).}  There is a trade-off between stability and accuracy in DL, with limits on how well a stable NN can perform in inverse problems (e.g. if a NN is trained to accurately recover vectors whose difference lies close to the kernel of the forward map, the recovery is necessarily unstable - see \S \ref{ononsfbsrgh}). Figure \ref{fig:false_neg} demonstrates this with a U-net trained on images consisting of ellipses and which is quite stable. However, when a detail not in the training set is added, it washes it out almost entirely. This is a problem in real-world clinical practice - Facebook and NYU's 2019 FastMRI challenge reported that networks that performed well in terms of standard image quality metrics were prone to false negatives, failing to reconstruct small, but physically-relevant image abnormalities \cite{knoll2020advancing}. The 2020 version of the challenge subsequently focused on pathologies and also included a generalisation track, noting the problem of AI-generated hallucinations, ``Our challenge confirmed areas in need of research, particularly those along the lines of evaluation metrics, error characterization, and AI-generated hallucinations'' \cite{muckley2020state}. In contrast, as demonstrated in \S \ref{nohonionavfo}, our NNs offer a blend of both stability and accuracy. However, they are by no means the end of the story. Tracing out the optimal stability vs. accuracy trade-off curve is crucial for applications and will no doubt require a myriad of different techniques to be developed to tackle different problems.

\item[(X)] {\bf{(Numerical stability)}} It is important to understand the difference between the stability (or conditioning) of a map and the stability of its numerical implementation (see, for example, the recent work in \cite{higham_IMA2020} which considers common functions used in classification methods).  As well as the former, our results cover the latter by performing an error analysis of the forward pass of the NNs, assuming each layer is computed with an error (see Remark \ref{nionin}). We show that only low precision is needed and that worst-case errors can only ever accumulate slowly as the number of layers increase.

\end{itemize}

\vspace{1mm}

\begin{center}
We call our neural networks \textbf{F}ast \textbf{I}terative \textbf{RE}started \textbf{NET}works, or \textbf{FIRENETs}.
\end{center}

\subsection{Code and data}
The code and data, used to produce all figures in this manuscript are available from
\begin{displayquote}
\centering
\url{www.github.com/Comp-Foundations-and-Barriers-of-AI/firenet}
\end{displayquote}
Further details can be found in \S \ref{s:code_and_data}.

\subsection{Notation}
Here we briefly collect some basic notation - further notation will be introduced throughout where appropriate. We use $\mathcal{N}_{m,N}$ to denote the class of neural networks (NNs) from $\mathbb{C}^m$ to $\mathbb{C}^N$ (see \S \ref{dfojon} for the precise definition of a NN). Given a metric space $(\mathcal{M},d)$, $x\in\mathcal{M}$ and $X\subset \mathcal{M}$, we let
$
d(x,X)=\mathrm{dist}(x,X)=\inf_{y\in X}d(x,y).
$
For a matrix $A\in\mathbb{C}^{m\times N}$, the norm $\|A\|$ refers to the operator norm of $A$ when $\mathbb{C}^m$ and $\mathbb{C}^N$ are equipped with the standard $l^2$-norm. For $x\in\mathbb{C}^N$ and $p\in[1,\infty]$, $\|x\|_{l^p}$ refers to the $l^p$-norm of $x$. For a set of indices $S$ and vector $x$, $x_S$ is the vector defined by $(x_S)_j=x_j$ if $j\in S$ and $(x_S)_j=0$ if $j\notin S$. The field of complex rationals $\mathbb{Q}+i\mathbb{Q}$ is denoted by $\mathbb{Q}[i]$. We use \hspace{-3mm}$\qed$ to denote the end of a proof and $\boxtimes$ to denote the end of a remark.

\section{Main Results I: Fundamental barriers and existence of algorithms }

The canonical problem studied in this paper is to solve an underdetermined system of equations. In particular, the inverse problem can be described as follows:
\begin{equation}
\label{eq:inv_prob0}
\mbox{Given noisy measurements $y = A x + e \in \mathbb{C}^m$ of $x \in \mathbb{C}^N$, recover $x$.}
\end{equation}
Here $A \in \mathbb{C}^{m \times N}$ represents a model of typically undersampled sampling modality ($m < N$), for example, $A$ could be a subsampled discrete Fourier transform as in Magnetic Resonance Imaging (MRI). Moreover, $e \in \mathbb{C}^m$ denotes the noise vector. The problem (\ref{eq:inv_prob0}) is the basis for much of inverse problems and image analysis. The possibility of $y\neq Ax$ models noise or perturbations.

\subsection{Existence of NNs is not enough, algorithms may not compute them sufficiently accurately}\label{niahbignion}
In order to demonstrate that the results we introduce also are present in applications, we consider basic mappings used in modern mathematics of information, inverse problems and optimisation. Given a matrix $A \in \mathbb{C}^{m \times N}$ and vector $y \in \mathbb{C}^m$, we consider the following three minimisation problems:
\begin{align}
(P_1)\quad&\argmin_{x \in \mathbb{C}^{N}} F_{1}^A(x)\coloneqq \|x\|_{l^1_w},\text{ such that }\|Ax-y\|_{l^2}\leq \epsilon,\label{def_P1}\\
(P_2)\quad&\argmin_{x \in \mathbb{C}^{N}} F_2^A(x,y,\lambda)\coloneqq \lambda\|x\|_{l^1_w}+\|Ax-y\|_{l^2}^2,\label{def_P2}\\
(P_3)\quad&\argmin_{x \in \mathbb{C}^{N}} F_3^A(x,y,\lambda)\coloneqq \lambda\|x\|_{l^1_w}+\|Ax-y\|_{l^2},\label{def_P3}
\end{align}
known respectively as (quadratically constrained) basis pursuit \cite{candes2006stable,becker2011nesta,adcock2019uniform}, unconstrained LASSO \cite{tropp2006just,candes2011probabilistic,hastie2015statistical} and unconstrained square-root LASSO \cite{belloni2011square,bunea2013group,van2016estimation,adcock2019correcting}. In applications, one is often interested in the solution of (\ref{eq:inv_prob0}) rather than the problems $(P_j)$, however, sparse regularisation is often used as a benchmark method for (\ref{eq:inv_prob0}) and we prove impossibility results for approximations of the solution maps of these sparse regularisation problems. The solution maps of $(P_j)$ are considerably simpler than the problem (\ref{eq:inv_prob0}) (which is, in general, ill-posed) and an impossibility result for these simpler problems is a striking computational barrier given that one can prove the existence of accurate neural networks (Theorem \ref{thrm:exists}).

The parameters $\lambda$ and $\epsilon$ are assumed to be positive rational numbers, and the weighted $l^1_w$ norm is given by
$
\|x\|_{l^1_w}\coloneqq \sum_{l=1}^Nw_l|x_{l}|,
$
where the weight vector $w$ lies in $\mathbb{Q}_{>0}^N$ (so that each weight $w_j$ is a positive rational).
Throughout the paper, we use the following notation:
\begin{equation}\label{eq:Ay}
\Xi(A,y) \text{ is the set of minimisers for } (P_j) \text{ given input } A \in \mathbb{C}^{m\times N}, y \in \mathbb{C}^m,
\end{equation}
where, for notational convenience, we have suppressed the dependence on $\epsilon$ or $\lambda$ (which are usually fixed parameters) and the index $j$. As our results hold for all of the problems $(P_j)$, we will in general not have any subscript index on $\Xi$ unless specifically needed. In certain cases, we will write $\Xi_j$ to specify minimisers of problem $(P_j)$.
Let
\[
A\in\mathbb{C}^{m\times N}, \quad \mathcal{S} = \{y_k\}_{k=1}^{R} \subset \mathbb{C}^m, \quad R < \infty.
\]
We consider the following key question:
\begin{displayquote}
\normalsize
 {\it Given a collection $\Omega$ of such pairs $(A, \mathcal{S})$, does there exists a neural network approximating the mapping $\Xi$, and can it be trained by an algorithm?}
\end{displayquote}
To make this question precise, we first note that $A$ and the elements in $\mathcal{S}$ will typically never be exact, but can be approximated to arbitrary precision. This would be the case if, for example, $A$ was a subsampled discrete cosine transform. Thus, we can access approximations $\{y_{k,n}\}_{k=1}^R\subset\mathbb{Q}[i]^m$ and $A_n\in\mathbb{Q}[i]^{m\times N}$ such that
\begin{equation}
\label{delt1_inf1}
\|y_{k,n} - y_k\| \leq 2^{-n},\quad \|A_n - A\| \leq 2^{-n}, \quad \forall n \in \mathbb{N}.
\end{equation}
(The bounds $2^{-n}$ are not important in what follows and can be replaced by any other sequence of positive rationals converging to zero.) We also assume access to $\left\{x_{k,n}\right\}_{k=1}^R\subset\mathbb{Q}[i]^N$ such that
\begin{equation}
\label{delt1_inf2}
\inf_{x^*\in\Xi(A_n,y_{k,n})}\left\|x_{k,n}-x^*\right\| \leq 2^{-n}, \quad \forall n \in \mathbb{N}.
\end{equation}
Hence, the training set associated with $(A,\mathcal{S})\in\Omega$ for training a suitable NN must be
\begin{equation}
\label{training_TT}
\iota_{A,\mathcal{S}} \coloneqq  \left\{\left(y_{k,n}, A_n, x_{k,n}\right) \, \vert \, k = 1,\hdots, R, \text{ and }n \in \mathbb{N}\right\}.
\end{equation}
Thus, given a collection of $(A, \mathcal{S})$, we denote the class of all such admissible training data by
\[
\Omega_{\mathcal{T}}\coloneqq \left\{   \iota_{A,\mathcal{S}} \text{ as in \eqref{training_TT}} \, \vert \, (A,\mathcal{S})\in\Omega,\text{ (\ref{delt1_inf1}) and (\ref{delt1_inf2}) hold}\right\}.
\]
The precise statement addressing the above question is summarised in the following theorems (the first of which follows directly from the universal approximation theorems).

\begin{theorem}[{\bf Neural networks exist for $\Xi$}]\label{thrm:exists}
Consider the problem $(P_j)$ ($j = 1,2,3$) for fixed dimensions $m < N$ and for fixed parameters $\lambda$ or $\epsilon$. Then, for any family $\Omega$ of such $(A,\mathcal{S})$ described above, there exists a mapping
$$
\mathcal{K}\colon \Omega_{\mathcal{T}}\rightarrow \mathcal{N}_{m,N}, \quad \mathcal{K}(\iota_{A,\mathcal{S}}) = \varphi_{A,\mathcal{S}}
$$
such that
$$
\varphi_{A,\mathcal{S}}(y)\in \Xi(A,y),\quad \forall y\in\mathcal{S}.
$$
In other words, $\mathcal{K}$ maps the training data $\Omega_{\mathcal{T}}$ to NNs that solve the optimisation problem $(P_j)$ for each $(A,\mathcal{S})\in\Omega$.
\end{theorem}

Despite the existence of NNs guaranteed by Theorem \ref{thrm:exists}, the problem of computing such a NN from training data is a most delicate issue, as described in the following theorem (proven in \S \ref{niahoigvbojib}).

\begin{theorem}[{\bf Despite existence, neural networks may only be computed to a certain accuracy}]
\label{impossibility_theorem}
For $j=1,2$ or $3$, consider the optimisation problem $(P_j)$ for fixed parameters $\lambda\in(0,1]$ or $\epsilon\in(0,1/2]$ and $w_l=1$, where $N \geq 2$ and $m<N$. Let $K > 2$ be a positive integer and let $L \in \mathbb{N}$. Then there exists a class $\Omega$ of elements $(A,\mathcal{S})$ as in \eqref{eq:Ay}, with the following properties. The class $\Omega$ is well-conditioned with condition numbers of the matrices $AA^*$ and the solution maps $\Xi$, as well as the feasibility primal local condition number (see \S \ref{computational_setup}), all bounded by 1 independent of all parameters. However, the following hold:
\begin{itemize}
\item[(i)] There does not exist any algorithm that, given a training set $\iota_{A,\mathcal{S}}\in\Omega_{\mathcal{T}}$, produces a neural network $\phi_{A,\mathcal{S}}$ with
\begin{equation}
\label{impossible_1}
\min_{y \in \mathcal{S}}\inf_{x^*\in \Xi(A,y)} \left\|\phi_{A,\mathcal{S}}(y)-x^*\right\|_{l^2}\leq 10^{-K}, \quad \forall \, (A,\mathcal{S}) \in \Omega.
\end{equation}
Furthermore, for any $p>1/2$, no probabilistic algorithm (BSS, Turing or any model of computation) can produce a neural network $\phi_{A,\mathcal{S}}$ such that (\ref{impossible_1}) holds with probability at least $p$.
\item[(ii)] There does exist a deterministic Turing machine that, given $\iota_{A,\mathcal{S}}\in\Omega_{\mathcal{T}}$, produces a neural network $\phi_{A,\mathcal{S}}$ such that
\begin{equation}
\label{impossible_2}
\max_{y \in \mathcal{S}} \inf_{x^*\in \Xi(A,y)} \left\|\phi_{A,\mathcal{S}}(y)-x^*\right\|_{l^2}\leq 10^{-(K-1)}, \quad \forall \, (A,\mathcal{S}) \in \Omega.
\end{equation}
However, for any probabilistic Turing machine $(\Gamma,\mathbb{P})$, $M\in\mathbb{N}$ and $p\in\left[0,\frac{N-m}{N+1-m}\right)$ that produces a neural network $\phi_{A,\mathcal{S}}$, there exists a training set $\iota_{A,\mathcal{S}} \in \Omega_{\mathcal{T}}$ such that for all $y\in\mathcal{S}$,
\begin{equation}
\label{impossible_2b}
\begin{split}
&\mathbb{P}\Big(\inf_{x^*\in \Xi(A,y)} \left\|\phi_{A,\mathcal{S}}(y)-x^*\right\|_{l^2}>10^{1-K} \\
& \qquad \qquad \qquad  \text{ or the training data size needed to construct }\phi_{A,\mathcal{S}}>M\Big)>p.
\end{split}
\end{equation}
\item[(iii)] There does exist a deterministic Turing machine that, given $\iota_{A,\mathcal{S}}\in\Omega_{\mathcal{T}}$ and using only $L$ training data from each $\iota_{A,\mathcal{S}}$, produces a neural network $\phi_{A,\mathcal{S}}(y)$ such that
\begin{equation}
\label{impossible_3}
\max_{y \in \mathcal{S}}\inf_{x^*\in \Xi(A,y)} \left\|\phi_{A,\mathcal{S}}(y)-x^*\right\|_{l^2}\leq 10^{-(K-2)},\quad \forall \, (A,\mathcal{S}) \in \Omega.
\end{equation}
\end{itemize}
\end{theorem}

\begin{remark}[Meaning of the notation $(\Gamma,\mathbb{P})$]
\label{probs}
The notation $(\Gamma,\mathbb{P})$ in (ii) is used to denote a (possibly) randomised algorithm $\Gamma$ and its law $\mathbb{P}$. This includes scenarios such as stochastic gradient descent, random selection of training data, random computation with training data etc. The precise setup is detailed in \S \ref{computational_setup}.\hfill$\boxtimes$
\end{remark}

\begin{remark}[Generalisations of Theorem \ref{impossibility_theorem}]
\label{all_m_comp}
For simplicity, we have stated Theorem \ref{impossibility_theorem} for errors measured in the $l^2$-norm and the case of unweighted $l^1$ regularisation (all the $w_l=1$) in the problems $(P_j).$ However, the proof can be adapted, and similar results hold for any norm replacing the $l^2$-norm, and any non-singular weighted $l^1$ regularisation. Moreover, result (i) in Theorem \ref{impossibility_theorem} holds regardless of the model of computation, even if we allowed real number arithmetic (see Definition \ref{definition:Algorithm}). For further details on the precise setup, including the definition of condition numbers, which are standard in the literature, see \S \ref{computational_setup}. Finally, the theorem remains true if we restrict ourselves to real-valued matrices and vectors.\hfill$\boxtimes$
\end{remark}

\begin{remark}[Theorem \ref{impossibility_theorem} and impossibility results in the SCI hierarchy]
The impossibility results in Theorem \ref{impossibility_theorem} are connected to the concept of breakdown epsilons (strong, weak and probabilistic) that were introduced (in \cite{opt_big}) as new pillars in the SCI hierarchy to establish impossibility results. These concepts are fundamental to understand the limitations of algorithms and address key issues such as runtime and complexity. The results in Theorem \ref{impossibility_theorem}, however, address a different problem, namely the amount of training data needed to construct the NN. Hence, our results in Theorem \ref{impossibility_theorem} extend the work in \cite{opt_big} to include training data as a resource to the algorithm, which is crucial to understand limitations in modern AI. To capture this, we introduce Sequential General Algorithms into the SCI hierarchy (Definition \ref{definition:Algorithm}) which require new proof techniques.\hfill$\boxtimes$
\end{remark}

\subsubsection{{\bf The neural network exists, but no algorithm can compute it -- Numerical example}}
To highlight the impossibility (as stated in Theorem \ref{impossibility_theorem}) of computing neural networks -- despite their existence by Theorem \ref{thrm:exists} -- we consider the following numerical example.  Consider the problem $(P_3)$, with $w_l=1$ and $\lambda = 1$. We note that Theorem \ref{impossibility_theorem} is stated for a specific input class $\Omega$, and in this example we will consider three different such classes $\Omega_{K}$, depending on the accuracy parameter $K$. In Theorem \ref{impossibility_theorem} we required  that $K>2$, to ensure that $K-2$ is positive. However, to show the impossibility statement (i), this is not necessary, and we consider the choices $K=1,3,6$.

To show that it is impossible to compute neural networks which can solve $(P_3)$ to arbitrary accuracy we consider (untrained) FIRENETs $\Phi_{A_n}$ and (trained) learned ISTA (LISTA) networks $\Psi_{A_n}$ based on the architecture choice from \cite{chen2018theoretical}. That is networks on the form
\begin{equation} \label{eq:LISTA}
 \Psi_{A_n}(y)=x_{T}, \quad\text{where}\quad x_{i+1} = S_{\theta_{i}}\left(x_i + W_{i}^{\top}(x_i - A_{n}^{\top}y)\right), \quad \text{ for } i =0,\ldots,T-1, \text{ and } x_0= 0.
\end{equation}
for real-valued inputs $y$ and matrices $A_n$. Here $S_{\theta}$ is the soft thresholding operator with threshold $\theta\geq 0$ and the weights $(\theta_{i},W_{i})_{i=0}^{T-1} \subset \mathbb{R}_{\geq 0} \times \mathbb{R}^{m\times N}$ are learned from the data using a squared $l^2$ norm loss function $(T=10)$. The LISTA networks are trained to high accuracy on training data on the form \eqref{training_TT} with $R=8000$ training samples and $n$ given as in Table \ref{tab:inexact}. In all cases $N=20$, $m=N-1$ and the $x_{k,n}$'s minimising $(P_3)$ with input data $(y_{k,n}, A_n)$, are all $6$-sparse. We highlight that the choice of $N$, $m$ and sparsity in this example is chosen to allow for fast training. Other choices are certainly possible.

In Table \ref{tab:inexact}, we show the distance $\text{dist}(\Psi_{A_n}(y_n), \Xi_{3}(A,y))$ and $\text{dist}(\Phi_{A_n}(y_n), \Xi_{3}(A,y))$ for both the trained LISTA networks $\Psi$ and the FIRENETs $\Phi$. Both network types are given input data $(y_n,A_n)$, approximating the true data $(y,A)$. As is clear from the table, none of the networks are able to compute an approximation to the true minimiser in $\Xi_{3}(A,y)$ to $K$ digits accuracy. However, both networks are able to compute an approximation with $K-1$ digits accuracy. These observations agree with Theorem \ref{impossibility_theorem}. Further details on how the input data and neural networks in this experiment is constructed can be found in \S \ref{desc_num_exp}.

\begin{table}
\begin{tabular}{ccccc}
\toprule
$ \text{dist}( \Psi_{A_n}(y_n), \Xi_3 (A,y) )$ & $\text{dist}( \Phi_{A_n}(y_n), \Xi_{3}(A,y) )$ & \begin{tabular}{c} $\|A_n-A\| \leq 2^{-n} $ \\ $\|y_n - y\|_{l^2} \leq 2^{-n}$ \end{tabular}  &  $10^{-K}$  & $\Omega_K$ \\
\midrule
 0.2999690 & 0.2597827 & $n = 10$ &   $10^{ -1  }$ & $K = 1$ \\
 0.3000000 & 0.2598050 & $n = 20$ &   $10^{ -1  }$ & $K = 1$ \\
 0.3000000 & 0.2598052 & $n = 30$ &   $10^{ -1  }$ & $K = 1$ \\
 0.0030000 & 0.0025980 & $n = 10$ &   $10^{ -3  }$ & $K = 3$ \\
 0.0030000 & 0.0025980 & $n = 20$ &   $10^{ -3  }$ & $K = 3$ \\
 0.0030000 & 0.0025980 & $n = 30$ &   $10^{ -3  }$ & $K = 3$ \\
 0.0000030 & 0.0000015 & $n = 10$ &   $10^{ -6  }$ & $K = 6$ \\
 0.0000030 & 0.0000015 & $n = 20$ &   $10^{ -6  }$ & $K = 6$ \\
 0.0000030 & 0.0000015 & $n = 30$ &   $10^{ -6  }$ & $K = 6$ \\
\bottomrule \\
\end{tabular}

\caption{\label{tab:inexact} (\textbf{Impossibility of computing the existing neural network to arbitrary accuracy}). We demonstrate the impossibility statement (i) from Theorem \ref{impossibility_theorem} on FIRENETs $\Phi_{A_n}$, and trained LISTA networks $\Psi_{A_n}$. The table shows the shortest $l^2$ distance between the output from the networks, and the true solution of the problem $(P_3)$, with $w_l=1$ and $\lambda =1$, for different values of $n$ and $K$. Note that none of the trained networks can compute the existing correct NN (that exists by Theorem \ref{thrm:exists} and coincides with $\Xi_3$) to $10^{-K}$ digits accuracy, while all of them are able to compute approximations which are $10^{-K+1}$ digits accurate (for the input class $\Omega_K$). This agrees exactly with Theorem \ref{impossibility_theorem}. }
\end{table}

\section{Main Results II: Algorithms compute stable and accurate NNs in specific cases}\label{ononsfbsrgh}
Theorem \ref{impossibility_theorem} shows that the optimisation problems $(P_1)$, $(P_2)$, and $(P_3)$ (defined in (\ref{def_P1}), (\ref{def_P2}), and (\ref{def_P3})) cannot, in general, be solved by an algorithm. Hence any attempt at solving the \textit{general} inverse problem \eqref{eq:inv_prob0} by such methods, without additional assumptions, is doomed to fail. This is not just the case for reconstruction using sparse regularisation. In fact, any stable and accurate reconstruction procedure must be ``kernel aware'' (see, for example, \cite{gottschling2020troublesome}) - a reconstruction method lacks kernel awareness (and thus has a large local Lipschitz constant) if it approximately recovers two vectors $x$ and $x'$ whose difference lies close to the null space of $A$.

\subsection{The subtlety and difficulty of removing instabilities and the need for additional assumptions} As discussed in \S \ref{sec_intro_lab} (see also \S \ref{sec:prev_work} on previous work), the instability problem is the current Achilles' heel of modern AI and DL. For example, adversarial training has become a standard attempt to remedy instabilities (see, for example, \cite{raj2020improving}). However, such a strategy may yield poor performance. Indeed, consider the following optimisation problem which seeks to generate a reconstruction in the form of a NN, given samples $\Theta=\{(y_s,x_s):s=1,...,R, Ax_s=y_s\}$ and $\epsilon,\lambda>0$:
\begin{equation}
\label{opt_GANS}
\min_{\phi\in\mathcal{N}_{m,N}}\frac{1}{R}\sum_{s=1}^R\max_{\|z\|_{l^2}\leq\epsilon}\left\{\|x_s-\phi(y_s)\|_{l^2}^2+\lambda \|x_s-\phi(y_s+z)\|_{l^2}^2\right\}.
\end{equation}
In other words, for each training point $(y,x)\in\Theta$ we find the worst-case perturbation $z$ in the $\epsilon$-ball around $y$ (measured by the $l^2$-norm). This is a simplified model of what one might do using Generative Adversarial Networks (GANs) to approximate adversarial perturbations \cite{goodfellow2014generative,arjovsky2017wasserstein}. For simplicity, assume that $A$ has full row rank $m$ and that we have access to exact measurements $y_s=Ax_s$. Suppose that our sample is such that $\min_{i\neq j}\|y_i-y_j\|_{l^2}>2\epsilon$. Any $\phi$ that minimises (\ref{opt_GANS}) must be such that
$$
\phi(y_s+z)=x_s,\quad \forall z\text{ with } \|z\|_{l^2}\leq \epsilon.
$$
Such networks can easily be constructed using, say, ReLU activation functions. Now suppose that $x_2$ is altered so that $x_1-x_2$ lies in the kernel of $A$. Then for any minimiser $\phi$, we must have
$$
\phi(y_1+z)=\phi(y_2+z)=\frac{x_1+x_2}{2},\quad \forall z\text{ with } \|z\|_{l^2}\leq \epsilon
$$
and hence we can never be more than $\|x_1+x_2\|_{l^2}/2$ accurate over the whole test sample. Given these types of examples and Theorem \ref{impossibility_theorem}, we arrive at the following question:

\begin{displayquote}
\normalsize
 {\it Are there sufficient conditions on the matrix $A$ that will imply the existence of an algorithm that can compute a neural network that is both accurate and stable for the image reconstruction problem \eqref{eq:inv_prob0}?}
\end{displayquote}

\subsection{A sufficient condition yielding an algorithm computing accurate and stable neural networks}
Precise theorems can be found in \S \ref{construct_sec}, and for ease of exposition, we present a simplified version here. To state this version, we need the concept of sparsity in levels from compressed sensing. This local sparsity structure was introduced in \cite{adcock2017breaking} (see also \cite{tsaig2006extensions} for different sparsities within wavelet scales), and was demonstrated empirically to play a key role in the quality of the image recovery via the ``flip test'' in \cite{roman2014asymptotic,adcock2017breaking,bastounis2017absence}.
The sparsity in levels model is crucial in demonstrating that compressed sensing and sparse regularisation is near-optimal for image recovery \cite{adcock2019log}. The issue is that natural images are not sparse in X-lets (wavelets, curvelets, shearlets etc.); they are sparse in levels.
Thus, sparsity in levels is needed since the classical theoretical model in compressed sensing, sparsity in one level, does not account for the recovery often found in practice for problems such as the Fourier-wavelet problem. ({The main problem for this example is that the Fourier-wavelet matrix is coherent.}) For example, the seminal work of Lustig, Donoho \& Pauly on compressed sensing for MRI \cite{lustig2007sparse} observed both poor recovery from uniform random sampling and the improvement offered by variable density sampling

\begin{definition}[Sparsity in levels]
\label{levels}
Let $\text{\upshape{\textbf{M}}}=(M_1,\ldots,M_r) \in {\mathbb{N}^r}$, where $1\leq M_1<\cdots<M_r=N$, and $\text{\upshape{\textbf{s}}}=(s_1,\ldots,s_r)\in {\mathbb{N}^{r}_{0}},$ where $s_k\leq M_k-M_{k-1}$ for $k=1,\ldots,r$ and $M_0=0$. A vector $x\in\mathbb{C}^N$ is $(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}})$-sparse in levels if
$$
\left|\mathrm{supp}(x)\cap\{M_{k-1}+1,...,M_k\}\right|\leq s_k,\quad k=1,...,r.
$$
The total sparsity is $s=s_1+...+s_r$. We denote the set of $(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}})$-sparse vectors by $\Sigma_{\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}}}$. We also define the following measure of distance of a vector $x$ to $\Sigma_{\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}}}$ by
$$
\sigma_{\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}}}(x)_{l^1_w}=\inf\{\|x-z\|_{l^1_w}:z\in\Sigma_{\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}}}\}.
$$
\end{definition}

Since its introduction, this model has been used to explain the effectiveness of compressed sensing in real life applications \cite{wang2014novel, bastounis2017absence, SIAM_News, jones2016continuous}. For simplicity, we will assume throughout that each $s_k > 0$ and that
\begin{equation}
\label{weight_assumption}
w_i=w_{(j)},\quad \text{if }M_{j-1}+1\leq i\leq M_{j}.
\end{equation}
In other words, the weights in the $l^1_w$ norm are constant in each level. For an image $c$ which is compressible in the wavelet basis, $\sigma_{\textbf{s},\textbf{M}}(x)_{l^1_w}$ is expected to be small if $x$ is the vector of wavelet coefficients and the levels correspond to wavelet levels, see \cite{jpeg2000, devore1998nonlinear} and \cite[Ch.\ 9]{Mallat09}. In general, the weights are a prior on anticipated support of the vector \cite{friedlander2012recovering,rauhut2016interpolation,traonmilin2018stable} and we discuss some specific choices in \S \ref{examples_in_CS}.

The key ``kernel aware'' property that guarantees algorithms that compute stable and accurate NNs (with uniform recovery guarantees) for the inverse problem (\ref{eq:inv_prob0}), is Definition \ref{def_rNSPL}, first used in the context of compressed sensing in \cite{bastounis2017absence} for the unweighted $l^1$ norm, and extended to $l^1_w$ in \cite{adcock2019uniform}. In \S \ref{examples_in_CS}, we give examples common in applications where this condition holds.

\begin{definition}[weighted rNSP in levels]\label{def_rNSPL}
Let $(\textup{\textbf{s}},\textup{\textbf{M}})$ be local sparsities and sparsity levels respectively. For weights $\{w_i\}_{i=1}^N$ $(w_i>0)$, we say that $A\in\mathbb{C}^{m\times N}$ satisfies the weighted robust null space property in levels (weighted rNSPL) of order $(\textup{\textbf{s}},\textup{\textbf{M}})$ with constants $0<\rho<1$ and $\gamma>0$ if for any $(\textup{\textbf{s}},\textup{\textbf{M}})$ support set $\Delta$,
$$
\|x_{\Delta}\|_{l^2}\leq\frac{\rho\|x_{\Delta^c}\|_{l^1_w}}{\sqrt{\xi}}+\gamma\|Ax\|_{l^2},
\quad\quad \text{for all $x\in\mathbb{C}^N$.}
$$
\end{definition}

A simplified version of Theorem \ref{key_CS_prop} can then be stated as follows, where suppressed constants are independent of $n$ and given explicitly in Theorem \ref{key_CS_prop}.

\vspace{2mm}

\noindent\textbf{Simplified version of Theorem \ref{key_CS_prop} (Computing stable and accurate NNs with exponential convergence).} \textit{There exists an algorithm such that for any input sparsity parameters $(\textup{\textbf{s}},\textup{\textbf{M}})$, weights $\{w_i\}_{i=1}^N$, $A \in \mathbb{C}^{m \times N}$ (with the input $A$ given by $\{A_l\}$) satisfying the rNSPL with constants $0<\rho<1$ and $\gamma>0$ (also input), and input parameters $n \in \mathbb{N}$ and $\{\delta,b_1,b_2\}\subset\mathbb{Q}_{>0}$, the algorithm outputs a neural network $\phi_n$ with $\mathcal{O}(n)$ layers and the following property. For any $x\in\mathbb{C}^N$ and $y\in\mathbb{C}^m$ with
$$
\sigma_{\textup{\textbf{s}},\textup{\textbf{M}}}(x)_{l^1_w}+\|Ax-y\|_{l^2}\lesssim \delta,\quad \|x\|_{l^2}\lesssim b_1,\quad \|y\|_{l^2}\lesssim b_2,
$$
we have the following exponential convergence guarantee in $n$
\[
\|\phi_{n}(y)-x\|_{l^2}\lesssim{\delta}+e^{-n}.
\]
(This is often called ``linear convergence'' or ``geometric convergence'', though we have used the term exponential convergence throughout this paper for clarity.)
}

Hence, up to the small error term $\sigma_{\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}}}(x)_{l^1_w}$, and in the limit $n\rightarrow\infty$ (with exponential convergence), we recover $x$ stably with an error proportional to the measurement error $\|Ax-y\|_{l^2}$. Further interpretations of the terms in this bound can be found in Remark \ref{nionin}. Theorem \ref{key_CS_prop} also bounds the error when we only approximately apply the nonlinear maps of the NNs: in other words, we also gain a form of stability of the forward pass of the NN. In addition to providing stability, it is precisely the rNSPL that allows us to prove exponential convergence through a careful restarting and reweighting scheme. We therefore call our NNs {F}ast {I}terative {RE}started {NET}works (FIRENETs). Note that, even when ignoring issues such as stability and inexact arithmetic, a naive unrolling of iterative methods commonly used in compressed sensing gives a slow first-order convergence rate with error bounds $\mathcal{O}(\delta+n^{-1})$ (and in certain regimes second-order $\mathcal{O}(\delta+n^{-2})$): see Remark \ref{no_restart}. In contrast to naive unrolling and previous approaches, we use novel restarting tools and analysis of the square-root LASSO problem ($P_3$) to gain good theoretical bounds (accuracy and stability).

\begin{remark}[Logarithmic grid search when parameters are unknown]
In the case that we do not know $\rho$ or $\gamma$ (the constants in the definition of rNSPL), we can perform a log-scale grid search for suitable parameters. Given a total budget of $\mathcal{O}(n\log(n))$ layers, we can still gain exponential convergence in $n$ by choosing the parameters in the grid search that lead to the vector with minimal $F_3^A$ (the objective function of $(P_3)$). In some cases, such as Theorem \ref{NN_FW_recov_thm}, it is possible to prove probabilistic results where $\rho$ and $\gamma$ are known.\hfill$\boxtimes$
\end{remark}

Finally, in \S \ref{examples_in_CS}, we apply Theorem \ref{key_CS_prop} to examples in compressive imaging (see \S \ref{CS_comp_spec_examples} for computational examples), with $A$ arising from multilevel random subsampling (Definition \ref{msrsUUU}) of Fourier and Walsh (or binary) measurements, and sparsity in levels measured in terms of wavelet coefficients. Theorem \ref{NN_FW_recov_thm} gives precise bounds on the number of samples needed, $m$, and the number of layers of a NN needed to achieve a specified accuracy:

\vspace{2mm}

\noindent\textbf{Simplified version of Theorem \ref{NN_FW_recov_thm}.} \textit{We show that stable and accurate (with exponential convergence in the number of layers) neural networks for image reconstruction can be constructed via an algorithm, with the number of samples needed the same, up to logarithmic terms, as the current best-known oracle estimators.}

\begin{remark}[Stability in the $l^2$-norm and kernel awareness properties for other norms]
We prove stability results (in the $l^2$ and $l^1_w$ norms) for perturbations of input measured in the $l^2$-norm. Since our vector space is finite-dimensional, this immediately provides stability in any norm. However, for norms different from the $l^2$-norm, the constants involved may vary with the dimension of the problem. This raises the question of whether kernel awareness properties similar to Definition \ref{def_rNSPL} can be directly used to prove stability results in other norms and avoid this dependence.\hfill$\boxtimes$
\end{remark}

\section{FIRENET: Balancing the trade-off between stability and accuracy}\label{nohonionavfo}

As demonstrated in \cite{antun2020instabilities}, current DL methods for image reconstruction can be unstable in the sense that (1) a tiny perturbation, in either the image or sampling domain, can cause severe artefacts in the reconstructed image (instability - see Figure \ref{fig:automap_pert}), and/or (2) a tiny detail in the image domain might be washed out in the reconstructed image (lack of accuracy), resulting in potential false negatives. Inevitably, there is a stability-accuracy trade-off, for this type of linear inverse problem, making it impossible for any reconstruction method to become arbitrarily stable without sacrificing accuracy or visa versa (see \S \ref{ononsfbsrgh}). In this section, we show that the NNs computed by our algorithm (FIRENETs) are stable with respect to adversarial perturbations and accurate for images which are sparse in wavelets. As most images are sparse in wavelets, these networks also show great generalisation properties to unseen images.

\textit{\textbf{Computing worst-case (adversarial) perturbations --}} First we describe the algorithm developed in \cite{antun2020instabilities} for computing perturbations meant to simulate worst-case effect in terms of reconstruction artefacts. It is this algorithm which has been used to compute the perturbed images, seen in Figures \ref{fig:automap_pert}, \ref{fig:automap_pert3}, \ref{fig_FIRENET_gen}, and \ref{fig:automap_pert4}. The algorithm does the following. Given an image $x\in\mathbb{C}^N$ and a NN $\phi\in\mathcal{N}_{m,N}$, designed for image reconstruction from samples $y$ provided by a specific sampling modality described by the matrix $A$, the algorithm searches for a perturbation of the image that makes the most severe change in the output of the network while still keeping the perturbation small. The algorithm seeks a vector $r\in\mathbb{C}^N$ such that
$$
\|\phi(y + Ar) - \phi(Ax)\|_{l^2}\text{ is large, while }\|r\|_{l^2}\text{ is small.}
$$
Specifically, consider the optimisation problem
\begin{equation}
    \label{eq:net_lasso2}
    Q^\phi_y(r) = \frac{1}{2}\|\phi(y + Ar) - x\|_{l^2}^{2} - \frac{\lambda}{2} \|r\|_{l^2}^{2},\quad r^*(y) \in \mathop{\mathrm{arg max}}_{r}  Q^\phi_y(r).
\end{equation}
The problem \eqref{eq:net_lasso2} seeks perturbations in the image domain since this provides an easy way to compare the original image and deduce whether the reconstruction of the perturbed image is acceptable/unacceptable. Of course, we could have just as easily considered perturbations in the sampling domain instead.

The non-concavity of the objective function in \eqref{eq:net_lasso2} means that finding a global maximiser of \eqref{eq:net_lasso2} is very difficult (if not impossible), even for small values of $m$ and $N$. The test aims to locate local maxima of $\eqref{eq:net_lasso2}$ by using a gradient search method. A natural method to find local maxima is gradient ascent with momentum. This uses the gradient of $Q^\phi_y$ (which can easily be written down) along with two parameters $\gamma > 0$ (the momentum) and $\eta > 0$ (the learning rate) in each step towards a local maximum. Namely, $r(0)$ is initialised randomly and then we update the perturbation at the $j$th step via $v(j+1)=\gamma v(j)+\eta \nabla_{r} Q_y^\phi(r(j))$ and $r({j+1})=r(j)+v(j+1)$. The final perturbation is taken after $M$ steps, where typically we run 10-100 steps, seeking the perturbation which causes the worst reconstructed image. Just as in the case when training NNs using stochastic gradient descent with momentum, choosing the parameters $\gamma$ and $\eta$ is an art of engineering, and the optimal choices of $\gamma, \eta$ are based on empirical testing.

\begin{figure*}
    \centering
    \setlength{\tabcolsep}{2pt}
    \resizebox{1\textwidth}{!}{
    \begin{tabular}{@{}>{\centering}m{0.33\textwidth}>{\centering}m{0.33\textwidth}>{\centering\arraybackslash}m{0.33\textwidth}@{}}
    $|x+r_3|$ & $\Psi(\tilde{y}), ~\tilde{y} = A(x+r_3)$ & $\Phi\left(\tilde{y}, \Psi(\tilde{y}) \right)$ \\
     \includegraphics[width=0.33\textwidth]{im_auto_as_input_to_lasso_rID_19_im_nbr_3_r_4_orig.png}
    & \includegraphics[width=0.33\textwidth]{runner_19_r_idx_4_noisy_rec.png}
    &\includegraphics[width=0.33\textwidth]{im_auto_as_input_to_lasso_rID_19_im_nbr_3_r_4_rec.png}
    \end{tabular}
    }
    \caption{(\textbf{Adding a few FIRENET layers at the end of AUTOMAP makes it stable}). The FIRENET $\Phi\colon \mathbb{C}^m\times\mathbb{C}^N \to \mathbb{C}^N$ takes as input measurements $y \in \mathbb{C}^m$ and an initial guess for $x$, which we call $x_0 \in \mathbb{C}^N$. We now concatenate a 25-layer ($p=5$, $n=5$) FIRENET $\Phi$ and the AUTOMAP network $\Psi \colon \mathbb{C}^{m} \to \mathbb{C}^{N}$, by using the output from AUTOMAP as initial guess $x_0$, i.e., we consider the neural network mapping $y\mapsto \Phi(y,\Psi(y))$. In this experiment we consider the perturbed image $x+r_3$ from Figure \ref{fig:automap_pert} and the perturbed measurements $\tilde{y} = A(x+r_3)$ (here $A$ is as in Figure \ref{fig:automap_pert}). As can be seen from the figure, the new network is stable with respect to AUTOMAP's worst-case perturbation $r_3$. Note that in all other experiments we use the initial guess $x_0 =0$, and consider $\Phi$ as a mapping $\Phi \colon \mathbb{C}^m \to \mathbb{C}^N$.}
\label{fig:stabilisation}
\end{figure*}

\begin{figure*}
    \setlength{\tabcolsep}{2pt}
    \resizebox{1\textwidth}{!}{
    \begin{tabular}{@{}>{\centering}m{0.33\textwidth}>{\centering}m{0.33\textwidth}>{\centering\arraybackslash}m{0.33\textwidth}@{}}
        $|x_1+v_1|$ & $|x_2 + v_2|$ & $ |x_3 + v_3|$ \\
      \includegraphics[width=0.33\textwidth]{im_im_nbr_3_experi_006_orig_p_noise_p_8_contrast_adj.png}
     &\includegraphics[width=0.33\textwidth]{im_im_nbr_4_experi_006_orig_p_noise_p_8_contrast_adj.png}
     &\includegraphics[width=0.33\textwidth]{im_im_nbr_5_experi_006_orig_p_noise_p_8_contrast_adj.png} \\
     $\Phi(A(x_1+v_1))$ &  $\Phi(A(x_2+v_2))$ &  $\Phi(A(x_3 + v_3))$ \\
      \includegraphics[width=0.33\textwidth]{im_im_nbr_3_experi_006_rec.png}
     &\includegraphics[width=0.33\textwidth]{im_im_nbr_4_experi_006_rec.png}
     &\includegraphics[width=0.33\textwidth]{im_im_nbr_5_experi_006_rec.png} \\
    \end{tabular}
    }
    \caption{
    (\textbf{FIRENET withstand worst-case perturbations and
generalises well}). To show that FIRENET generalises well and are stable, we consider three different images $x_j$, $j=1, 2, 3$. For each image $x_j$ we compute a perturbation $v_j$ meant to simulate worst-case effect for a FIRENET $\Phi$ with $n=5$ and $p=5$.
The first row shows the perturbed images $x_j+v_j$, whereas the second row shows the FIRENET reconstructions from data $A(x_j + v_j)$. Here $A \in\mathbb{C}^{m\times N}$ is a subsampled discrte Fourier transform with $m/N = 0.25$ and $N=256^2$. The perturbations $v_j$ have magnitude $\|Av_j\|_{l^2}/\|Ax_j\|_{l^2} \geq 0.05$ in the measurement domain.
}\label{fig_FIRENET_gen}
\end{figure*}

\textit{\textbf{Worst-case (adversarial) perturbations for AUTOMAP and FIRENETs --}} Figure \ref{fig:automap_pert} in the introduction shows the algorithm applied to the AUTOMAP \cite{Bo-18} network used for MRI reconstruction with 60\% subsampling. The network weights are provided by the authors of \cite{Bo-18} and had been trained on de-identified brain images from the MGH--USC HCP dataset\cite{fan2016mgh}, where the image measurements $y= Ax+e$ were contaminated with small Gaussian noise $e$. The image $x$ seen in Figure \ref{fig:automap_pert} is taken from the mentioned dataset, the algorithm is run on the AUTOMAP network to find a sequence of perturbations $\left|r_1\right| < \left|r_2\right| < \left| r_3\right|$. In order to illustrate the smallness of the perturbations, we have visualised $|x + r_j|$ in the first row of Figure \ref{fig:automap_pert}. As can be seen from the second row in the figure, the network reconstruction completely deforms the image and the reconstruction is severely unstable (similar results for other networks are demonstrated in \cite{antun2020instabilities}).

In contrast, we have applied the same algorithm, but now for the new NNs (FIRENETs) reported in this paper. Figure \ref{fig:automap_pert3} shows the algorithm applied to the constructed FIRENETs described by Theorems \ref{key_CS_prop} and \ref{NN_FW_recov_thm} (we have renamed the perturbations $v_j$ to emphasise the fact that these perturbations are sought for the new NNs and have nothing to do with the adversarial perturbations in Figure \ref{fig:automap_pert}). We now see that despite the search for adversarial perturbations, the reconstruction remains stable. The error in the reconstruction was also found to be at most of the same order of the perturbation (as expected from the stability in Theorems \ref{key_CS_prop} and \ref{NN_FW_recov_thm}). In applying the test to FIRENETs, we tested/tuned the parameters in the gradient ascent algorithm considerably (much more so than was needed for applying the test to AUTOMAP, where finding instabilities was straightforward) in order to find the worst reconstruction results, yet the reconstruction remained stable. Finally, it should be mentioned that this search algorithm is just one form of test and it is likely that there are many other tests for creating instabilities for NNs for inverse problems. This highlights the importance of results such as Theorems \ref{key_CS_prop} and \ref{NN_FW_recov_thm}, which guarantee stability regardless of the perturbation.

\textit{\textbf{Stabilising unstable NNs with FIRENETs --}} Our NNs also act as a stabiliser. For example, Figure \ref{fig:stabilisation} shows the adversarial example for AUTOMAP (taken from Figure \ref{fig:automap_pert}), but now shows what happens when we take the reconstruction from AUTOMAP as an input to our FIRENETs. Here we are using the fact that we can view our networks as approximations of unrolled (or unfolded) and restarted iterative methods, allowing us to use the output of AUTOMAP as the initial image for the reconstruction. We see that FIRENETs fix the output of AUTOMAP and stabilise the reconstruction.

\begin{figure*}
    \centering
    \setlength{\tabcolsep}{2pt}
    \resizebox{1\textwidth}{!}{

    \begin{tabular}{@{}>{\centering}m{0.15\textwidth}>{\centering}m{0.27\textwidth}>{\centering}m{0.27\textwidth}>{\centering\arraybackslash}m{0.27\textwidth}@{}}
     &Original $x$ & Original & Original + detail ($x+h_1$)\\
     &(full size)  & (cropped, blue frame) & (cropped, red frame) \\
    &\includegraphics[width=0.27\textwidth]{adv_rec_id_96_nlevel_50_iter_500_blue_131.png}
    &\includegraphics[width=0.27\textwidth]{adv_rec_id_96_nlevel_50_iter_500_crop_131.png}
    &\includegraphics[width=0.27\textwidth]{rec_N_1024_srate_15_tp_1_epoch_57_crop_220.png}
     \\
    \toprule
    & Orig.\ + worst-case noise & Rec.\ from worst-case noise & Rec. of detail \\
    \begin{turn}{90}FIRENET $(n=5, p=5)$\end{turn}
    &\includegraphics[width=0.27\textwidth]{im_experi_010_orig_p_noise_crop.png}
    &\includegraphics[width=0.27\textwidth]{im_experi_010_rec_crop.png}
    &\includegraphics[width=0.27\textwidth]{im_rec_text_firenet_N_1024_id_96_int_150_n_5_p_5_srate_15_crop.png}
    \\

    \toprule
    & Orig.\ + worst-case noise & Rec.\ from worst-case noise & Rec. of detail \\
    \tikzmark[xshift=-19pt,yshift=25ex]{x}\begin{turn}{90}{\parbox{1.4in}{\centering Network trained without noise}}\end{turn}
    & \includegraphics[width=0.27\textwidth]{im_mod_27_attack_nbr_0_crop_131.png}
    & \includegraphics[width=0.27\textwidth]{im_mod_27_attack_nbr_0_crop_132.png}
    & \includegraphics[width=0.27\textwidth]{im_mod_27_accuracy_nbr_0_crop_132.png}
     \\
    \tikzmark[xshift=-13pt,yshift=-1ex]{y}\begin{turn}{90}{\parbox{1.6in}{\centering Network trained with measurements contaminated with random noise.}}\end{turn}
    & \includegraphics[width=0.27\textwidth]{im_mod_31_attack_nbr_0_crop_131.png}
    & \includegraphics[width=0.27\textwidth]{im_mod_31_attack_nbr_0_crop_132.png}
    & \includegraphics[width=0.27\textwidth]{im_mod_31_accuracy_nbr_0_crop_132.png}
     \\
 \end{tabular}
\drawcurvedarrow[-latex,ultra thick]{x}{y}
\annote[left]{arrow-1}{\begin{turn}{90}{\parbox{1.6in}{\centering \textbf{Increased stability, but Decreased accuracy}}}\end{turn}}
}
    \caption{\label{fig:automap_pert4} (\textbf{Trained neural networks with limited performance can be stable}). We examine the accuracy/stability trade-off for linear inverse problems, by considering three reconstruction networks $\Phi_{j}\colon \mathbb{C}^{m} \to \mathbb{C}^{N}$, $j=1,2,3$. Here $\Phi_1$ is a FIRENET, whereas $\Phi_{2}$ and $\Phi_{3}$ are trained NNs, trained without and with noisy measurements, respectively (see \S \ref{nohonionavfo} for details). For each network, we compute a perturbation $w_j \in \mathbb{C}^{N}$ meant to simulate the worst-case effect, and we show a cropped version of the perturbed images $x+w_j$ in the first column (row 2-4). In the middle column (row 2-4), we show the reconstructed images $\Phi_{j}(A(x+w_j))$ from each of the networks. In the last column (row 2-4) we test the networks ability to reconstruct a tiny detail $h_1$, in the form of the text \enquote{can u see it?}. As we see from the figure, the network trained on noisy measurements is stable to worst-case perturbations, but it is not accurate. Conversely, the network trained without noise is accurate but not stable. The FIRENET is balancing this trade-off and is accurate for images which are sparse in wavelets, and stable to worst-case perturbations.
}\label{fig:false_neg}

\end{figure*}

\textit{\textbf{Generalisation --}} In order to demonstrate the generalisation properties of our NNs, Figure \ref{fig_FIRENET_gen} shows the stability algorithm applied to FIRENETs for a range of images. This shows stability across different types of images and also highlights an important fact. Namely, methods based on the conditions in \S \ref{ononsfbsrgh} allow great generalisation properties and avoid the need for time-consuming and expensive retraining of NNs for different classes of images.

\textit{\textbf{The accuracy/stability trade-off and false negatives --}} It is easy to produce a perfectly stable network: the zero network is the obvious candidate! However, this network would obviously have poor performance and produce many false negatives. The challenge is to simultaneously ensure performance and stability. Figure \ref{fig:false_neg} highlights this issue. Here we have trained two NNs to recover a set of ellipses images from noise-free and noisy Fourier measurements. The noise-free measurements are generated as $y=Ax$, where $A \in \mathbb{C}^{m\times N}$ is a subsampled discrete Fourier transform, with $m/N = 0.15$ and $N=1024^2$. The noisy measurements are generated as $y=Ax+ce$, where $A$ is as before, and the real and imaginary components of $e \in \mathbb{C}^{m}$ are drawn from a zero mean and unit variance normal distribution $\mathcal{N}(0,1)$, and $c \in \mathbb{R}$ is drawn from the uniform distribution $\mathrm{Unif([0, 100])}$. The noise $ce \in \mathbb{C}^m$, is generated on the fly during the training process.

The trained networks are using a standard benchmarking architecture for image reconstruction, and maps $y \mapsto \phi(A^* y)$, where $\phi\colon \mathbb{C}^{N} \to \mathbb{R}^N$ is a trainable U-net NN \cite{long2015fully, jin17}. Training networks with noisy measurements, using for example this architecture, have previously been used as an example of how to create NNs which are robust towards adversarial attacks \cite{genzel2020solving}. As we can see from Figure \ref{fig:automap_pert4} (bottom row) this is the case, as it does indeed create a NN which is stable with respect to worst-case perturbations.  However, a key issue is that it is also producing false negatives due to its inability to reconstruct details. Similarly, as reported in the 2019 FastMRI challenge, trained NNs that performed well in terms of standard image quality metrics were prone to false negatives: they failed to reconstruct small, but physically-relevant image abnormalities \cite{knoll2020advancing}. Pathologies, generalisation and AI-generated hallucinations were a focus of the 2020 challenge \cite{muckley2020state}. FIRENET, on the other hand, has a guaranteed performance (on images being sparse in wavelet bases) and stability, given specific conditions on the sampling procedure. The challenge is to determine the optimal balance between accuracy/stability, a problem that is well known in numerical analysis.

\section{Main Results III: Precise formulations of Theorem \ref{key_CS_prop} and Theorem \ref{NN_FW_recov_thm} }
\label{construct_sec}

\subsection{Neural networks and notational conventions}
\label{dfojon}

To state our theorems, we need to be precise about the definition of a NN. For introductions to the field of DL and NNs, we refer the reader to \cite{LeCun_Nature, higham2019deep} {and \cite{pinkus1999approximation}, respectively,} and the references therein. In order to capture standard architectures used in practice such as skip connections, we consider the following definition of a NN. Without loss of generality and for ease of exposition, we also work with complex-valued NNs. Such networks can be realised by real-valued NNs by splitting into real and imaginary parts. A NN is a mapping $\phi \colon \mathbb{C}^{m}\rightarrow\mathbb{C}^{N}$ that can be written as a composition
$$
\phi(y)=V_T(\rho_{T-1}(...\rho_1(V_1(y)))),
$$
where
\begin{itemize}
	\item Each $V_j$ is an affine map $\mathbb{C}^{N_{j-1}}\rightarrow\mathbb{C}^{N_{j}}$ given by $V_j(x)=W_jx+b_j(y)$ where $W_j\in\mathbb{C}^{N_{j}\times N_{j-1}}$ and the $b_j(y)=R_jy+c_j\in\mathbb{C}^{N_{j}}$ are affine functions of the input $y$.
	\item Each $\rho_j\colon \mathbb{C}^{N_j}\rightarrow \mathbb{C}^{N_j}$ is one of two forms:
	\begin{enumerate}
		\item[(i)] There exists an index set $I_j\subset\{1,...,N_j\}$ (possibly a strict subset) such that $\rho_j$ applies a possibly non-linear function $f_j:\mathbb{C}\rightarrow\mathbb{C}$ element-wise on the input vector's components with indices in $I_j$. In other words
		$$
		\rho_j(x)_k=\begin{dcases}
f_j(x_k),&\text{if }k\in I_j\\
x_k,&\text{otherwise}.
\end{dcases}
		$$
		\item[(ii)] There exists a possibly non-linear function $f_j:\mathbb{C}\rightarrow\mathbb{C}$ such that, after decomposing the input vector $x$ as $(x_0,X^{\top},Y^{\top})^{\top}$ (${\top}$ here denotes transpose) for scalar $x_0$ and $X\in\mathbb{C}^{m_j}$ (so that $Y\in\mathbb{C}^{N_j-1-m_j}$), we have
		\begin{equation}
		\label{weird}
		\rho_j:\begin{pmatrix}
x_0\\
X\\
Y
\end{pmatrix}\rightarrow\begin{pmatrix}
0\\
{f}_j(x_0)X\\
Y
\end{pmatrix}.
	\end{equation}
	\end{enumerate}
\end{itemize}
The affine dependence of the bias terms $b_j(y)$ on $y$ allows skip connections from the input to the current level as in standard definitions of feed-forward NNs (see, for example, \cite{shalev2014understanding} page 269). The above type of architecture has become standard \cite{jin17, DeepMRI2d, DAGAN, hammernik2018learning,devore2020neural}.

\begin{remark}[On the use of multiplication]
The use of non-linear functions of the form (ii) may be re-expressed using a combinations of element-wise logarithm and exponential functions (of the form (i)). Another option is to use the following element-wise squaring trick:
\begin{align*}
\begin{pmatrix}
x_0\\
X\\
Y
\end{pmatrix}\rightarrow\begin{pmatrix}
f_j(x_0)\\
X\\
Y
\end{pmatrix}
&\rightarrow\begin{pmatrix}
f_j(x_0)\textbf{1}\\
X\\
f_j(x_0)\textbf{1}+X\\
Y
\end{pmatrix}\\
&\rightarrow\begin{pmatrix}
f_j(x_0)^2\textbf{1}\\
X^2\\
[f_j(x_0)\textbf{1}+X]^2\\
Y
\end{pmatrix}
\rightarrow\begin{pmatrix}
0\\
\frac{1}{2}\left[[f_j(x_0)\textbf{1}+X]^2-f_j(x_0)^2\textbf{1}-X^2\right]=f_j(x_0)X\\
Y
\end{pmatrix},
\end{align*}
where $\textbf{1}$ denotes a vector of ones of the same size as $X$ (so that $f_j(x_0)\rightarrow f_j(x_0)\textbf{1}$ is a linear map). However, this is not done in practice since (\ref{weird}) is directly trainable via backpropagation.\hfill$\boxtimes$
\end{remark}

Note that we do not allow the matrices $W_j$ to depend on $y$. In other words, the NN is not adaptive dependent on its input. We will denote the collection of all NNs of the above form by $\mathcal{N}_{\text{\textbf{D}},T,q}$, where the vector $\text{\textbf{D}}=(N_0=m,N_1,...,N_T=N)$ denotes the dimensions in each layer, $T$ denotes the number of layers and $q$ denotes the number of different non-linear functions applied (including the count of different $I_j$ and $m_j$). In general, we will require that the layer sizes $N_j$ do not grow with $j$ so that the size of each layer is of the same order as the sampling matrix $A$.

The problem considered in this paper is stable reconstruction from noisy undersampled measurements, as in (\ref{eq:inv_prob0}). More precisely, we consider NNs that can be constructed via algorithms. To make this precise, we assume that we have access to a sequence of matrices $A_{l}\in\mathbb{Q}+i\mathbb{Q}=:\mathbb{Q}[i]$ such that $\|A-A_l\|\leq q_l$ for some known null sequence $\{q_l\}$. This is consistent with the training set given by (\ref{training_TT}). To construct NNs via an algorithm, care must be taken with the non-linear activation functions. We assume that for each $\theta\in\mathbb{Q}_{>0}$ we have access to a routine ``$\mathrm{sqrt}_\theta$'' such that $|\mathrm{sqrt}_\theta(x)-\sqrt{x}|\leq\theta$ for all $x\in\mathbb{R}_{\geq 0}$. In what follows, the non-linear maps $f_j$ used in the NNs are either arithmetic or constructed using arithmetic operations and $\mathrm{sqrt}_\theta$. We will always ensure that $\mathrm{sqrt}_\theta$ acts only on non-negative real numbers and on rational inputs if the input to the NN is rational. We refer to the pair $(\phi,\theta)$ as a NN.

\begin{remark}[Approximating $\sqrt{x}$ with neural networks]
On any bounded set (for bounded input our constructed NNs only require the routine $\mathrm{sqrt}_\theta$ on a bounded set), we can construct an approximation to $\sqrt{x}$ using standard non-linear activation functions such as ReLU \cite{leshno1993multilayer} (more efficient approximations may be achieved by using other activation functions such as rational maps \cite{telgarsky2017neural,boulle2020rational}). Note that this can be done in the Turing sense \cite{turing1937computable} if we restrict to rational input $w\in \mathbb{Q}_{\geq 0}$. When considering real number arithmetic, this can be done in the Blum--Shub--Smale sense \cite{BCSS}. The choice of the square root function is somewhat arbitrary, but simplifies our proof of Theorem \ref{key_CS_prop}. Similar results hold for other activation functions.\hfill$\boxtimes$
\end{remark}

\begin{remark}[An interpretation of $\theta$]\label{nionin} As well as being necessary from a foundations point of view, an important interpretation of $\theta$ is numerical stability, or accumulation of errors, of the forward pass of the NN. In particular, a larger value of $\theta$ shows greater stability for applying the nonlinear maps in finite precision. The results we prove are typically of the form
\begin{equation}
\label{sgahon}
	\|\phi_n(y)-x\|_{l^2}\leq \varepsilon+c_1(A,x)\|Ax-y\|_{l^2}+c_2(A,x)\upsilon^n,\quad \forall x\in S\subset\mathbb{C}^N,y\in\mathbb{C}^m,
\end{equation}
where $(\phi_{n},\theta_n)$ is a (sequence of) NN(s) with $\mathcal{O}(n)$ layers that is computed by an algorithm, $\upsilon\in(0,1)$ describes the exponential rate of convergence in the number of layers, $\varepsilon>0$ (in our results $\varepsilon$ will be related to the distance to vectors that are sparse in levels: $\sigma_{\mathbf{s},\mathbf{M}}(x)_{l^1_w}$ in Definition \ref{levels}), and $\theta_n^{-1}=\theta^{-1}$ is bounded independent of $n$. Up to the error tolerance $\varepsilon$, the constant $c_1(A,x)$ can be thought of as an asymptotic \textit{local Lipschitz constant} for the NNs as $n\rightarrow\infty$, and thus measures \textit{stability of inexact input $y$}. In practice one would use floating point arithmetic to approximate square roots. Hence, the boundedness of $\theta_n^{-1}$ is a \textit{numerical notion of stability} - the accuracy needed for approximating square roots (and the non-linear maps) does not become too great and errors do not accumulate as $n$ increases. Moreover, in practice, the value of $\theta^{-1}$ needed is well below what is achieved using standard floating-point formats.\hfill$\boxtimes
\end{remark}

\subsection{The construction of stable and accurate neural networks}\label{nohoagh}

The main result of this section, Theorem \ref{key_CS_prop}, uses the concept of sparsity in levels and weighted robust null space property in levels defined in \S \ref{ononsfbsrgh}. We also define the following quantities:
$$
\xi=\xi(\textbf{s},\textbf{M},w)\coloneqq \sum_{k=1}^rw_{(k)}^2s_k,\quad
\zeta=\zeta(\textbf{s},\textbf{M},w)\coloneqq \min_{k=1,...,r}w_{(k)}^2s_k,\quad
\kappa=\kappa(\textbf{s},\textbf{M},w)\coloneqq \frac{\xi(\textbf{s},\textbf{M},w)}{\zeta(\textbf{s},\textbf{M},w)}.
$$
Unless there is ambiguity, we will drop the $(\textbf{s},\textbf{M},w)$ from the notation of these parameters. Recall the setup throughout this paper of a matrix $A\in\mathbb{C}^{m\times N}$ ($m<N$), where we have access to an approximation sequence $A_{l}$ such that $\|A-A_l\|\leq q_l$ with known $q_l\rightarrow 0$ as $l\rightarrow\infty$. In this regard, the following simple perturbation lemma is useful (whose proof is given in \S \ref{pf_of_constructive_res}).

\begin{lemma}[{\textbf{The weighted rNSP in levels is preserved under perturbations or approximations}}]
\label{rNSPL_perturb}
Assume that (\ref{weight_assumption}) holds and that $A$ satisfies the weighted rNSPL of order $(\textup{\textbf{s}},\textup{\textbf{M}})$ with constants $0<\rho<1$ and $\gamma>0$. Let $\hat A$ be an approximation of $A$ such that
$$
\|\hat A-A\|<\frac{1-\rho}{\gamma\left(1+\frac{\sqrt{\xi}}{\min_{k=1,...,r}w_{(k)}}\right)}
$$
Then $\hat A$ satisfies the weighted rNSPL of order $(\textup{\textbf{s}},\textup{\textbf{M}})$ with new constants
$$
\hat\rho=\frac{\rho+\frac{\gamma\sqrt{\xi}\|\hat A-A\|}{\min_{k=1,...,r}w_{(k)}}}{1-\gamma\|\hat A-A\|},\quad \hat\gamma=\frac{\gamma}{1-\gamma\|\hat A-A\|}.
$$
\end{lemma}

Lemma \ref{rNSPL_perturb} says that if $A$ satisfied the weighted rNSPL of order $(\textup{\textbf{s}},\textup{\textbf{M}})$, then so does $A_l$ for large enough $l$. Moreover, given the null sequence $\{q_l\}$, we can compute how large $l$ must be and the new constants. For ease of exposition, we drop the notational hats from these constants. We are now ready to state our main result, which is proven in \S \ref{pf_of_constructive_res}.

\begin{theorem}[\textbf{Stable and accurate neural networks with uniform recovery guarantees can be constructed}]
\label{key_CS_prop}
There exists an algorithm such that for any input sparsity parameters $(\textup{\textbf{s}},\textup{\textbf{M}})$, weights $\{w_i\}_{i=1}^N$, $A \in \mathbb{C}^{m \times N}$ (with the input $A$ given by $\{A_l\}$) satisfying the rNSPL with constants $0<\rho<1$ and $\gamma>0$ (also input), and input parameters $n \in \mathbb{N}$, $\{\delta,b_1,b_2\}\subset\mathbb{Q}_{>0}$ and $\upsilon\in(0,1)\cap\mathbb{Q}_{>0}$, the algorithm outputs a neural network $\phi_n$ such that the following holds. For
\begin{align*}
C_1&=\left(\frac{1+\rho}{2}+(3+\rho)\frac{\kappa^{1/4}}{4}\right)\left(\frac{3+\rho}{1-\rho}\right)\sim\frac{\kappa^{1/4}}{1-\rho},\quad &&C_2=2\left(\frac{3+\rho}{1-\rho}+\frac{7+\rho}{1-\rho}\frac{\kappa^{1/4}}{2}\right)\gamma\sim\frac{\kappa^{1/4}\gamma}{1-\rho},
\end{align*}
\begin{enumerate}
	\item (Size) $\phi_{n}\in\mathcal{N}_{\textup{\textbf{D}}_{(n,p)},3np+1,3}$ with
$$
\textup{\textbf{D}}_{(n,p)}=(m,\underbrace{2N+m,2(N+m),2N+m+1}_{\text{$np$ times}},N),
$$
where $p\in\mathbb{N}$ with the bound $p\leq \left\lceil\frac{3C_2\|A\|}{\upsilon}\right\rceil$. Moreover, $\theta^{-1}\sim {p^2(1+\|w\|_{l^2})}\max\left\{1,\frac{\|w\|_{l^2}}{\|A\|\gamma \sqrt{\xi}}\right\}.$
\item (Exponentially Convergent, Uniform and Stable Recovery Guarantees)
For any pair $(x,y)\in\mathbb{C}^N\times\mathbb{C}^m$ such that
$$
\frac{2C_1}{C_2\sqrt{\xi}}\cdot\sigma_{\textup{\textbf{s}},\textup{\textbf{M}}}(x)_{l^1_w}+2\|Ax-y\|_{l^2}\leq \delta,\quad \|x\|_{l^2}\leq b_1,\quad \|y\|_{l^2}\leq b_2,
$$
we have the following exponentially convergent, uniform and stable recovery guarantees:
\begin{align}
\|\phi_{n}(y)-x\|_{l^2}&\leq\frac{2C_1}{\sqrt{\xi}}\cdot\sigma_{\textup{\textbf{s}},\textup{\textbf{M}}}(x)_{l^1_w}+2C_2\cdot\|Ax-y\|_{l^2}+\left(\frac{1+\upsilon}{1-\upsilon}\right)C_2\cdot {\delta}+{b_2}C_2\cdot \upsilon^n,\label{best1}\\
\|\phi_{n}(y)-x\|_{l^1_w}&\leq \left(\frac{3+\rho}{1-\rho}\right)\frac{\sqrt{\xi}}{C_1}\left(\frac{2C_1}{\sqrt{\xi}}\cdot\sigma_{\textup{\textbf{s}},\textup{\textbf{M}}}(x)_{l^1_w}+2C_2\cdot\|Ax-y\|_{l^2}+\left(\frac{1+\upsilon}{1-\upsilon}\right)C_2\cdot {\delta}+{b_2}C_2\cdot \upsilon^n\right).\label{best2}
\end{align}
\end{enumerate}
\end{theorem}

\begin{remark}[The optimal choice of $\upsilon$]
For a total budget of $T= 3pn+1$ layers,
$$
\upsilon^n=\exp\left(\frac{(T-1)}{3}\left\lceil\frac{3C_2\|A\|}{\upsilon}\right\rceil^{-1}\log(\upsilon)\right)
$$
If we ignore the ceiling function, then the optimal choice of $\upsilon$ is $\upsilon=e^{-1}$ (strictly speaking Theorem \ref{key_CS_prop} is only stated for rational $\upsilon$, but we can easily approximate $e^{-1}$). This yields the error term
$$
\upsilon^n= \exp\left(-\frac{(T-1)}{3}\left\lceil{3C_2e\|A\|}\right\rceil^{-1}\right)
$$
and exponential convergence in the number of layers $T$. This is not optimal. For example, a study of the proof of Theorem \ref{key_CS_prop} shows that we can replace $3C_2$ in the exponential by
$$
2\left(\frac{1+\rho}{1-\rho}+\frac{3+\rho}{1-\rho}\frac{\kappa^{1/4}}{2}\right)\gamma+\epsilon
$$
for arbitrary $\epsilon>0$. Suppose that we want $b_2C_2\cdot r^n\sim\delta$, then the number of layers required is proportional to $C_2\|A\|\log(b_2\delta^{-1})$, and only grows logarithmically with the precision $\delta^{-1}$. This is made precise in Theorem \ref{NN_FW_recov_thm}, where we apply Theorem \ref{key_CS_prop} to examples in compressive imaging.\hfill$\boxtimes$
\end{remark}

The proof of Theorem \ref{key_CS_prop} uses the optimisation problem $(P_3)$ (defined in (\ref{def_P1})), in the construction of $\phi_n$. It is also possible to prove similar results using $(P_1)$ and $(P_2)$, but we do not provide the details. The NNs constructed are approximations of unrolled primal-dual iterations for $(P_3)$, with a careful restart scheme to ensure exponential convergence in the number of layers. Pseudocode is provided in \ref{CS_comp_spec_examples}, as well as computational experiments. {The bounds in (\ref{best1}) and (\ref{best2}) are not quite optimal. If we were able to work in exact arithmetic (taking $\theta\rightarrow 0$ and $A_l\rightarrow A$), we obtain slightly smaller constants, though these do not affect the asymptotic rates. The pseudocode is written with $\theta\rightarrow 0$ and $A_l\rightarrow A$, and slightly different parameters accordingly.} The approach of unrolling iterative methods as NNs has a rich history in DL, as discussed in \S \ref{sec:prev_work}.

\begin{remark}[What happens without restart or with unknown $\delta$?]\label{no_restart}
Without the restart scheme, the convergence in the number of layers scales as $\mathcal{O}(n^{-1})$. However, one can get rid of the assumption ${2C_1}/({C_2\sqrt{\xi}})\sigma_{\textup{\textbf{s}},\textup{\textbf{M}}}(x)_{l^1_w}+2\|Ax-y\|_{l^2}\leq \delta$. (The assumption is to ensure that the reweighting of the restarts do not become too small - in practice, we found that this was not an issue and the assumption was not needed, with (up to small constants) $\delta$ replaced by ${2C_1}/({C_2\sqrt{\xi}})\sigma_{\textup{\textbf{s}},\textup{\textbf{M}}}(x)_{l^1_w}+2\|Ax-y\|_{l^2}$ in (\ref{best1}). See also the discussion in \S \ref{CS_comp_spec_examples}.) More precisely, the proof of Theorem \ref{key_CS_prop} can be adapted to show the following. For an additional input $\beta\in\mathbb{Q}_{>0}$ (and without inputs $b_2$, $\delta$ and $\upsilon$), there exists an algorithm that computes $\widehat\phi_n\in\mathcal{N}_{\textup{\textbf{D}}_{n},3n+1,3}$ with
$$
\textup{\textbf{D}}_{n}=(m+N,\underbrace{2N+m,2(N+m),2N+m+1}_{\text{$n$ times}},N),
$$
such that for any $x,x_0\in\mathbb{C}^N$ with $\|x\|_{l^2}\leq b_1$ and all $y\in\mathbb{C}^m$, the following reconstruction guarantees hold:
\begin{align}
\|\widehat\phi_{n}(y,x_0)-x\|_{l^2}&\leq\frac{2C_1}{\sqrt{\xi}}\sigma_{\textup{\textbf{s}},\textup{\textbf{M}}}(x)_{l^1_w}+2C_2\left[\|Ax-y\|_{l^2}+\frac{\|A\|}{n}\left(\frac{\|x-x_0\|_{l^2}^2}{\beta}+{\beta}\right)\right],\label{old_BOUND1}\\
\|\widehat\phi_{n}(y,x_0)-x\|_{l^1_w}&\leq \left(\frac{3+\rho}{1-\rho}\right)\frac{\sqrt{\xi}}{C_1}\left(\frac{2C_1}{\sqrt{\xi}}\sigma_{\textup{\textbf{s}},\textup{\textbf{M}}}(x)_{l^1_w}+2C_2\left[\|Ax-y\|_{l^2}+\frac{\|A\|}{n}\left(\frac{\|x-x_0\|_{l^2}^2}{\beta}+{\beta}\right)\right]\right).\label{old_BOUND2}
\end{align}
Here, $x_0$ should be interpreted as an initial guess (an arbitrary input to the NNs) and $\beta$ should be interpreted as a scaling parameter, with optimal scaling $\beta\sim\|x-x_0\|_{l^2}$. A good choice for $\beta$ is $\|x\|_{l^2}$, or, in the case that this is unknown, $\|y\|_{l^2}/\|A\|$. For completeness, we have provided a proof sketch of (\ref{old_BOUND1}) and (\ref{old_BOUND2}) at the end of \S \ref{nnoinav9bh}.\hfill$\boxtimes$
\end{remark}

Algorithm unrolling is particularly well-suited to scenarios where it is difficult to collect large training samples, and can provide a means of interpretability, overcoming the black-box nature of many NNs. However, training a finite fixed number of layers typically incurs the same stability and generalisation issues mentioned above. Moreover, learning the weights and biases usually prevents the convergence analysis of standard (unlearned) iterative methods carrying over. In particular, there is no guarantee of objective function minimisation (let alone convergence of the iterated arguments) or any form of convergence as the number of layers increases. A subtle, yet fundamental, point regarding iterative methods, whether they are unrolled as a NN and possible supplemented with learned parameters or not, is the following. Theorem \ref{impossibility_theorem} states that, in general, the optimisation problems $(P_1)$, $(P_2)$, and $(P_3)$ are non-computable. This is despite the fact that there are many results in the literature describing rates of convergence for iterative methods. The resolution of this apparent puzzle is that convergence results regarding iterative methods are typically given in terms of the \textit{objective function} that is being minimised (see also Theorem \ref{BP_prop1}, which we use to prove Theorem \ref{key_CS_prop}). As the proof of Theorem \ref{key_CS_prop} shows, it is crucial to have conditions such as the rNSPL to convert these objective function bounds to the desired error bounds on the distance to the \textit{minimisers} or vector $x$. Moreover, this property has the key effect of allowing exponential convergence through restarting and reweighting.

\subsection{Examples in compressive imaging}\label{examples_in_CS}

As an example application of Theorem \ref{key_CS_prop}, we consider the case of Fourier and Walsh sampling, using the Haar wavelets as the sparsifying transform. {Our results can be generalised to the infinite-dimensional setting with the use of higher-order Daubechies wavelets (though the results are more complicated to write down), and we refer the reader to \cite{adcock2012generalized,adcock2016generalized} for compressed sensing in infinite dimensions.} We first define the concept of multilevel random subsampling \cite{adcock2017breaking}.

\begin{definition}[Multilevel random subsampling]
\label{msrsUUU}
Let $\mathbf{N} = (N_1, \ldots, N_l) \in \mathbb{N}^{l}$, where $1 \leq N_1 < \cdots < N_l = N$ and $\mathbf{m} = (m_1,\ldots, m_l) \in \mathbb{N}^l$ with $m_k \leq N_{k}-N_{k-1}$ for $k=1,\ldots, l$, and $N_0=0$. For each $k=1,\ldots,l$, let $\mathcal{I}_{k} = \{N_{k-1}+1,\ldots, N_{k} \}$ if $m_{k} = N_{k}-N_{k-1}$ and if not, let $t_{k,1}, \ldots, t_{k,m_k}$ be chosen uniformly and independently from the set $\{N_{k-1}+1, \ldots, N_k\}$ (with possible repeats), and set $\mathcal{I}_{k}= \{ t_{k,1}, \ldots, t_{k,m_k} \}$. If $\mathcal{I} = \mathcal{I}_{\mathbf{N},\mathbf{m}} = \mathcal{I}_{1} \cup \cdots \cup \mathcal{I}_{l}$ we refer to $\mathcal{I}$ as an $(\mathbf{N}, \mathbf{m})$-multilevel subsampling scheme.
\end{definition}

\begin{definition}[Multilevel subsampled unitary matrix]
\label{msUnitary}
A matrix $A\in\mathbb{C}^{m\times N}$ is an $(\mathbf{N}, \mathbf{m})$-multilevel subsampled unitary matrix if $A=P_{\mathcal{I}}DU$ for a unitary matrix $U\in\mathbb{C}^{N\times N}$ and $(\mathbf{N}, \mathbf{m})$-multilevel subsampling scheme $\mathcal{I}$. Here $D$ is a diagonal scaling matrix with diagonal entries
$$
D_{ii}=\sqrt{\frac{N_{k}-N_{k-1}}{m_k}},\quad i=N_{k-1}+1,...,N_k,\quad k=1,...,l
$$
and $P_{\mathcal{I}}$ denotes the projection onto the linear span of the subset of the canonical basis indexed by $\mathcal{I}$ (i.e. subsampled rows).
\end{definition}

Throughout this subsection, we let $K = 2^{r}$ for $r \in\mathbb{N}$, and we consider vectors on $\mathbb{C}^{K}$ or $d$-dimensional tensors on $\mathbb{C}^{K\times \cdots \times K}$. To keep consistent notation with previous sections, we set $N=K^d$ so that the objective is to recover a vectorised $x\in\mathbb{C}^N$. The following can also be generalised to rectangles (i.e. $\mathbb{C}^{2^{r_1}\times \cdots \times 2^{r_d}}$ with possibly different $r_1,...,r_d$) or dimensions that are not powers of two, but we stick to the case of multi-dimensional dyadic cubes for simplicity.

Let $V \in \mathbb{C}^{N \times N}$ be either the matrix $F^{(d)}$ or $W^{(d)}$, corresponding to the $d-$dimensional discrete Fourier or Walsh transform (see \S \ref{ON_bases_def_sec}). In the Fourier case, we divide the different frequencies $\{-K/2+1,\ldots,K/2\}^d$ into dyadic bands. For $d=1$, we let $B_1 = \{0, 1\}$ and
$$
B_k=\left\{-2^{k-1}+1,\ldots,-2^{k-2}\right\}\cup\left\{2^{k-2}+1,\ldots,2^{k-1}\right\},\quad k=2,\ldots,r.
$$
In the Walsh case, we define the frequency bands $B_1 = \{0,1\}$ and $B_{k} = \{2^{k-1}, \ldots, 2^{k}-1\}$ for $k=2,\ldots, r$ in the one-dimensional case. In the general $d$-dimensional case for Fourier or Walsh sampling, we set
$$
B_\textbf{k}^{(d)}=B_{k_1}\times\ldots\times B_{k_d},\quad \textbf{k}=(k_1,\ldots,k_d)\in\mathbb{N}^d.
$$
For a $d$-dimensional tensor $c \in \mathbb{C}^{K\times \cdots\times K}$, we will assume we can observe subsampled measurements of $V \textnormal{vec}(c)$, {where $\textnormal{vec}(c) \in \mathbb{C}^{N}$ is a vectorised version of $c$}. To recover a sparse representation, we consider the Haar wavelet coefficients. {We} denote the discrete Haar Wavelet transform by $\Phi {\in \mathbb{C}^{N \times N}}$, and note that $\Psi^* = \Psi^{-1}$ since $\Psi$ is unitary. In other words, we consider a multilevel subsampled unitary matrix (Definition \ref{msUnitary}), with $U=V\Psi^*$. Given $\{m_{\textbf{k}=(k_1,\ldots,k_d)}\}_{k_1,\ldots,k_d=1}^r$, we use a multilevel random sampling such that $m_\textbf{k}$ measurements are chosen from $B_{\textbf{k}}^{(d)}$ according to Definition \ref{msrsUUU}. This corresponds to $l = r^d$ and the $N_i$'s can be chosen given a suitable ordering of the Fourier/Walsh basis. The sparsity in levels structure (Definition \ref{levels}) is chosen to correspond to the $r$ wavelet levels. A pictorial representation is given in Figure \ref{sample_boxes}. Finally, we define the quantities
\begin{align}
\mathcal{M}_{\mathcal{F}}(\textbf{s},\textbf{k})&\coloneqq \sum_{j=1}^{\|\text{\upshape{\textbf{k}}}\|_{l^\infty}}s_{j}\prod_{i=1}^d 2^{-\left|k_i-j\right|}+\sum_{j=\|\text{\upshape{\textbf{k}}}\|_{l^\infty}+1}^{r}{s_j2^{-2(j-\|\text{\upshape{\textbf{k}}}\|_{l^\infty})}}\prod_{i=1}^d 2^{-\left|k_i-j\right|}\label{M1F}\\
\mathcal{M}_{\mathcal{W}}(\textbf{s},\textbf{k})&\coloneqq s_{\|\text{\upshape{\textbf{k}}}\|_{l^\infty}}\prod_{i=1}^d 2^{-\left|k_i-\|\text{\upshape{\textbf{k}}}\|_{l^\infty}\right|}.\label{M1W}
\end{align}
For notational convenience, we also define
$$
\mathcal{Z}=\max\left\{1,\frac{\max_{j=1,...,r}w_{(j)}\sqrt{(M_j-M_{j-1})}}{\sqrt{\xi(\textup{\textbf{s}},\textup{\textbf{M}},w)}}\right\}.
$$

\begin{figure}
\begin{center}
    \setlength{\tabcolsep}{2pt}
    \resizebox{1\textwidth}{!}{
    \begin{tabular}{@{}>{\centering}m{0.5\textwidth}>{\centering\arraybackslash}m{0.5\textwidth}@{}}
\textbf{Fourier sampling regions} & \textbf{Walsh sampling regions} \\
\begin{tikzpicture}[scale=0.87]
{\footnotesize
    \input{samp_patt_fourier}
}
\end{tikzpicture} &
\begin{tikzpicture}[scale=0.87]
{\footnotesize
    \input{samp_patt_walsh}
}
\end{tikzpicture}
\end{tabular}
}
\end{center}
\caption{The different sampling regions used for the sampling patterns for Fourier (left, $r=3$) and Walsh (right, $r=4$). The axis labels correspond to the frequencies in each band and the annular regions are shown as the shaded greyscale regions.}\label{sample_boxes}
\end{figure}

We can now state the main theorem of this section (proven in \S \ref{CS_proofs_F}), which states how many samples are needed and the number of layers of the NN needed. The key bound on the number of layers is that it only depends logarithmically on the error $\delta$, a consequence of the exponential convergence in Theorem \ref{key_CS_prop}. We discuss the sampling conditions below.

\begin{theorem}
\label{NN_FW_recov_thm}
Consider the above setup of recovering a $d$-dimensional tensor $c \in \mathbb{C}^{K^d}$ ($N=K^d$) from subsampled Fourier or Walsh measurements $V c$, such that $A$ is a multilevel subsampled unitary matrix with respect to $U=V\Psi^*$. Let $\epsilon_{\mathbb{P}}\in(0,1)$ and
$$
\mathcal{L}= d\cdot r^2 \cdot \log(2m) \cdot \log^2\left(s\cdot\kappa(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}},w)\right)  + \log(\epsilon_{\mathbb{P}}^{-1}).
$$
Suppose that:
\begin{itemize}
	\item (a) In the Fourier case
	\begin{equation}
\label{needed_bounda}
m_\text{\upshape{\textbf{k}}}\gtrsim \kappa(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}},w)\cdot \mathcal{M}_{\mathcal{F}}(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{k}}})\cdot \mathcal{L}.
\end{equation}
\item (b) In the Walsh case
\begin{equation}
\label{needed_boundb}
m_\text{\upshape{\textbf{k}}}\gtrsim \kappa(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}},w)\cdot \mathcal{M}_{\mathcal{W}}(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{k}}})\cdot \mathcal{L}.
\end{equation}
\end{itemize}
Then with probability at least $1-\epsilon_{\mathbb{P}}$, $A$ satisfies the weighted rNSPL of order $(\textup{\textbf{s}},\textup{\textbf{M}})$ with constants $\rho=1/2$ and $\gamma=\sqrt{2}$. The conclusion of Theorem \ref{key_CS_prop} then holds for the uniform recovery of the Haar wavelet coefficients
\begin{equation}
x=\Psi c\in\mathbb{C}^N.
\end{equation}
Moreover, for any $\delta\in(0,1)$, let $\mathcal{J}({\delta,\mathbf{s},\mathbf{M},w})$ be the collection of all $y\in\mathbb{C}^m$ such that $y=P_{\mathcal{I}}DVc+e$ where
\begin{equation}
\label{bdd_for_delta}
\|c\|_{l^2}\leq 1,\quad \max\left\{\frac{\sigma_{\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}}}(\Psi c)_{l^1_w}}{\sqrt{\xi}},\|e\|_{l^2}\right\}\leq \delta.
\end{equation}
Then we can construct, via an algorithm, a neural network $\phi\in\mathcal{N}_{\textup{\textbf{D}},3n+1,3}$ such that with probability at least $1-\epsilon_{\mathbb{P}}$,
\begin{equation}
\label{FW_recov_bound_J}
\left\|\phi(y)-c\right\|_{l^2}\lesssim \kappa^{1/4}\delta,\quad \forall y=P_{\mathcal{I}}DVc+e\in\mathcal{J}({\delta,\mathbf{s},\mathbf{M},w}).
\end{equation}
The network parameters are
$$
\textup{\textbf{D}}=(m,\underbrace{2N+m,2(N+m),2N+m+1}_{\text{$n$ times}},N),
$$
where
\begin{equation}
\label{how_many_layers}
n\leq\left\lceil{\log\left({\delta^{-1}}\mathcal{Z}\right)\kappa^{1/4}\mathcal{Z}}\right\rceil.
\end{equation}
\end{theorem}

The sampling conditions (\ref{needed_bounda}) and (\ref{needed_boundb}) are optimised by minimising $\kappa(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}},w)$. Up to a constant scale, this corresponds to the choice $w_{(j)}=\sqrt{s/s_j}$ and
$$
n=\left\lceil{\log\left({\delta^{-1}}\max_{j=1,...,r}\sqrt{\max\left\{1,\frac{M_j-M_{j-1}}{rs_j}\right\}}\right)r^{1/4}\max_{j=1,...,r}\sqrt{\max\left\{1,\frac{M_j-M_{j-1}}{rs_j}\right\}}}\right\rceil.
$$
Up to log-factors, the measurement condition then becomes equivalent to that for the currently best-known oracle estimator (where one assumes apriori knowledge of the support of the vector) \cite[Prop. 3.1]{adcock2018oracle}. In the case of Fourier measurements, we can interpret the condition as follows. For $d=1$, this estimate yields the multilevel sampling estimates
$$
m_k \gtrsim \left(\sum_{j=1}^k s_j 2^{-\left|k-j\right|}+\sum_{j=k+1}^r s_j 2^{-3\left|k-j\right|}\right)\cdot r\cdot \mathcal{L}.
$$
In other words, up to logarithmic factors and exponentially small terms, $s_j$ measurements are needed in each level. Furthermore, if $s_1=\ldots=s_r=s_*$ and $d=2$ then (\ref{needed_bounda}) holds if
\begin{equation}
\label{how_many2}
m_{(k_1,k_2)} \gtrsim s_*2^{-\left|k_1-k_2\right|}\cdot r \cdot \mathcal{L}.
\end{equation}
Another interpretation is gained by considering
$$
m_k=\sum_{\left\|\textbf{k}\right\|_{l^\infty}=k}m_{\textbf{k}},\quad k=1,\ldots,r,
$$
the number of samples per annular region. We then have
\begin{equation}
\label{how_many4}
m_k \gtrsim 3^d d\left(s_k+\sum_{l=1}^{k-1}s_l2^{-(k-l)}+\sum_{l=k+1}^rs_l2^{-3(l-k)}\right)\cdot r \cdot \mathcal{L},
\end{equation}
which is the same estimate as the one dimensional case for bounded $d$. Note that the number of samples required in each annular region is (up logarithmic factors) proportional to the corresponding sparsity $s_k$ with additional exponentially decaying terms dependent on $s_l,l\neq k$. In the case of Walsh sampling, (\ref{how_many2}) remains the same whereas (\ref{how_many4}) becomes
$$
m_k \gtrsim 2^d\cdot d\cdot r\cdot \mathcal{L} \cdot s_k,
$$
and there are no terms from the sparsity levels $s_l,l\neq k$.

\section{FIRENET: Example of the exponential convergence and pseudocode}\label{CS_comp_spec_examples}

We now provide a computational experiment to demonstrate Theorem \ref{NN_FW_recov_thm} (and Theorem \ref{key_CS_prop}). Note that the matrix $A$ and its adjoint can be implemented rapidly using the fast Fourier transform (or fast Walsh--Hadamard transform). We take the image shown in Figure \ref{fig:exp_conv1}, a subsampling rate of only $15\%$, and corrupt the measurements by adding $2\%$ Gaussian noise. Figure \ref{fig:exp_conv1} shows the reconstructions using Fourier and Walsh sampling and Haar wavelets. {Similar results hold for other wavelets, such as Daubechies wavelets with a larger number of vanishing moments. In fact, the reconstruction results are better than what we have shown for the Haar wavelet system. We have chosen to show the Haar wavelet results because this is the system for which Theorem \ref{NN_FW_recov_thm} is stated.} Pseudocode for the reconstruction is shown in Algorithm \ref{exp_alg}. For the reconstruction, we take $\lambda=0.00025,\tau=\sigma=1$, $p=5$ and the weights as discussed in \S \ref{examples_in_CS}. {In the spirit of no parameter tuning, the weights were selected based on a standard phantom image, and not the image we use to test the algorithm.} These parameters are certainly not optimal, and instead were chosen simply to emphasise that we have deliberately avoided parameter tuning. Moreover, we found that the choice of $\delta$ in the algorithm was of little consequence, so have taken $\delta=10^{-9}$.

\begin{figure}
\vspace{2mm}
  \centering
  \begin{minipage}[b]{0.32\textwidth}
    \begin{overpic}[width=\textwidth,trim={25mm 6mm 25mm 6mm},clip]{hr_true-eps-converted-to.pdf}
     \put (45,101) {\textbf{Image}}
     \end{overpic}
  \end{minipage}
  \begin{minipage}[b]{0.32\textwidth}
    \begin{overpic}[width=\textwidth,trim={25mm 6mm 25mm 6mm},clip]{hr_F_recon-eps-converted-to.pdf}
    \put (25,101) {\textbf{Fourier Sampling}}
     \end{overpic}
  \end{minipage}
	\begin{minipage}[b]{0.32\textwidth}
    \begin{overpic}[width=\textwidth,trim={25mm 6mm 25mm 6mm},clip]{hr_W_recon-eps-converted-to.pdf}
    \put (25,101) {\textbf{Walsh Sampling}}
     \end{overpic}
  \end{minipage}

	\begin{minipage}[b]{0.32\textwidth}
    \begin{overpic}[width=\textwidth,trim={25mm 6mm 25mm 6mm},clip]{hr_true_zoom-eps-converted-to.pdf}
     \end{overpic}
  \end{minipage}
  \begin{minipage}[b]{0.32\textwidth}
    \begin{overpic}[width=\textwidth,trim={25mm 6mm 25mm 6mm},clip]{hr_F_recon_zoom-eps-converted-to.pdf}
     \end{overpic}
  \end{minipage}
	\begin{minipage}[b]{0.32\textwidth}
    \begin{overpic}[width=\textwidth,trim={25mm 6mm 25mm 6mm},clip]{hr_W_recon_zoom-eps-converted-to.pdf}
     \end{overpic}
  \end{minipage}
  \caption{Left: The true image. Middle: Reconstruction from noisy Fourier measurements. Right: Reconstruction from noisy Walsh measurements. Both images were reconstructed using only a $15\%$ sampling rate according to the sampling patterns in Figure \ref{sample_boxes} and $n=p=5$. The top row shows the full image and the bottom row shows a zoomed in section (corresponding to the red boxes in the top row).}
\label{fig:exp_conv1}
\end{figure}

\begin{figure}
\vspace{2mm}
  \centering
  \begin{minipage}[b]{0.49\textwidth}
    \begin{overpic}[width=\textwidth,trim={0mm 0mm 0mm 0mm},clip]{hr_F_conv-eps-converted-to.pdf}
    \put (22,73) {\textbf{Convergence, Fourier Sampling}}
     \end{overpic}
  \end{minipage}
	\begin{minipage}[b]{0.49\textwidth}
    \begin{overpic}[width=\textwidth,trim={0mm 0mm 0mm 0mm},clip]{hr_W_conv-eps-converted-to.pdf}
    \put (23,73) {\textbf{Convergence, Walsh Sampling}}
     \end{overpic}
  \end{minipage}
  \caption{The convergence of the algorithm in the number of inner iterations. The dashed line shows the relative error for the solution set of $(P_3)$. In both cases, the error between the reconstruction and the image decreases exponentially until this bound is reached. The objective function gap decreases exponentially slightly beyond this point, demonstrating that the robust null space property (in levels) controls the $l^2$-norm difference between vectors (locally around $c^*$) down to the error $\|c-c^*\|_{l^2}$ (see the bound (\ref{3bound}) in our proof).}
\label{fig:exp_conv2}
\end{figure}

\begin{algorithm}
 \SetKwInOut{Input}{Input}\SetKwInOut{Output}{return}
 \BlankLine
\SetKwProg{Fn}{Function}{}{end}
 \SetKwFunction{FIRENETcomp}{FIRENETcomp}
 \SetKwData{Left}{left}\SetKwData{This}{this}\SetKwData{Up}{up}
\SetKwFunction{Union}{Union}\SetKwFunction{FindCompress}{FindCompress}
 \Fn{\FIRENETcomp{$A, p, \tau, \sigma, \lambda,\{w_j\}_{j=1}^N, \epsilon_0, \delta, n$}}{
 \BlankLine
 Initiate with $\phi_0\equiv 0$ (other initial vectors can also be chosen).\\%Set $\epsilon_0=\|y\|_{l^2}$ and
(NB: $\epsilon_0$ should be of the same order as $\|y\|_{l^2}$ for inputs $y\in\mathbb{C}^m$.)\\
\For{$k=1,...,n$}{
$\epsilon_k=e^{-1}(\delta+\epsilon_{k-1}),\quad\quad$  \\
$\beta_k=\frac{\epsilon_k}{2\|A\|}$\\
$\phi_k(\cdot)=p\beta_k\cdot\texttt{InnerIt}\left(\frac{\cdot}{p\beta_k},\frac{\phi_{k-1}(\cdot)}{p\beta_k},A,p,\sigma, \tau,\lambda, \{w_j\}_{j=1}^N\right)$
}
\Output{FIRENET $\phi_n:\mathbb{C}^m\rightarrow \mathbb{C}^N$}
}

 \SetKwInOut{Input}{Input}\SetKwInOut{Output}{return}
 \BlankLine
\SetKwProg{Fn}{Function}{}{end}
 \SetKwFunction{InnerIt}{InnerIt}
 \SetKwData{Left}{left}\SetKwData{This}{this}\SetKwData{Up}{up}
\SetKwFunction{Union}{Union}\SetKwFunction{FindCompress}{FindCompress}
 \Fn{\InnerIt{$y, x_0, A, p, \tau, \sigma, \lambda, \{w_j\}_{j=1}^N$}}{
  \BlankLine
	Set $B=\mathrm{diag}(w_1,...,w_N)\in\mathbb{C}^{N\times N}$.\\
	Initiate with $x^0=x_0$, $y^0=0\in\mathbb{C}^m$ (the superscripts denote indices not powers).\\
\For{$k=0,...,p-1$}{
$x^{k+1}=B\varphi_{\tau\lambda}(B^{-1}(x^k-\tau A^*y^k))$ \\
$y^{k+1}=\psi^1(y^k+\sigma A(2x^{k+1}-x^k)-\sigma y)$
}
$X=\sum_{k=1}^p\frac{x^k}{p}$ \\
\Output{$X\in\mathbb{C}^N$ (ergodic average of $p$ iterates)}
}

\caption{\texttt{FIRENETcomp} constructs a FIRENET which corresponds to $n$ iterations of \texttt{InnerIt} with a rescaling scheme. We have written the output as the map $\phi_n$ to emphasise that \texttt{FIRENETcomp} defines a NN. \texttt{InnerIt} performs $p$ iterations of Chambolle and Pock's primal-dual algorithm for square-root LASSO (the order of updates is swapped compared to \cite{chambolle2011first}). The functions $\varphi_s$ and $\psi^1$ are proximal maps given by
$$
[\varphi_s(x)]_j=\max\left\{0,1-\frac{s}{|x_j|}\right\}x_j,\quad \psi^1(y)=\min\left\{1,\frac{1}{\|y\|_{l^2}}\right\}y.
$$
Both of these are approximated by NNs in our proof.}\label{exp_alg}
 \end{algorithm}

Figure \ref{fig:exp_conv2} shows the convergence in the number of inner iterations (or, equivalently, $n$ - the total number of inner iterations is $np$ and hence we have not specified $n$, which is typically chosen to be $5$). We show the error between the constructed image after $j$ iterations (denoted by $c_j$) and the true image (denoted by $c$), as well as the convergence of the objective function which we denote by $F$ in the figure caption. To compute the minimum of $F$, denoted $F^*$, we ran several thousand iterates of the non-restarted version of the algorithm so that the error in the value of $F^*$ is at least an order of magnitude smaller than the shown values of $F(c_j)-F^*$. Whilst the objective function is guaranteed to converge to the minimum value when computing $F^*$ this way, there is no guarantee that the vectors computed by the non-restarted version converge to a minimiser, as demonstrated by the non-computability results in Theorem \ref{impossibility_theorem}. However, in this case, the non-restarted version converged to a vector $c^*$ up to an error much smaller than $\|c-c^*\|_{l^2}$. Hence $\|c-c^*\|_{l^2}$ gives an indication of the minimum error we can expect from using $(P_3)$ to recover the image.

The figure shows the expected exponential convergence, as the number of inner iterations increases, of the objective function values as well as $c_j$ to $c$ until the error is of the order $\|c-c^*\|_{l^2}$. This corresponds to an initial phase of exponential convergence, where the $\upsilon^{-n}$ term (with $\upsilon=e^{-1}$) is dominant in Theorem \ref{key_CS_prop}, followed by a plateau to the minimal error $\|c-c^*\|_{l^2}$ (shown as the dotted line). This plateau occurs due to inexact measurements (the noise) and the fact that the image does not have exactly sparse wavelet coefficients. This corresponds to the robust null space property (in levels) only being able to bound the distance $\|c-c_j\|_{l^2}$ up to the same order as $\|c-c^*\|_{l^2}$. In other words, we can only accelerate convergence up to this error bound. The error plateau disappears in the limit of exactly sparse vectors and zero noise (in the limit $\delta\downarrow 0$ in Theorem \ref{key_CS_prop}), and one gains exponential convergence down to essentially machine precision. Finally, the acceleration is of great practical interest. Rather than the several hundred (or even thousands) of iterations that are typically needed for solving compressed sensing optimisation problems with first-order iterative methods, we obtain optimal accuracy in under 20 iterations. This was found for a range of different images, subsampling rates etc. The fact that so few layers are needed, coupled with the fast transforms for implementing the affine maps in the NNs, makes the NNs very computationally efficient and competitive speed-wise with state-of-the-art DL

\section{Connections with previous work}\label{sec:prev_work}
The paper touches on many different areas of mathematics and AI, thus we have divided the areas into the following subsections:  (i) Computational barriers in DL, foundations and the SCI hierarchy; (ii) Instabilities in DL; (iii) DL in inverse problems; and (iv) Compressed sensing, optimisation and unrolling of algorithms.

\begin{itemize}[leftmargin=1mm]
\item[(i)] {\bf  \emph{Computational barriers in DL, foundations and the SCI hierarchy}}:
The SCI hierarchy and its foundations framework provide the basis for the techniques for the computational barriers and foundations (what is and what is not computationally possible) results in DL proved in this paper. The SCI hierarchy has recently been used to solve longstanding questions on existence of algorithm \cite{ben2015can,Hansen_JAMS,colbrook2019compute,colbrook2019infinite,Jonathan_res,opt_big,benartzi2020computing,colbrook2020foundations}. It was introduced in \cite{Hansen_JAMS} and is an effective tool to establish the boundaries of what computers can achieve in scientific computing and also in computer assisted proofs, see for example the work by C. Fefferman \& L. Seco (Dirac-Schwinger conjecture) \cite{fefferman1996interval,fefferman1990,fefferman1994_2} and T. Hales et. al (Kepler's conjecture/Hilbert's 18th problem) \cite{Hales_Annals, hales_Pi} that implicitly prove results in the SCI hierarchy (see \cite{ben2015can} for details).  It generalises S. Smale's seminal work \cite{Smale2, Smale_Acta_Numerica}  with L. Blum, F. Cucker, M. Shub \cite{ BSS_Machine, BCSS} and his program on the foundations of scientific computing  and existence of algorithms pioneered by C. McMullen \cite{McMullen1,mcmullen1988braiding,mcmullen_int}  and P. Doyle \& C. McMullen \cite{Doyle_McMullen}. In particular, the work by F. Cucker \cite{Cucker_AH_real} can be viewed as an early version of the SCI hierarchy.

In \cite{ben2015can} J. Ben-Artzi, O. Nevanlinna, M. Seidel and two of the authors established a collection of techniques for proving sharp results on existence of algorithms that form a basis for techniques in the SCI hierarchy framework. These ideas were extended to randomised algorithms (crucial for this paper) by A. Bastounis, V. Vlacic and one of the authors in \cite{opt_big}. The SCI hierarchy has also been used in signal processing and sampling theory by H. Boche \& V. Pohl in \cite{boche2019solvability}, see also the work by H. Boche \& V. Pohl \cite{boche2020algorithmic}, H. Boche \& U. M{\"o}nich \cite{boche2020turing} and M. Koller, J. Gro{\ss}mann, U. M{\"o}nich \& H. Boche \cite{koller2018deformation} (which concerns stability of convolutional NNs). Recent results using the SCI hierarchy to establish boundaries in computational mathematics and scientific computing include the work by S. Olver \& M. Webb \cite{webb2017spectra} and the work by J. Ben-Artzi, M. Marletta \& F. R\"{o}sler \cite{Jonathan_res,benartzi2020computing}. There is a vast literature on impossibility results regarding existence of algorithms for different problems in mathematics. The seminal work of S. Weinberger \cite{Weinberger} is a great example. These results can also be interpreted as classification results in the SCI hierarchy, as the framework is flexible and can encompass any model of computation.

Our results should be viewed in connection with the vast literature in approximation theory -- starting with the universal approximation theorem (e.g. see the survey paper \cite{pinkus1999approximation} by A. Pinkus) -- and many follow up papers establishing the great approximation qualities of NNs, see, for example, the work by H. B\"{o}lcskei, P. Grohs, G. Kutyniok \& P. Petersen \cite{Blcskei2019OptimalAW} and the results by I. Daubechies, R. DeVore, S. Foucart, B. Hanin \& G. Petrova \cite{Daubechies2019NonlinearAA} (the article \cite{devore2020neural} of R. DeVore, B. Hanin \& G. Petrova surveys current challenges of NN approximation). What all these results have in common is that there is no algorithm constructing the approximating NNs that are proven to exist. Indeed, the literature is full of existence results that will typically not imply the existence of algorithms computing the NNs. Our results are therefore very much related to the gap between the theory and practice when it comes to computing NNs, as pointed out by B. Adcock \& N. Dexter \cite{adcock2020gap}.

\item[(ii)] {\bf  \emph{Instabilities in DL}}: Initiated by the work of C. Szegedy et. al.\cite{SzZ-14}, there is now a vast literature on the instability phenomenon in DL in a wide variety of applications
\cite{SzZ-14,moosavi2016deepfool,akhtar2018threat,carlini2018audio,huang2018some,antun2020instabilities, finlayson2019adversarial} ranging from image recognition and classification, via audio and speech recognition to automatic diagnosis in medicine, image reconstruction and inverse problems. Thus we can only highlight a small subset here. A significant development was the \textit{DeepFool} research programme and software package of S. Moosavi--Dezfooli, A. Fawzi \& P. Frossard \cite{moosavi2016deepfool}, which was followed by the construction of so-called \textit{universal adversarial perturbations} \cite{DezFaFr-17}.  The construction of and mitigation against \textit{adversarial attacks} is now an active area of research.  To the best of our knowledge, \cite{antun2020instabilities} and \cite{huang2018some} were the first works to demonstrate the instability phenomenon for inverse problems in imaging. There is also an increasing amount of work dedicated to the important problem of numerical instability in ML, see, for example, the recent work of P. Blanchard, D. Higham \& N. Higham \cite{higham_IMA2020}.

\item[(iii)]{\bf  \emph{DL in inverse problems}}: The work of K. Jin, M. McCann, E. Froustey \& M. Unser \cite{jin17} was influential in highlighting the promise of DL for inverse problems in imaging. This is now a rapidly evolving area of research, which we will not attempt to summarise.  See \cite{arridge2019solving, Baraniuk2020} for overviews of current techniques. Note that sparse regularisation has been used as the basis for some DL technique, e.g.\ by using DL to recover the parts of an image that sparse regularisation cannot such as in \cite{BubbaEtAlInvisible}, or by designing NN architectures through the process of \textit{unrolling} an optimisation algorithm (see, e.g., \cite{arridge2019solving}) discussed below. Another approach is to learn variational regularisers, see, for example, \cite{kobler2020total}.

\item[(iv)]{\bf  \emph{Compressed sensing, optimisation and unrolling of algorithms}}:
Our positive results rely on theory from the compressed sensing literature initiated by E. Candes, J. Romberg \& T. Tao \cite{candes2006robust} and D. Donoho \cite{donoho2006compressed}. In particular, our results rely on the many results on structured sampling in structured compressed sensing, see the work by B. Adcock et. al. \cite{adcock2017breaking, adcock2016note, adcock2018oracle}, A. Bastounis et. al. \cite{SIAM_News, bastounis2017absence},  J. Bigot, C. Boyer \& P. Weiss \cite{Boyer_2016, BOYER_ACHA_2019} and G. Kutyniok \& W. Lim \cite{Kutyniok_Lim_2018}, see also the work by F. Krahmer and R. Ward \cite{Felix_2014}. Moreover, our results are closely related to the work of A. Ben-Tal \& A. Nemirovski \cite{benlectures}, who were one of the first to realise how key assumptions in compressed sensing -- such as the \emph{robust nullspace property} -- help bound the error of the approximation to a minimiser (produced by an optimisation algorithm) in terms of error bounds on the approximation to the objective function. Our construction of stable and accurate NNs uses the optimisation problem  $(P_3)$ and approximations of unrolled (or unfolded) primal-dual iterations with a restart scheme: see the discussion at the start of \S \ref{pf_of_constructive_res}. Unrolling iterative methods as NNs was first developed by Gregor \& LeCun in \cite{gregor2010learning} who considered LISTA, a learned version of ISTA, for recover of sparse vectors. This work has been followed by theoretical grantees \cite{chen2018theoretical, liu2018alista}, ensuring the existence of NN with linear convergence towards the minimiser. Yet, neither \cite{chen2018theoretical} nor \cite{liu2018alista}, use the theoretically correct weights, as these can only be computed as solutions of intractably large optimisation problems. Moreover, it is not clear whether the needed assumptions on $A$ ever hold in practice.

Many other algorithms have been unrolled as NNs. For example, see the work of R. Cohen, G. Dardikman--Yoffe, Y. Eldar, J. Geng, Q. He, Y. Li, J. Luo, V. Monga, R. van Sloun, O. Solomon, M. Tofighi, Y. Yang, Y. Zhang \cite{li2020efficient,solomon2019deep,dardikman2020learned}, and A. Aberdam, A. Beck, A. Golts, M. Elad, V. Papyan, Y. Romano, J Sulam \cite{papyan2017convolutional,sulam2019multi,aberdam2020ada}. Typically, such approaches train the weights and biases, and even the activation functions of the NNs (see also the comments after Remark \ref{no_restart}). We shall not attempt a broad survey, and instead point the reader to the papers of M. McCann, K. Jin \& M. Unser \cite{mccann2017convolutional} and V. Monga, Y. Li \& Y. Eldar \cite{monga2019algorithm} for up to date reviews.

To prove convergence of vectors (as opposed to objective functions), it is crucial to have conditions such as the robust null space property (in levels) of Definition \ref{def_rNSPL}. Moreover, this property enables a restart scheme to achieve exponential convergence in the number of layers. Typically, exponential convergence for restart schemes requires a \L{}ojasiewicz-type inequality of the form $\gamma d(x,X^*)^{\nu}\leq f(x)-f^*$, where $f$ is the objective function and $d(x,X^*)$ denotes the distance to the set of minimisers \cite{bolte2007lojasiewicz,karimi2016linear,bolte2017error}. For example, V. Roulet, N. Boumal \& A. d'Aspremont \cite{roulet2020computational} achieve exponential convergence, using the restarted NESTA algorithm \cite{becker2011nesta}, for exact recovery (noiseless) of sparse vectors if $A$ satisfies the null space property.
 By taking into account computability and construction of NNs, and allowing noise, approximate sparsity, sparsity in levels and complex-valued vectors and matrices, our results and framework are more general than \cite{roulet2020computational}. The work of D. Donoho \& Y. Tsaig \cite{donoho2008fast} was one of the first to connect statistical and computational performance by empirically showing that recovery problems with larger sample sizes can also be easier to solve numerically. This message is echoed in our Theorem \ref{NN_FW_recov_thm}.

\end{itemize}

\section{Proof of Theorem \ref{impossibility_theorem} and tools from the SCI hierarchy}\label{niahoigvbojib}

In this section, we prove Theorem \ref{impossibility_theorem}. In order to do this, we will need some analytical results regarding phase transitions of solutions of $(P_1),(P_2)$ and $(P_3)$ which are given in \S \ref{sec:phase_trans}. However, before analysing these phase transitions, we need some preliminary definitions regarding algorithms, inexact inputs, and condition numbers. There are two main reasons for this framework. First, because our definitions are general, they lead to stronger impossibility results than when restricted to specific models of computation. Second, our framework greatly simplifies the proofs and makes it clear what the key mechanisms behind the proofs are (\S \ref{sec:phase_trans} describes this in terms of phase transitions of minimisers). We have sought to make these discussions widely accessible and self-contained.

\subsection{Algorithmic preliminaries: a user-friendly guide}\label{computational_setup}

We begin with a definition of a computational problem, which is deliberately general in order to capture any computational problem.

\begin{definition}[Computational problem]\label{definition:ComputationalProblem}
Let $\Omega$ be some set, which we call the {domain},
and $\Lambda$ be a set of complex valued functions on $\Omega$ such that for $\iota_1, \iota_2 \in \Omega$, then $\iota_1 = \iota_2$ if and only if $f(\iota_1) = f(\iota_2)$ for all $f \in \Lambda$, called an {evaluation} set. Let $(\mathcal{M},d)$ be a metric space, and finally let $\Xi:\Omega\to \mathcal{M}$ be a function which we call the {problem} function. We call the collection $\{\Xi,\Omega,\mathcal{M},\Lambda\}$ a {computational problem}. When it is clear what $\mathcal{M}$ and $\Lambda$ are we will sometimes write $\{\Xi,\Omega\}$ for brevity.
\end{definition}

\begin{remark}[Multivalued problems]
\label{mulivalued_remark}
In some cases, such as when considering the optimisation problems $(P_j)$ that may have more than one solution, we consider $\Xi(\iota)\subset \mathcal{M}$. With an abuse of notation, we then set $d(x,\Xi(\iota))=\mathrm{dist}(x,\Xi(\iota))=\inf_{y\in\Xi(\iota)}d(x,y)$ and this distinction will be made clear from context.\hfill$\boxtimes$
\end{remark}

The set $\Omega$ is the set of objects that give rise to our computational problems. The problem function $\Xi : \Omega\to \mathcal{M}$ is what we are interested in computing. Finally, the set $\Lambda$ is the collection of functions that provide us with the information we are allowed to read as input to an algorithm. For example, $\Omega$ could consist of a collection of matrices $A$ and data $y$ in (\ref{eq:inv_prob0}), $\Lambda$ could consist of the pointwise entries of the vectors and matrices in $\Omega$, $\Xi$ could represent the solution set (with the possibility of more than one solution as in Remark \ref{mulivalued_remark}) of any of the problems $(P_j)$ and $(\mathcal{M},d)$ could be $\mathbb{C}^N$ with the usual Euclidean metric (or any other suitable metric).

Given the definition of a computational problem, we need the definition of a general algorithm, whose conditions hold for any reasonable notion of a deterministic algorithm. Throughout this paper, we deal with the case that $\Lambda=\{f_j\}_{j\in\beta}$, where $\beta$ is some (at most) countable index set.

\begin{definition}[General Algorithm]\label{definition:Algorithm}
Given a  computational problem $\{\Xi,\Omega,\mathcal{M},\Lambda\}$, a {general algorithm} is a mapping $\Gamma:\Omega\to\mathcal{M}$ such that for each $\iota\in\Omega$
\begin{enumerate}[label=(\roman*)]
\item There exists a non-empty finite subset of evaluations $\Lambda_\Gamma(\iota) \subset\Lambda$, \label{property:AlgorithmFiniteInput}
\item The action of $\,\Gamma$ on $\iota$ only depends on $\{\iota_f\}_{f \in \Lambda_\Gamma(\iota)}$ where $\iota_f \coloneqq  f(\iota),$\label{property:AlgorithmDependenceOnInput}
\item For every $\kappa\in\Omega$ such that $\kappa_f=\iota_f$ for every $f\in\Lambda_\Gamma(\iota)$, it holds that $\Lambda_\Gamma(\kappa)=\Lambda_\Gamma(\iota)$.\label{property:AlgorithmSameInputSameInputTaken}\end{enumerate}
If, in addition, there exists a canonical ordering $\Lambda_\Gamma(\iota)=\{f^{\Gamma}_{\iota,1}=f_{k_1},...,f^{\Gamma}_{\iota,S_{\Gamma}(\iota)}=f_{k_{S_{\Gamma}(\iota)}}\}$, where $S_{\Gamma}(\iota)=\left|\Lambda_\Gamma(\iota)\right|$, such that if $\kappa\in\Omega$ and $f^{\Gamma}_{\iota,j}(\iota)=f^{\Gamma}_{\iota,j}(\kappa)$ for all $j\leq r< S_{\Gamma}(\iota)$, then $f^{\Gamma}_{\iota,j}=f^{\Gamma}_{\kappa,j}$ for all $j\leq r+1$, then we call $\Gamma$ a Sequential General Algorithm. In this case, we use the notation $k_j(\Gamma,\iota)$ to denote the ordered indices corresponding to the evaluation functions that the algorithm reads.
\end{definition}

The three properties of a general algorithm are the most basic natural properties we would expect any deterministic computational device to obey. The first condition says that the algorithm can only take a finite amount of information, though it is allowed adaptively to choose, depending on the input, the finite amount of information that it reads. The second condition ensures that the algorithm's output only depends on its input, or rather the information that it has accessed (or ``read''). The final condition is very important and ensures that the algorithm produces outputs and consistently accesses information. In other words, if it sees the same information for two different inputs, then it cannot behave differently for those inputs. Note that the definition of a general algorithm is more general than the definition of a Turing machine \cite{turing1937computable} or a Blum--Shub--Smale (BSS) machine \cite{BCSS}, which can be thought of as digital and analog computational devices respectively. In particular, a general algorithm has no restrictions on the operations allowed. The extra condition for a sequential general algorithm is satisfied by any algorithm defined by a computational machine with input tape of readable information (one should think of the ordered indices of the evaluation functions as corresponding to sequentially reading the tape which encodes the input information). Hence, a sequential general algorithm is still more general than a Turing or a BSS machine. Complete generality in Definition \ref{definition:Algorithm} is used for two primary reasons:
\begin{itemize}
	\item[(i)] {\it Strongest possible bounds:} Since Definition \ref{definition:Algorithm} is completely general, the lower bounds hold in any model of computation, such as a Turing machine or a BSS machine. On the other hand, the algorithms we construct in this paper are made to work using only arithmetic operations over the rationals. Hence, we obtain the strongest possible lower bounds and the strongest possible upper bounds.
	\item[(ii)] {\it Simplified exposition:} Using the concept of a general algorithm considerably simplifies the proofs of lower bounds and allows us to see precisely the mechanisms behind the proofs.
\end{itemize}

Next, we consider the definition of a randomised general algorithm, which again is more general than a probabilistic Turing or probabilistic BSS machine. Randomised algorithms are widely used in practice in areas such as optimisation, algebraic computation, machine learning, and network routing. See, for example, the early papers of von Neumann \cite{MR0077479} and de Leeuw et al. \cite{MR0079550} that describe probabilistic Turing machines. In the case of Turing machines, it is currently unknown, in the sense of polynomial runtime, whether randomisation is beneficial from a complexity class viewpoint \cite[Ch. 7]{arora2009computational}, however, rather intriguingly this is not the case for BSS machines \cite[Ch. 17]{BCSS} (some of the proofs in this reference are non-constructive - it is an open problem whether any probabilistic BSS machine can be simulated by a deterministic machine having the same machine constants and with only a polynomial slowdown). Nevertheless, randomisation is an extremely useful tool in practice. ({A famous example is the problem of primality testing. The first algorithm that did this in polynomial-time (in the size of the representation of the number) was probabilistic \cite{solovay1977fast} and it was only decades later that a deterministic polynomial time algorithm was discovered \cite{agrawal2004primes}.}) From a machine learning point of view, we also want to consider randomised algorithms to capture procedures such as stochastic gradient descent which are commonly used to train NNs. As developed in \cite{opt_big}, the concept of a general algorithm can be extended to a randomised general algorithm. This concept allows for universal impossibility results regardless of the computational model.

\begin{definition}[Randomised General Algorithm]\label{definition:ProbablisticAlgorithm}
Given a  computational problem $\{\Xi,\Omega,\mathcal{M},\Lambda\}$, a {Randomised General Algorithm} (RGA) $\Gamma^{\mathrm{ran}}$ is a collection $X$ of general algorithms $\Gamma:\Omega\to\mathcal{M}$, a sigma-algebra $\mathcal{F}$ on $X$ and a family of probability measures $\{\mathbb{P}_{\iota}\}_{\iota \in \Omega}$ on $\mathcal{F}$ such that the following conditions hold:
		\begin{enumerate}
			\item For each $\iota \in \Omega$, the mapping $\Gamma^{\mathrm{ran}}_{\iota}:(X,\mathcal{F}) \to (\mathcal{M}, \mathcal{B})$ defined by $\Gamma^{\mathrm{ran}}_{\iota}(\Gamma) = \Gamma(\iota)$ is a random variable, where $\mathcal{B}$ is the Borel sigma-algebra on $\mathcal{M}$. \label{property:PAlgorithmMeasurable}
			\item For each $n \in \mathbb{N}$ and $\iota \in\Omega$, the set $\lbrace \Gamma \in X \, : \, \sup\{m\in\mathbb{N}: f_m\in\Lambda_{\Gamma}(\iota)\} \leq n \rbrace \in \mathcal{F}$.
			 \label{property:PAlgorithmRTMeasurable}

			\item For each $\iota_1,\iota_2 \in \Omega$ and $E \in \mathcal{F}$, such that for every $\Gamma \in E$ we have $f(\iota_1) = f(\iota_2)$ for every $f \in \Lambda_{\Gamma}(\iota_1)$, then $\mathbb{P}_{\iota_1}(E) = \mathbb{P}_{\iota_2}(E)$.
			\label{property:PAlgorithmConsistent}
			\end{enumerate}
With slight abuse of notation, we denote the family of randomised general algorithms by $\mathrm{RGA}$.
\end{definition}

The first two conditions are measure theoretic to avoid pathological cases and ensure that ``natural sets'' one might define for a random algorithm (such as notions of stopping times) are measurable. These conditions hold for all standard probabilistic machines (such as a Turing or BSS machine). The third condition ensures consistency, namely, that in the case of identical evaluations, the laws of the output cannot change. Finally, we will use the standard definition of a probabilistic Turing machine (which is a particular case of Definition \ref{definition:ProbablisticAlgorithm}). However, to make sense of probabilistic Turing machines in our context (in particular, to restrict operations to the rationals which can be encoded in the natural numbers), we must define the notion of inexact input.

Suppose we are given a computational problem $\{\Xi, \Omega, \mathcal{M}, \Lambda\}$, and that $\Lambda = \{f_j\}_{j \in \beta}$, where we remind the reader that $\beta$ is some index set that can be finite or countably infinite. However, obtaining $f_j$ may be a computational task on its own, which is exactly the problem in most areas of computational mathematics. In particular, for $\iota \in \Omega$, $f_j(\iota)$ could be the number $e^{\frac{\pi}{j} i }$ for example. Hence, we cannot access or store $f_j(\iota)$ on a computer, but rather $f_{j,n}(\iota)$ where $f_{j,n}(\iota) \rightarrow f_{j}(\iota)$ as $n \rightarrow \infty$. This idea is formalised in the definition below, however, to put this in perspective it is worth mentioning the Solvability Complexity Index (SCI) hierarchy.

\begin{remark}[The Solvability Complexity Index (SCI) hierarchy]
The SCI of a computational problem is the smallest number of limits needed in order to compute the solution. The full hierarchy is described in \cite{ben2015can}, and the mainstay of the hierarchy are the $\Delta^{\alpha}_k$ classes. The $\alpha$ denotes the model of computation that would indicate the Turing or the BSS model. Informally, we have the following description. Given a collection $\mathcal{C}$ of computational problems, then
\begin{itemize}
\item[(i)] $\Delta^{\alpha}_0$ is the set of problems that can be computed in finite time, the SCI $=0$.
\item[(ii)] $\Delta^{\alpha}_1$ is the set of problems that can be computed using one limit (the SCI $=1$) with control of the error, i.e. $\exists$ a sequence of algorithms $\{\Gamma_n\}$ such that $d(\Gamma_n(\iota), \Xi(\iota)) \leq 2^{-n}, \, \forall \iota \in \Omega$.
\item[(iii)] $\Delta^{\alpha}_2$ is the set of problems that can be computed using one limit (the SCI $=1$) without error control, i.e. $\exists$ a sequence of algorithms $\{\Gamma_n\}$ such that $\lim_{n\rightarrow \infty}\Gamma_n(\iota) = \Xi(\iota), \, \forall \iota \in \Omega$.
\item[(iv)] $\Delta^{\alpha}_{m+1}$, for $m \in \mathbb{N}$, is the set of problems that can be computed by using $m$ limits, (the SCI $\leq m$), i.e. $\exists$ a family of algorithms $\{\Gamma_{n_m, \hdots, n_1}\}$ such that
\[
\lim_{n_m \rightarrow\infty}\hdots \lim_{n_1\rightarrow\infty}\Gamma_{n_m,\hdots, n_1}(\iota) = \Xi(\iota), \, \forall \iota \in \Omega.\eqno\boxtimes
\]
 \end{itemize}
\end{remark}
The above hierarchy gives rise to the concept of '$\Delta_1$-information'. That is, in informal terms, the problem of obtaining the inexact input to the computational problem is a $\Delta_1$ problem. One may think of an algorithm taking the number $\exp(1)$ or $\sqrt{2}$ as input. Indeed, one can never produce an exact version of these numbers to the algorithm, however, one can produce an approximation to an arbitrarily small error.

\begin{definition}[$\Delta_{1}$-information]\label{definition:Lambda_limits}
	Let $\{\Xi, \Omega, \mathcal{M}, \Lambda\}$ be a computational problem. We say that $\Lambda$ has $\Delta_{1}$-information if each $f_j \in \Lambda$ is not available, however, there are mappings $f_{j,n}: \Omega \rightarrow \mathbb{Q} + i \mathbb{Q}$ such that $|f_{j,n}(\iota)-f_j(\iota)|\leq 2^{-n}$ for all $\iota\in\Omega$. Finally, if $\widehat \Lambda$ is a collection of such functions described above such that $\Lambda$ has $\Delta_1$-information, we say that $\widehat \Lambda$ provides $\Delta_1$-information for $\Lambda$. Moreover, we denote the family of all such $\widehat \Lambda$ by $\mathcal{L}^1(\Lambda)$.
\end{definition}

Note that we want to have algorithms that can handle all  computational problems $\{\Xi,\Omega,\mathcal{M},\widehat \Lambda\}$ whenever $\widehat \Lambda \in  \mathcal{L}^1(\Lambda)$. In order to formalise this, we define what we mean by a computational problem with $\Delta_1$-information.

\begin{definition}[Computational problem with $\Delta_1$-information]
A computational problem where $\Lambda$ has $\Delta_1$-information is denoted by
	$
	\{\Xi,\Omega,\mathcal{M},\Lambda\}^{\Delta_1} \coloneqq  \{\widetilde \Xi,\widetilde \Omega,\mathcal{M},\widetilde \Lambda\},
	$
where
\[
\widetilde \Omega = \left\{ \widetilde \iota = \{f_{j,n}(\iota)\}_{j, n\in \beta \times \mathbb{N}} \, : \, \iota \in \Omega, \{f_j\}_{j \in \beta} = \Lambda, |f_{j,n}(\iota)-f_j(\iota)|\leq 2^{-n}\right\},
\]
Moreover, if $\widetilde \iota = \{f_{j,n}(\iota)\}_{j, n\in \beta \times \mathbb{N}}\in \widetilde \Omega$ then we define $\widetilde \Xi(\widetilde \iota) = \Xi(\iota)$ and $\widetilde f_{j,n}(\widetilde \iota) = f_{j,n}(\iota)$. We also set $\widetilde \Lambda = \{\widetilde f_{j,n}\}_{j,n \in \beta \times \mathbb{N}}$. Note that $\widetilde \Xi$ is well-defined by Definition \ref{definition:ComputationalProblem} of a computational problem and the definition of $\widetilde \Omega$ includes all possible instances of $\Delta_1$-information $\widehat \Lambda\in \mathcal{L}^1(\Lambda)$.
\end{definition}

We can now define a probabilistic Turing machine for $\{\Xi,\Omega,\mathcal{M}, \Lambda\}$, where the algorithm $\Gamma$ is executed by a Turing machine \cite{turing1937computable}, that has an oracle tape consisting of $\{\widetilde\iota_f\}_{f \in \widetilde\Lambda}$. In what follows, we have deliberately not written down the (lengthy) definition of a Turing machine (found in any standard text \cite{du2011theory,arora2009computational}), which one should think of as an effective algorithm or computer programme (the famous Church--Turing thesis).

\begin{definition}
\label{oaoingvoa}
Given the definition of a Turing machine, a probabilistic Turing machine for $\{\Xi,\Omega,\mathcal{M}, \Lambda\}$ is a Turing machine that has an oracle tape consisting of $\{\widetilde\iota_f\}_{f \in \widetilde\Lambda}$ (for $\widetilde\iota\in\widetilde\Omega$), with an additional read only tape containing independent binary random numbers ($0$ or $1$ with equal probability), and which halts with probability one and outputs a single element of $\mathcal{M}$. The law of such a machine will be denoted by $\mathbb{P}$. With an abuse of notation, we sometimes denote the probabilistic Turing machine by $(\Gamma,\mathbb{P})$.
\end{definition}

\begin{remark}[Where does the output live?]
Strictly speaking, when we say that the output of a probabilistic Turing machine lies in $\mathcal{M}$, we mean that the output is a natural number that corresponds, via an encoding, to an element of a subset of $\mathcal{M}$ such as $(\mathbb{Q}+i\mathbb{Q})^N\subset\mathbb{C}^N$. However, we follow the usual convention of suppressing such encodings in our discussions and proofs. Occasionally we will also consider sequences of (probabilistic) Turing machines, in which case there is an additional input parameter $n$, which ranges over the natural numbers.\hfill$\boxtimes$
\end{remark}

One should think of Definition \ref{oaoingvoa} as an algorithm for the computational problem with inexact input, but with the additional ability to generate random numbers (this corresponds to the binary input tape) and execute commands based on the sequence of random numbers that are generated. The reader should intuitively think of this as a computer program with a random number generator. For equivalent definitions and the basic properties of such machines, we refer the reader to \cite{du2011theory,arora2009computational}. For simplicity, we have only considered probabilistic Turing machines that halt with probability one, though extensions can be made to non-halting machines. Note that Definition \ref{oaoingvoa} is a special case of Definition \ref{definition:ProbablisticAlgorithm}, where $\mathbb{P}_{\iota}=\mathbb{P}$ is fixed across different $\iota$. In particular, given a probabilistic Turing machine, the sigma-algebra and probability distribution generated by the standard product topology on $\{0,1\}^{\mathbb{N}}$ induce the relevant collection $X$ of Turing machines and sigma-algebra $\mathcal{F}$, as well as $\mathbb{P}$.

Finally, we also recall the standard definitions of condition used in optimisation \cite{Renegar1, Renegar2,BCSS,Condition}. The classical condition number of an invertible matrix $A$ is given by
$
\mathrm{Cond}(A) = \|A\|\|A^{-1}\|.
$
For different types of condition numbers related to a possibly multivalued (signified by the double arrow) mapping $\Xi:{\Omega} \subset \mathbb{C}^n \rightrightarrows \mathbb{C}^N$ we need to establish what types of perturbations we are interested in. For example, if ${\Omega}$ denotes the set of diagonal matrices (which we treat as elements of $\mathbb{C}^n$ for some $n$), we may not be interested in perturbations in the off-diagonal elements as they will always be zero. In particular, we may only be interested in perturbations in the coordinates that are varying in the set ${\Omega}$. Thus, given ${\Omega} \subset  \mathbb{C}^n$ we define the active coordinates of ${\Omega}$ to be
$\mathrm{Act}=\mathrm{Act}\left({\Omega}\right) = \{j \, : \, \exists \, x,y \in {\Omega}, x_j \neq y_j\}$. Moreover, for $\nu > 0$ (including the obvious extension to $\nu = \infty$) we define
$$
{\Omega}_{\nu} = \left\{x \, : \, \exists \, y \in {\Omega} \text{ such that }  \|x-y\|_{l^\infty} \leq \nu,  x_{\mathrm{Act}^c} = y_{\mathrm{Act}^c} \right\}.
$$
In other words, ${\Omega}_{\nu}$ is the set of $\nu$-perturbations along the non-constant coordinates of elements in ${\Omega}$. We can now recall some of the classical condition numbers from the literature \cite{Condition,BCSS,Renegar1,Renegar2}.

\begin{labeling}{\,\,}

	\item[\,\,\,\,(1) {\it Condition of a mapping}:]
	Let
	$\Xi: {\Omega} \subset \mathbb{C}^n \rightrightarrows \mathbb{C}^m$ be a linear or non-linear mapping, and suppose that $\Xi$ is also defined on ${\Omega}_{\nu}$ for some $\nu >0$. Then,
	$$
	\mathrm{Cond}\left(\Xi,{\Omega}\right)= \sup_{x \in {\Omega}} \lim_{\epsilon \rightarrow 0^+} \sup_{\substack{x+z \in {\Omega}_{\nu}\\ 0 < \| z \|_{l^2}  \leq \epsilon}} \left \{ \frac{\mathrm{dist}(\Xi(x+z),\Xi(x))}{\| z \|_{l^2}} \right \},
	$$
	where we allow for multivalued functions by defining
	$
	\mathrm{dist}(\Xi(x+z),\Xi(z)) = \inf_{w_1 \in \Xi(x+z), w_2 \in \Xi(z)}\|w_1-w_2\|_{l^2}
	$ (see Remark \ref{mulivalued_remark}). We will use this notion of condition number for $(P_1)$, $(P_2)$ and $(P_3)$.
	\vspace{2mm}

	\item[\,\,\,\,(2) {\it Distance to infeasibility - the Feasibility Primal condition number}:]
For the problem $(P_1)$ of basis pursuit (for $(P_2)$ and $(P_3)$ the following condition number is always zero) we set
	$$
	\nu(A,y) = \sup\big\{\epsilon\geq0 \, : \,  \|\hat y\|_{l^2}, \|\hat A\| \leq \epsilon, (A+\hat A,y+\hat y) \in \Omega_{\infty} \Rightarrow  (A+\hat A,y+\hat y) \text{ are feasible inputs to }\Xi_{P_1}\big\},
	$$
	and define the {\it Feasibility Primal} (FP) local condition number
	$$
	C_{\mathrm{FP}}(A,y)\coloneqq  \frac{\max\left\{\|y\|_{l^2}, \|A\|\right\}}{\nu(A,y)}.
	$$
We then define the FP global condition number via
	$$
	C_{\mathrm{FP}}\left(\Xi_{P_1},{\Omega}\right)\coloneqq \sup_{(A,y)\in{\Omega}}C_{\mathrm{FP}}(A,y).
	$$
	\end{labeling}

\begin{remark}[Condition numbers in different norms]
One can also define the above condition numbers using norms different from the $l^2$-norm. However, the results of Theorem \ref{impossibility_theorem} are given for the $l^2$-norm so we stick to the above cases (also note that for finite-dimensional spaces, all norms are equivalent).\hfill$\boxtimes$
\end{remark}

\subsection{Phase transitions}
\label{sec:phase_trans}

In order to prove Theorem \ref{impossibility_theorem}, we will need the following lemmas which describe phase transitions of the minimisers of the respective optimisation problems.

\begin{lemma}[\textbf{Phase transition for basis pursuit}]
\label{phase_1}
Let $N\geq 2$ and consider the problem $(P_1)$ for
$$A=\begin{pmatrix}\frac{w_1}{\rho_1}&\frac{w_2}{\rho_2}&\cdots&\frac{w_N}{\rho_N}\end{pmatrix}\in\mathbb{C}^{1\times N}, \quad y=1,\quad \epsilon\in[0,1),$$ where $\rho_j>0$ for $j=1,...,N$. Then the set of solutions is given by
\begin{equation}
\label{PTB11}
\sum_{j=1}^N\left[t_j(1-\epsilon)\frac{\rho_j}{w_j}\right]e_j,\text{ s.t. }t_j\in[0,1],\sum_{j=1}^Nt_j=1\text{ and }t_j=0\text{ if }\rho_j>\min_{k}\rho_k,
\end{equation}
where $e_j$ correspond to the canonical basis of $\mathbb{C}^N$.
\end{lemma}

\begin{proof}
Let $\hat x_j=x_jw_j\rho_j^{-1}$, then the optimisation problem becomes
\begin{equation}
\label{eq:bp_rewritten}
\argmin_{\hat x \in \mathbb{C}^{N}} f(\hat x)\coloneqq \sum_{j=1}^N\rho_j\left|\hat x_j\right|\text{ such that }\left|1-\sum_{j=1}^N\hat x_j\right|\leq\epsilon.
\end{equation}
Since $\epsilon<1$ and the $(\rho_1,\rho_2,...\rho_N)$ weighted $l^1$ norm is convex, it follows that the solution must lie on the hypersurface segment $\hat x_1+\hat x_2+...+\hat x_j=1-\epsilon$ for $\hat x_j\in\mathbb{R}_{\geq 0}$. We now claim that if $\hat{x}$ is a solution of \eqref{eq:bp_rewritten}, and $\rho_j > \min_{k}\rho_k$, then $\hat{x}_{j} = 0$. Suppose for a contradiction that there exists a solution $\hat x$  of $\eqref{eq:bp_rewritten}$ where $\hat x_j>0$ and $\rho_j>\min_{k}\rho_k$. Pick any $l$ such that $\rho_l=\min_{k}\rho_k$, then $\hat x+\hat x_j (e_l-e_j)$ is feasible with $f(\hat x+\hat x_j (e_l-e_j))<f(\hat x)$, a contradiction. Similarly, if $x$ is of the form (\ref{PTB11}), then $f(\hat x)=(1-\epsilon)\min_{k}\rho_k$. In particular, the objective function is constant over the set of all such vectors and the result follows.
\end{proof}

\begin{lemma}[\textbf{Phase transition for LASSO}]
\label{phase_2}
Let $N\geq 2$ and consider the problem $(P_2)$ for
$$A=\lambda\begin{pmatrix}\frac{w_1}{\rho_1}&\frac{w_2}{\rho_2}&\cdots&\frac{w_N}{\rho_N}\end{pmatrix}\in\mathbb{C}^{1\times N}, \quad y=1,$$ where $0<\rho_j<2$ for $j=1,...,N$.  Then the set of solutions is given by
\begin{equation}
\label{PTB12}
\left(1-\frac{\min_{k}\rho_k}{2}\right)\sum_{j=1}^N\frac{\rho_jt_j}{\lambda w_j}e_j,\text{ s.t. }t_j\in[0,1],\sum_{j=1}^Nt_j=1\text{ and }t_j=0\text{ if }\rho_j>\min_{k}\rho_k,
\end{equation}
where $e_j$ correspond to the canonical basis of $\mathbb{C}^N$.
\end{lemma}

\begin{proof}
Let $\hat x_j=x_j\lambda w_j\rho_j^{-1}$, then the optimisation problem becomes
$$
\argmin_{\hat x \in \mathbb{C}^{N}} f(\hat x)\coloneqq \left|1-\sum_{j=1}^N\hat x_j\right|^2+\sum_{j=1}^N\rho_j\left|\hat x_j\right|.
$$
It is clear that any optimal solution must be real and hence we restrict our argument to real $\hat x$. Define the $2^N$ quadrant subdomains $D_{k_1,\ldots, k_N}  =\{\hat x_j \cdot(-1)^{k_j}>0\}$ for $k_j\in\{0,1\}$, and notice that
\[ \nabla f(\hat x) = \begin{bmatrix}
-2(1-\sum_{j=1}^{N} \hat x_j) + (-1)^{k_1}\rho_1 \\
\vdots \\
-2(1-\sum_{j=1}^{N} \hat x_j) + (-1)^{k_N}\rho_N
\end{bmatrix}, \quad \text{for } \hat x \in D_{k_1,\ldots, k_N}. \]
We first look for stationary points of the objective function in the subdomains $D_{k_1,...,k_N}$. The condition for a stationary point in the interior of such a domain leads to the constraint that $k_1=k_2=...=k_N$. If $k_1=k_2=...=k_N=1$, then $\nabla f=0$ leads to the contradiction $\rho_j=2(\hat x_1+...+\hat x_N)-2<0$. Finally, in the case (and only in the case) of $\rho_1=\rho_2=...=\rho_N$, there is a hypersurface segment of stationary points in $D_{0,0,...,0}$ given by $\hat x_1+...+\hat x_N=1-\rho_1/2$ (recall that we assumed $\rho_1<2$ so this segment exists).

First, consider the case that $\rho_1=...=\rho_N$. Then any optimal solution must either lie on the boundary of some $D_{k_1,...,k_N}$ or on the hypersurface segment $\hat x_1+...+\hat x_N=1-\rho_1/2$ in $D_{0,0,...,0}$. A simple case by case analysis now yields that the solutions $\hat x$ are given by convex combinations of $(1-\rho_j/2)e_j$ for $j=1,...,N$. Now consider the case that not all of the $\rho_j$ are equal. Then any optimal solution must lie on the boundary of some $D_{k_1,...,k_N}$. A simple case by case analysis now yields that the solutions $\hat x$ are given by convex combinations of $(1-\rho_j/2)e_j$ for $j$ such that $\rho_j=\min_{k}\rho_k$. Rescaling back to the variable $x$ now gives the result.
\end{proof}

\begin{lemma}[\textbf{Phase transition for square-root LASSO}]
\label{phase_3}
Let $N\geq 2$ and consider the problem $(P_3)$ for
$$A=\lambda\begin{pmatrix}\frac{w_1}{\rho_1}&\frac{w_2}{\rho_2}&\cdots&\frac{w_N}{\rho_N}\end{pmatrix}\in\mathbb{C}^{1\times N}, \quad y=1,$$ where $0<\rho_j<1$ for $j=1,...,N$.  Then the set of solutions is given by
\begin{equation}
\label{PTB13}
\sum_{j=1}^N\frac{\rho_jt_j}{\lambda w_j}e_j,\text{ s.t. }t_j\in[0,1],\sum_{j=1}^Nt_j=1\text{ and }t_j=0\text{ if }\rho_j>\min_{k}\rho_k,
\end{equation}
where $e_j$ correspond to the canonical basis of $\mathbb{C}^N$.
\end{lemma}

\begin{proof}
Let $\hat x_j=x_j\lambda w_j\rho_j^{-1}$, then the optimisation problem becomes
$$
\argmin_{\hat x \in \mathbb{C}^{N}} f(\hat x)\coloneqq \left|1-\sum_{j=1}^N\hat x_j\right|+\sum_{j=1}^N\rho_j\left|\hat x_j\right|.
$$
It is clear that any optimal solution must be real and hence we restrict our argument to real $\hat x$. The objective function is piecewise affine and since $\rho_j< 1$, the gradient of $f$ is non-vanishing on the interior of any of the domains $D_{k_1,...,k_N}=\{\hat x_j \cdot(-1)^{k_j}>0\}$ for $k_j\in\{0,1\}$. It follows that the optimal solutions must lie on the boundaries of the domains $D_{k_1,...,k_N}$. A simple case by case analysis shows that the solutions $\hat x$ are given by convex combinations of $e_j$ for $j$ such that $\rho_j=\min_{k}\rho_k$. Rescaling back to the variable $x$ now gives the result.
\end{proof}

We will also need the following propositions, which give useful criteria for impossibility results.

\begin{proposition}\label{prop:DrivingNegativePropsition1}
Let $\{\Xi, \Omega, \mathcal{M}, \Lambda\}$ be a computational problem. Suppose that there are two  sequences $\{\iota^1_n\}_{n\in\mathbb{N}}$, $\{\iota^2_n\}_{n\in\mathbb{N}} \subset \Omega$ satisfying the following conditions:
	\begin{enumerate}[label=(\alph*)]
		\item There are sets $S^1, S^2 \subset \mathcal{M}$ and $\kappa > 0$ such that
		$
		\inf_{x_1 \in S^1, x_2 \in S^2}d(x_1,x_2) > \kappa
		$ and $\Xi(\iota^j_n) \subset S^j$ for $j=1,2$. \label{property:MinimisersOfNonZero}
		\item For every $f \in \Lambda$ there is a $c_f \in \mathbb{C}$ such that $|f(\iota^j_n) - c_f|\leq 1/4^n$ for all $n \in\mathbb{N}$ and $j = 1,2$.  \label{property:Del1Info}
	\end{enumerate}
	Then, if we consider $\{\Xi,\Omega,\mathcal{M},\Lambda\}^{\Delta_1}$, we have the following:
	\begin{enumerate}[label = (\roman*)]
	\item For any sequential general algorithm $\Gamma$ and $M\in\mathbb{N}$, there exists $\iota\in\Omega$ and $\widehat\Lambda\in\mathcal{L}^1(\Lambda)$ such that
	$$
	\mathrm{dist}(\Gamma(\iota),\Xi(\iota))> \kappa/2\quad\text{or}\quad S_{\Gamma}(\iota)> M.
	$$
	\item If there is an $\iota^0 \in \Omega$ such that for every $f \in \Lambda$ we have that \ref{property:Del1Info} is satisfied with $c_f = f(\iota^0)$, then for any RGA $\Gamma$ and $p\in[0,1/2)$, there exists $\iota\in\Omega$ and $\widehat\Lambda\in\mathcal{L}^1(\Lambda)$ such that
	$$
	\mathbb{P}_{\iota}\Big(\mathrm{dist}(\Gamma(\iota),\Xi(\iota))\geq \kappa/2\Big)> p.
	$$
	\end{enumerate}
\end{proposition}

\begin{proof}
Without loss of generality, we assume that $\Omega=\{\iota^1_n\}_{n\in\mathbb{N}}\cup\{\iota^2_n\}_{n\in\mathbb{N}}.$ Part (ii) follows immediately from a Proposition of \cite{opt_big}, so we only prove part (i). Let $\Gamma$ be a sequential general algorithm and $M\in\mathbb{N}$. We will construct the required $\widehat\Lambda\in\mathcal{L}^1(\Lambda)$ inductively. By Definition \ref{definition:Algorithm} and the setup of $\Delta_1-$information for $\Lambda$, there exists some $f_{k_1}\in\Lambda$ and $n_1\in\mathbb{N}$ such that for all $\iota\in\Omega$ and for all $\widehat\Lambda\in\mathcal{L}^1(\Lambda)$, we have $f^{\Gamma}_{\iota,1}=f_{k_1,n_1}$. We set $f_{k_1,n_1}(\iota_{m}^j)=c_{f_{k_1}}$ for all $m\geq n_1$ and choose $f_{k_1,n_1}(\iota_{m}^j)$ consistently for $m<n_1$. Again by Definition \ref{definition:Algorithm} and the setup of $\Delta_1-$information for $\Lambda$, it follows that there exists $f_{k_2}\in\Lambda$ and $n_2\in\mathbb{N}$ (which without loss of generality $\geq n_1$) such that for all $m\geq n_1$, either $\Lambda_{\Gamma}(\iota_m^j)=\{f_{k_1,n_1}\}$ or $f^{\Gamma}_{\iota^j_m,2}=f_{k_2,n_2}$. In the latter case, we set $f_{k_2,n_2}(\iota_{m}^j)=c_{f_{k_2}}$ for all $m\geq n_2$ and choose $f_{k_2,n_2}(\iota_{m}^j)$ consistently for $m<n_2$. We continue this process for a maximum of $M$ steps up to
\[
f^{\Gamma}_{\iota^j_m,\min\{M,|\Lambda_{\Gamma}(\iota^j_m)|\}}
\]
 as follows. At the $q$th step after defining $f_{k_q,n_q}$, by Definition \ref{definition:Algorithm} and the setup of $\Delta_1-$information for $\Lambda$, it follows that there exists $f_{k_{q+1}}\in\Lambda$ and $n_{q+1}\in\mathbb{N}$ (which without loss of generality $\geq n_q$) such that for all $m\geq n_q$, either $\Lambda_{\Gamma}(\iota_m^j)\subset\{f_{k_1,n_1},...,f_{k_q,n_q}\}$ or $f^{\Gamma}_{\iota^j_m,q+1}=f_{k_{q+1},n_{q+1}}$. In the latter case, we set $f_{k_{q+1},n_{q+1}}(\iota_{m}^j)=c_{f_{k_{q+1}}}$ for all $m\geq n_{q+1}$ and choose $f_{k_{q+1},n_{q+1}}(\iota_{m}^j)$ consistently for $m<n_{q+1}$. We can then choose the rest of the function values to obtain $\widehat\Lambda$.

Given this $\widehat\Lambda\in\mathcal{L}^1(\Lambda)$, suppose for a contradiction that for any $\iota\in\Omega$,
$$
\mathrm{dist}(\Gamma(\iota),\Xi(\iota))\leq \kappa/2\quad\text{and}\quad S_{\Gamma}(\iota)\leq M.
$$
Without loss of generality, we assume that the above construction is carried out for $M$ steps. It follows that we must have $f(\iota^1_{n_M})=f(\iota^2_{n_M})$ for all $f\in \widehat \Lambda_{\Gamma}(\iota^1_{n_M})$. By (ii) and (iii) of Definition \ref{definition:Algorithm}, it follows that $\Gamma(\iota^1_{n_M})=\Gamma(\iota^2_{n_M})$. Let $\epsilon>0$ be arbitrary. Since $\mathrm{dist}(\Gamma(\iota),\Xi(\iota))\leq \kappa/2$ for all $\iota\in\Omega$, there exists $s_j\in S^j$ such that $d(\Gamma(\iota^j_{n_M}),s_j)<\kappa/2+\epsilon$. It follows that
$$
\inf_{x_1 \in S^1, x_2 \in S^2}d(x_1,x_2)\leq d(s_1,s_2)\leq d(\Gamma(\iota^1_{n_M}),s_1)+d(\Gamma(\iota^2_{n_M}),s_2)<\kappa+2\epsilon.
$$
Since $\epsilon>0$ was arbitrary, we have
$$
\inf_{x_1 \in S^1, x_2 \in S^2}d(x_1,x_2)\leq \kappa,
$$
the required contradiction.
\end{proof}

\begin{proposition}\label{prop:DrivingNegativePropsition2}
Let $\{\Xi, \Omega, \mathcal{M}, \Lambda\}$ be a computational problem and $u>1$ be a positive integer. Suppose that there are $u$ sequences $\{\iota^j_n\}_{n\in\mathbb{N}}\subset \Omega$, for $j=1,...,u$, satisfying the following conditions:
	\begin{enumerate}[label=(\alph*)]
		\item There are sets $S^j\subset \mathcal{M}$, for $j=1,...,u$, and $\kappa > 0$ such that
		$
		\inf_{x_j \in S^j, x_k \in S^k}d(x_j,x_k) > \kappa
		$ for any $j\neq k$ and $\Xi(\iota^j_n) \subset S^j$ for $j=1,...,u$. \label{property:MinimisersOfNonZero2}
		\item For every $f \in \Lambda$, there is a $c_f \in \mathbb{C}$ such that $|f(\iota^j_n) - c_f|\leq 1/4^n$ for all $n \in\mathbb{N}$ and $j = 1,...,u$.
	\end{enumerate}
	Then, if we consider $\{\Xi,\Omega,\mathcal{M},\Lambda\}^{\Delta_1}$, for any halting probabilistic Turing machine $(\Gamma,\mathbb{P})$, $M\in\mathbb{N}$ and $p\in [0,(u-1)/u)$, there exists $\iota\in\Omega$ and $\widehat\Lambda\in\mathcal{L}^1(\Lambda)$ such that
	$$
	\mathbb{P}\Big(\mathrm{dist}(\Gamma(\iota),\Xi(\iota))> \kappa/2\quad\text{or}\quad S_{\Gamma}(\iota)> M\Big)> p.
	$$
\end{proposition}

\begin{proof}
Without loss of generality, we assume that $\Omega=\cup_{j=1}^u\{\iota^j_n\}_{n\in\mathbb{N}}$. Let $(\Gamma,\mathbb{P})$ be a halting probabilistic Turing machine and $M\in\mathbb{N}$, $p\in [0,(u-1)/u)$. Suppose for a contradiction that for all $\iota\in\Omega$ and all $\widehat\Lambda\in\mathcal{L}^1(\Lambda)$, we have
	$$
	\mathbb{P}\Big(\mathrm{dist}(\Gamma(\iota),\Xi(\iota))> \kappa/2\quad\text{or}\quad S_{\Gamma}(\iota)> M\Big)\leq p.
	$$
We will construct the required $\widehat\Lambda\in\mathcal{L}^1(\Lambda)$ inductively. Let $\beta\in(0,1)$ be such that $(1-\beta)^M-p>1/u$. Such a $\beta$ exists since $p\in [0,(u-1)/u)$. Since the Turing machine must halt with probability one, there exists finite sets $K_1,N_1\subset\mathbb{N}$ such that with probability (w.r.t. $\mathbb{P}$) at least $1-\beta$, for all $\iota\in\Omega$ and for all $\widehat\Lambda\in\mathcal{L}^1(\Lambda)$, it holds that $f^{\Gamma}_{\iota,1}=f_{k_1,n_1}$ for some $k_1\in K_1$ and $n_1\in N_1$. We set $f_{k_1,n_1}(\iota_{m}^j)=c_{f_{k_1}}$ for all $m\geq \max\{n_1:n_1\in N_1\}$ and choose $f_{k_1,n_1}(\iota_{m}^j)$ consistently otherwise.

We continue this process inductively for $M$ steps up to $f^{\Gamma}_{\iota^j_m,\min\{M,|\Lambda_{\Gamma}(\iota^j_m)|\}}$ as follows. At the $q$th step after defining $f_{k_q,n_q}$ for $k_q\in K_q$ and $n_q\in N_q$, it follows (again since the Turing machine halts with probability one) that there exists finite sets $K_{q+1},N_{q+1}\subset\mathbb{N}$ with the following property. Let $E_{q+1}$ be the event that for all $\iota^j_m\in\Omega$ with $m\geq \max\{n:n\in N_1\cup...\cup N_q\}$, either $f^{\Gamma}_{\iota^j_m,q+1}=f_{k_{q+1},n_{q+1}}$, for some $k_{q+1}\in K_{q+1}$ and $n_{q+1}\in N_{q+1}$, or $|\Lambda (\iota^j_m)|\leq q$. Then
$$
\mathbb{P}(E_{q+1}|\cap_{k\leq q}E_k)\geq 1-\beta.
$$
We then set $f_{k_{q+1},n_{q+1}}(\iota_{m}^j)=c_{f_{k_{q+1}}}$ for all $m\geq \max\{n:n\in N_1\cup...\cup N_{q+1}\}$ and choose $f_{k_{q+1},n_{q+1}}(\iota_{m}^j)$ consistently otherwise. This ensures the existence of $K_{q+2}$ and $N_{q+2}$. After completing the $M$th step, we can choose the rest of the function values to obtain $\widehat\Lambda$.

It follows that for $m\geq \max\{n:n\in N_1\cup...\cup N_{M}\}$, the outputs $\Gamma(\iota^j_m)$ conditional on the event \[
E_1\cap...\cap E_M\cap \{S_{\Gamma}(\cdot)\leq M\}
\] are equal for $j=1,...,u$. Since $\inf_{x_j \in S^j, x_k \in S^k}d(x_1,x_2)>\kappa$ for $j\neq k$, it follows that the events
$$
F_j\coloneqq \{\mathrm{dist}(\Gamma(\iota_m^j),\Xi(\iota_m^j))\leq \kappa/2\}\cap \{S_{\Gamma}(\iota_m^j)\leq M\}\cap E_1\cap...\cap E_M, \quad j=1,...,u
$$
are disjoint. Moreover, using the fact that $\mathbb{P}(A\cap B)=\mathbb{P}(A)+\mathbb{P}(B)-\mathbb{P}(A\cup B)$,
\begin{align*}
\mathbb{P}(F_j)&\geq \mathbb{P}(\{\mathrm{dist}(\Gamma(\iota_m^j),\Xi(\iota_m^j))\leq \kappa/2\}\cap \{S_{\Gamma}(\iota_m^j)\leq M\})+\mathbb{P}(E_1\cap...\cap E_M)-1\\
&\geq \mathbb{P}(\{\mathrm{dist}(\Gamma(\iota_m^j),\Xi(\iota_m^j))\leq \kappa/2\}\cap \{S_{\Gamma}(\iota_m^j)\leq M\})+(1-\beta)^M-1\\
&\geq (1-\beta)^M-p>1/u.
\end{align*}
But this contradicts the disjointness of the $F_j$'s.
\end{proof}

\subsection{Proof of Theorem \ref{impossibility_theorem}}

\begin{proof}[Proof of Theorem \ref{impossibility_theorem}]
We will argue for $m=1$ and construct such an $\Omega$ for this case. The general case of $m>1$ follows by embedding our construction for the case of a given $A\in\mathbb{C}^{1\times (N+1-m)}$ in the first row of matrices and vectors of the form
$$
\hat A=\begin{pmatrix} A & 0\\ 0 & \alpha I\end{pmatrix},\quad \hat y =(y,0)^{\top},\quad (A,y)\in\Omega,
$$
where $I\in\mathbb{C}^{(m-1)\times(m-1)}$ denotes the $(m-1)\times(m-1)$ identity matrix and $\alpha=\alpha(A)$ is chosen such that $\hat A\hat A^*$ is a multiple of the identity. In particular, such an embedding does not effect the relevant condition numbers (it is straightforward to see that the matrix norm, distance to infeasibility for ($P_1$) and condition numbers of the mappings are all unchanged).

For the classes we consider, the setup of Theorem \ref{impossibility_theorem} coincides with the $\Delta_1-$information model discussed in \S \ref{computational_setup}. In particular, we can use Lemmas \ref{phase_1}, \ref{phase_2} and \ref{phase_3} to derive the relevant $x_{s,n}$'s in (\ref{delt1_inf2}). This means that we can apply Propositions \ref{prop:DrivingNegativePropsition1} and \ref{prop:DrivingNegativePropsition2} with the metric corresponding to the $l^2$-norm. Recall that for this theorem, we assume that $w_1=w_2=...=w_N=1$.

\textbf{Step 1:} Proof for $(P_1)$. First, consider the class defined by
$$
\Omega_{1}=\left\{(A(\gamma_1;\rho),y):A(\gamma_1;\rho)\coloneqq \gamma_1\begin{pmatrix} \frac{1}{\rho_1}&\frac{1}{\rho_2} & \cdots & \frac{1}{\rho_N}\end{pmatrix}, y=1, \rho_j\in[1-2\delta,1-\delta]\right\},
$$
for fixed $\gamma_1>10$ and $\delta\in(0,1/4)$. We choose $\gamma_1$ and $\delta$ such that
\begin{align}
\frac{1-\epsilon}{\gamma_1}\cdot\sup_{\rho_j,\rho_k\in[1-2\delta,1-\delta],j\neq k}&\left\|\rho_je_j-{\rho_k}e_k\right\|_{l^2}=3\cdot 10^{-K},\label{con101}\\
\frac{1-\epsilon}{\gamma_1}\cdot\inf_{\rho_j,\rho_k\in[1-2\delta,1-\delta],j\neq k}&\left\|\rho_je_j-{\rho_k}e_k\right\|_{l^2}>2\cdot 10^{-K}\label{con11},
\end{align}
where the $e_j$ denote the canonical basis of $\mathbb{C}^N$. Note that we can ensure $\gamma_1>10$ since $\epsilon\leq 1/2$ and $K> 2$. If $\rho_j\in[1-2\delta,1-\delta)$, {for $j=1,2$,} then by (a simple rescale of) Lemma \ref{phase_1},
\begin{align*}
\Xi_{P_1}(A(\gamma_1;(\rho_1,1-\delta,...,1-\delta)^{\top}),1)&=\frac{(1-\epsilon)\rho_1}{\gamma_1}e_1,\\ \Xi_{P_1}(A(\gamma_1;(1-\delta,\rho_2,1-\delta,...,1-\delta)^{\top}),1)&=\frac{(1-\epsilon)\rho_2}{\gamma_1}e_2.
\end{align*}
Since (\ref{con11}) holds, it follows by selecting appropriate sequences $\iota_n^j$ for choices of $\rho_j=\rho_j^n\uparrow1-\delta$ that the conditions of Proposition \ref{prop:DrivingNegativePropsition1} hold for $\Omega_1$ with
\begin{equation}
\label{need_sjaihoi}
S^j=\left\{\frac{1-\epsilon}{\gamma_1}\rho e_j:\rho\in[1-2\delta,1-\delta]\right\},\quad \kappa=2\cdot 10^{-K}.
\end{equation}
Moreover, the condition for part (ii) of Proposition \ref{prop:DrivingNegativePropsition1} also holds with $\iota^0=(A(\gamma_1;(1-\delta,1-\delta,...,1-\delta)^{\top}),1)$.

Now suppose for a contradiction that there exists a (halting) RGA (with input $\iota_{A,\mathcal{S}}$) and $p>1/2$ that produces a NN $\phi_A$ such that
$$
\min_{y \in \mathcal{S}_A} \inf_{x^*\in \Xi_{P_1}(A,y)}\left\|\phi_A(y)-x^*\right\|_{l^2}\leq 10^{-K}
$$
holds with probability at least $p$ for all $(A,y)\in\Omega_1$. Then there exists a (halting) RGA, $\Gamma$, taking $\iota_{A,\mathcal{S}}$ as input that computes a solution of $(P_1)$ to $K$ correct digits with probability at least $p$ on each input in $\Omega_1$. However, this contradicts Proposition \ref{prop:DrivingNegativePropsition1} (ii).
Next, consider the class defined by
$$
\Omega_{2}=\left\{(A(\gamma_2;\rho),y):A(\gamma_2;\rho)=\gamma_2\begin{pmatrix} \frac{1}{\rho_1}&\frac{1}{\rho_2} & \cdots & \frac{1}{\rho_N}\end{pmatrix}, y=1, \rho_j\in[1-2\delta,1-\delta], \rho_j\neq\rho_k\text{ if }j\neq k\right\},
$$
where $\gamma_2=\gamma_1/10>1$ so that
\begin{align}
\sup_{\rho_j,\rho_k\in[1-2\delta,1-\delta],j\neq k}&\left\|\rho_je_j-{\rho_k}e_k\right\|_{l^2}=3\cdot 10^{-K+1}\cdot\frac{\gamma_2}{1-\epsilon},\label{con20}\\
\inf_{\rho_j,\rho_k\in[1-2\delta,1-\delta],j\neq k}&\left\|\rho_je_j-{\rho_k}e_k\right\|_{l^2}>2\cdot 10^{-K+1}\cdot\frac{\gamma_2}{1-\epsilon}\label{con2}.
\end{align}
By extending the argument above to $u=N+1-m=N$ (recall without loss of generality that $m=1$) sequences and sets $S^j$ defined as in (\ref{need_sjaihoi}), the conditions of Proposition \ref{prop:DrivingNegativePropsition2} hold with $\kappa=2\cdot 10^{-K+1}$. Now suppose that there exists a (halting) probabilistic Turing machine $(\Gamma,\mathbb{P})$, $M\in\mathbb{N}$ and $p\in\left[0,\frac{N-m}{N+1-m}\right)$, such that for any $(A,1)\in\Omega_2$, $\Gamma$ computes a NN $\phi_A$ with
$$
\mathbb{P}\left(\inf_{x^*\in \Xi_{P_1}(A,y)}\left\|\phi_A(y)-x^*\right\|_{l^2}>10^{1-K}\text{ or the sample size needed to construct }\phi_A>M\right)\leq p.
$$
Then there exists a (halting) probabilistic Turing machine that computes a solution of $(P_1)$ to $K-1$ correct digits on each input in $\Omega_2$ with sample size at most $M$ with probability at least $1-p$. However, this contradicts Proposition \ref{prop:DrivingNegativePropsition2}.

We now set $\Omega=\Omega_1\cup\Omega_2$. Note that the negative statements of part (i) and (ii) follow from the above arguments by considering restrictions to $\Omega_1$ and $\Omega_2$ respectively. Hence, we are left with proving the condition number bounds, part (iii) and the positive part of part (ii).  First, note that $\mathrm{Cond}(AA^*)=1$ for any $(A,y)\in\Omega$. For any $(A,y)\in\Omega$, we have $\nu(A,y)=\|A\|\geq 1=\|y\|_{l^2}$ and hence $C_{\mathrm{FP}}(\Xi_{P_1},\Omega)\leq 1.$ To bound the final condition number, first note that if $\rho_j,\rho_j'\leq 1$, then
$$
\left\|\rho-\rho'\right\|_{l^2}\leq \left\| \sum_{j=1}^N\left(\frac{1}{\rho_j}-\frac{1}{\rho_j'}\right)e_j\right\|_{l^2}.
$$
Let $(A(\gamma_1;\rho),1)\in\Omega_1$, then if $(A(\gamma_1;\rho'),1)\in\Omega_1$ with
$$
\Delta(\rho,\rho')\coloneqq \gamma_1\left\| \sum_{j=1}^N\left(\frac{1}{\rho_j}-\frac{1}{\rho_j'}\right)e_j\right\|_{l^2}
$$
sufficiently small, then we must have that
$$
\mathrm{dist}(\Xi_{P_1}(A(\gamma_1;\rho'),1),\Xi_{P_1}(A(\gamma_1;\rho),1))\leq \frac{1-\epsilon}{\gamma_1}\left\|\rho-\rho'\right\|_{l^2}.
$$
It follows that
$$
\lim_{\beta\downarrow 0}\sup_{\substack{(A(\gamma_1;\rho'),1)\in\Omega_1\\ \Delta(\rho,\rho')\leq \beta}}\frac{\mathrm{dist}(\Xi_{P_1}(A(\gamma_1;\rho'),1),\Xi_{P_1}(A(\gamma_1;\rho),1))}{\Delta(\rho,\rho')}\leq \frac{1-\epsilon}{\gamma_1^2}<1.
$$
A similar argument holds for $(A(\gamma_2;\rho),1)\in\Omega_2$, and hence $\mathrm{Cond}(\Xi_{P_1},\Omega)\leq 1$.

We now prove the positive parts of (ii) and (iii). We begin with (ii) and describe the algorithm informally, noting that the output of the algorithm, which we denote by $\Gamma(A)$, gives rise to a NN which maps $y=1$ to $\Gamma(A)\in\mathbb{C}^N$. Given an input $(A,y)\in\Omega$, the algorithm first tests the size of $A_{1,1}$ to determine whether $(A,y)\in\Omega_1$ or $(A,y)\in\Omega_2$. Explicitly, we note that $A_{1,1}$ is positive and bounded away from 0. Hence, with one sample from $\iota_{A,\mathcal{S}}$ we can determine $A_{1,1}$ to an accuracy of at least $0.01\cdot A_{1,1}$ and such that, simultaneously, the corresponding approximation of $A_{1,1}^{-1}$ is accurate to at least $10^{-K}$. Note that if $(A,y)\in\Omega_1$, then $A_{1,1}\in\gamma_1\cdot[1,2]$ whereas if $(A,y)\in\Omega_2$, then $A_{1,1}\in\gamma_2\cdot[1,2]$. Since $\gamma_2=\gamma_1/10$, this level of accuracy is enough to determine whether $(A,y)\in\Omega_1$ or $(A,y)\in\Omega_2$. Next, if the algorithm determines $(A,y)\in\Omega_1$, it outputs the corresponding approximation of
$$
(1-\epsilon)A_{1,1}^{-1}e_1=\frac{1-\epsilon}{\gamma_1}\rho_1e_1
$$
correct to $10^{-K}$ in the $l^2$-norm from the sample. Since
$$
\sup_{\rho_j,\rho_k\in[1-2\delta,1-\delta],j\neq k}\left\|\rho_je_j-{\rho_k}e_k\right\|_{l^2}=3\cdot 10^{-K}\cdot\frac{\gamma_1}{1-\epsilon},
$$
it follows that
$$
\inf_{x^*\in \Xi_{P_1}(A,y)}\|\Gamma(A)-x^*\|_{l^2}\leq 4 \cdot 10^{-K}<10^{-K+1}.
$$
On the other hand, if the algorithm determines $(A,y)=(A(\gamma_2;\rho),y)\in\Omega_2$, then we know that all of the $\rho_j$ are distinct. The algorithm continues to sample $\iota_{A,\mathcal{S}}$ until we determine $j$ such that $\rho_j=\min_{k}\rho_k$. It then outputs an approximation of
$$
(1-\epsilon)A_{1,j}^{-1}e_j=\Xi_{P_1}(A,y)
$$
correct to $10^{-K}$ in the $l^2$-norm. Such as approximation can be computed using $\iota_{A,\mathcal{S}}$. It then follows that
$$
\|\Gamma(A)-\Xi_{P_1}(A,y)\|_{l^2}\leq 10^{-K}<10^{-K+1}
$$
and this finishes the proof of (ii). Finally, to prove (iii), note that the arguments above show that, given an input $(A,y)\in\Omega$, we can use one sample ($L=1$) of $\iota_{A,\mathcal{S}}$ to compute an approximation of $A_{1,1}^{-1}$ with error bounded by $10^{-K}$. We simply set $\Gamma(A)$ to be $(1-\epsilon)e_1$ multiplied by the approximation of $A_{1,1}^{-1}$. Using (\ref{con101}) and (\ref{con20}), it follows that
$$
\inf_{x^*\in \Xi_{P_1}(A,y)}\|\Gamma(A)-x^*\|_{l^2}\leq 3 \cdot 10^{-K+1}+10^{-K}<10^{-K+2}.
$$

\textbf{Step 2:} Proof for $(P_2)$. This is almost identical step 1 with replacing Lemma \ref{phase_1} with Lemma \ref{phase_2}. The other changes are replacing $\epsilon$ with the suitable $\rho_i/2$ in the solution of each LASSO problem, including the additional scale $\lambda$ in the definition of the matrices (see Lemma \ref{phase_2}) and choosing $\gamma_1\lambda>10$ and $\delta\in(0,1/4)$ such that (recall that $\lambda\leq 1$)
\begin{align}
\sup_{\rho_j,\rho_k\in[1-2\delta,1-\delta],j\neq k}\frac{1}{\lambda\gamma_1}\cdot&\left\|\rho_j(1-\rho_j/2)e_j-{\rho_k}(1-\rho_k/2)e_k\right\|_{l^2}=3\cdot 10^{-K}\label{con10}\\
\inf_{\rho_j,\rho_k\in[1-2\delta,1-\delta],j\neq k}\frac{1}{\lambda\gamma_1}\cdot&\left\|\rho_j(1-\rho_j/2)e_j-{\rho_k}(1-\rho_k/2)e_k\right\|_{l^2}>2\cdot 10^{-K}\label{con1}.
\end{align}
Let $f(x)=(1-x/2)x$, then for $x\in[0,1]$, $|f'(x)|\leq 1$. It follows that
$$
\left\|\sum_{j=1}^N\left(\rho_j(1-\rho_j/2)-\rho_j'(1-\rho_j'/2)\right)e_j\right\|_{l^2}\leq \left\| \sum_{j=1}^N\left(\frac{1}{\rho_j}-\frac{1}{\rho_j'}\right)e_j\right\|_{l^2}.
$$
Hence, for sufficiently small $\delta$, for any $(A(\gamma_1;\rho),1)\in\Omega_1$  (recall the additional factor of $\lambda$) and $\rho'$ sufficiently close to $\rho$ with $(A(\gamma_1;\rho'),1)\in\Omega_1$,
$$
\frac{\mathrm{dist}(\Xi_{P_2}(A(\gamma_1;\rho'),1),\Xi_{P_2}(A(\gamma_1;\rho),1))}{\gamma_1\lambda\left\| \sum_{j=1}^N\left(\frac{1}{\rho_j}-\frac{1}{\rho_j'}\right)e_j\right\|_{l^2}}\leq \frac{1}{\gamma_1^2\lambda^2}<1,
$$
with the same bound holding for $\Omega_2$. It follows that $\mathrm{Cond}(\Xi_{P_2},\Omega)\leq 1$.

\textbf{Step 3:} Proof for $(P_3)$. This is almost identical step 2 with replacing Lemma \ref{phase_2} with Lemma \ref{phase_3}, and deleting the corresponding factors of $1-\rho_j/2$.
\end{proof}

\subsection{Details on numerical example in \S \ref{niahbignion}}
\label{desc_num_exp}

In this section, we elaborate on the numerical example in \S \ref{niahbignion}. The example is a simplification of the arguments found in the proof of Theorem \ref{impossibility_theorem} and is using Lemma \ref{phase_3} extensively. In our experiment, we use $N_1 = 2$ and $\lambda=1$, but for full generality we do not keep these parameters fixed in the discussion below. We assume throughout that $\lambda \in (0,1]$ and $N_1 \geq 2$. The experiment is done for real matrices, so that the LISTA network architecture can be used.

Let $\gamma > 0$, $\rho \in \mathbb{R}^{N_1}\setminus \{0\}$, and let $D \in \mathbb{C}^{N_2+1\times N_2 + 1}$ denote a unitary discrete cosine transform matrix. Define the matrix
\[ A(\gamma, \rho) \coloneqq D\begin{pmatrix} a(\gamma,\rho)^{\top} & 0 \\ 0 & \|a(\gamma, \rho)\|_{l^2} I \end{pmatrix}, \quad\text{where}\quad
a(\gamma,\rho)^{\top} \coloneqq \gamma\begin{pmatrix} \tfrac{1}{\rho_1} & \cdots & \tfrac{1}{\rho_{N_1}} \end{pmatrix} \in \mathbb{R}^{1 \times N_1},
\]
and $I \in \mathbb{R}^{N_2 \times N_2}$ is the identity matrix.
Observe that $A(\gamma, \rho) \in \mathbb{R}^{m\times N}$, with $N=N_1+N_2$ and $m=N_2 + 1$ and that $A$ has irrational entries (hence only approximations can be used in real-life computations). Furthermore, let $\delta = 1/6$ and $\gamma_K = \tfrac{\sqrt{2}}{3\lambda}(1-\delta)\cdot 10^{K}$, where $K$ is the parameter from Theorem \ref{impossibility_theorem}. Also let
\begin{equation} \label{eq:y}
y(x^{(2)}) = D \begin{pmatrix} 1 & \|a(\gamma,\rho)\|_{l^2}x^{(2)} \end{pmatrix}^{\top} \in \mathbb{R}^{N_2+1} \quad \text{ for }\quad x^{(2)} \in \mathbb{R}^{N_2}
\end{equation}
and
$
\Omega_K = \left\{ \left(y(x^{(2)}), A(\gamma_K,\rho)\right) : \rho_j \in [1-2\delta, 1-\delta] , x^{(2)} \in \mathbb{R}^{N_2} \right\}.
$
Next define
\[\rho' = \begin{pmatrix} \rho_{1}' & 1-\delta &\cdots &\cdots & 1-\delta \end{pmatrix}
\quad \text{ and }\quad
\rho^{\sharp} = \begin{pmatrix}  1-\delta & \rho_{2}^{\sharp} &1-\delta &\cdots & 1-\delta  \end{pmatrix}
\]
where $\rho_{1}^{\prime}, \rho_{2}^{\sharp} \in [1-2\delta, 1-\delta)$. We let $e_i$ denote the $i$'th canonical basis vector for $\mathbb{R}^{N_1}$ and define $x' = \tfrac{\rho_{1}^{\prime}}{\lambda \gamma_K}e_1$ and $x^{\sharp} = \tfrac{\rho_{1}^{\sharp}}{\lambda \gamma_K}e_2$. For this choice of parameters we have from Lemma \ref{phase_3} and the fact that $D$ is unitary that $\left(x^{\prime}, x^{(2)}\right)^{\top} \in \Xi_3\left(A(\gamma_K,\rho'), y(x^{(2)})\right)$ and $\left(x^{\sharp}, x^{(2)}\right)^{\top} \in \Xi_3\left(A(\gamma_K,\rho^{\sharp}), y(x^{(2)})\right)$.

Observer that we can let $A(\gamma_K, \rho')$ and $A(\gamma_K,\rho^{\sharp})$ become arbitrary close by letting $\rho_{1}^{\prime}, \rho_{2}^{\sharp} \uparrow 1-\delta$. We will let $\rho_{1}'=\rho_{2}^{\sharp}$, and notice that for this choice the data $y(x^{(2)})$ are the same for both inputs. However, regardless of the choice of $\rho_{1}', \rho_{2}^{\sharp} \in [1-2\delta,1-\delta)$ the minimisers for the two problems are bounded away from each other. In particular we have that
\begin{align*}
    \inf_{\rho_{1}', \rho_{2}^{\sharp} \in [1-2\delta, 1-\delta]} \frac{1}{\lambda \gamma_K} \| \rho_{1}'e_1 - \rho_{2}^{\sharp} e_2 \|_{l^2} > 2\cdot 10^K
\quad\text{and}\quad
    \sup_{\rho_{1}', \rho_{2}^{\sharp} \in [1-2\delta, 1-\delta]} \frac{1}{\lambda \gamma_K} \| \rho_{1}'e_1 - \rho_{2}^{\sharp} e_2 \|_{l^2} = 3\cdot 10^K
\end{align*}
which implies that $10^K < \| (x',x^{(2)})^{\top} - (x^{\sharp},x^{(2)})^{\top} \|_{l^2} < 10^{K+1}$.

In the numerical experiment in \S \ref{niahbignion}, we take $A = A(\gamma_K, \rho')$ with $\rho_{1}'= 1-\delta + 2^{-n-1}$, and approximate this matrix with the matrix $A_n = A(\gamma_K, \rho^{\sharp})$, where the parameter $\rho_{2}^{\sharp} = 1-\delta+2^{-n-1}$. This ensures that $\|A-A_n\| \leq 2^{n}$. For the trained neural network, we used 8000 triples of the form
\begin{equation} \label{eq:sparse_training}
\iota_{A,\mathcal{S}, n} = \left\{ \left(y_{k,n}(x^{(2)}_{k,n}), A_n, (x^{\sharp}_{n}, x^{(2)}_{k,n})^{\top}\right): k=1,\ldots, 8000, \text{ and } x^{(2)} \text{ is $5$-sparse}  \right\}.
\end{equation}
for $n=10,20,30$. Note that it is not necessary to make the $x^{(2)}$ component sparse. This is done merely to make the experiment more realistic, as the main usage of the problems $(P_j)$ are for recovery of sparse vectors.

\section{Proof of Theorem \ref{key_CS_prop}}
\label{pf_of_constructive_res}

A roadmap for the proof is as follows. We consider the problem $(P_3)$ and unroll iterations of Chambolle and Pock's primal-dual algorithm \cite{chambolle2011first, chambolle2016ergodic, he2012convergence}. These iterations are approximated by NNs in Theorem \ref{BP_prop1}, where we obtain bounds on a rescaled version of the objective function in (\ref{SQRT_LASSO_BD1}). The assumption of weighted rNSPL then allows us to relate the bounds proven in Theorem \ref{BP_prop1} to bounds on the distance of the output of the NN to the wanted vector and also, simultaneously, prove stability. This also allows the acceleration to exponential convergence through a restart scheme (with a reweighting at each restart).

We begin with the proof of Lemma \ref{rNSPL_perturb}, which allows us to consider the approximation matrices $A_l$ in the construction of the NNs. We also state some results from compressed sensing that are needed in our proofs. We then discuss preliminary results on unrolling iterative algorithms for $(P_3)$, which are used in the proof of Theorem \ref{key_CS_prop}. When writing out NNs in the proofs, we will use $\xrightarrow[]{\text{NL}}$ arrows to denote the non-linear maps and $\xrightarrow[]{\text{L}}$ arrows to denote the affine maps.

\subsection{Some results from compressed sensing}
\label{compressed_results}

\begin{proof}[Proof of Lemma \ref{rNSPL_perturb}]
Let $\Delta$ be a $(\textup{\textbf{s}},\textup{\textbf{M}})$ support set and $x\in\mathbb{C}^N$, then we have
\begin{align}
\|x_{\Delta}\|_{l^2}&\leq \frac{\rho\|x_{\Delta^c}\|_{l^1_w}}{\sqrt{\xi}}+\gamma\|Ax\|_{l^2}\notag\\
&\leq \frac{\rho\|x_{\Delta^c}\|_{l^1_w}}{\sqrt{\xi}}+\gamma\|\hat Ax\|_{l^2} +\gamma\|\hat A-A\|\|x\|_{l^2}.\label{fst_bd}
\end{align}
Note that
$$
\min_{k=1,...,r}w_{(k)}^2\sum_{j\in\Delta^c}\left|x_j\right|^2\leq \left(\sum_{j\in\Delta^c}\left|x_j\right|w_j\right)^2=\|x_{\Delta^c}\|^2_{l^1_w}.
$$
Hence, (\ref{fst_bd}) implies that
$$
\|x_{\Delta}\|_{l^2}\leq \frac{\rho\|x_{\Delta^c}\|_{l^1_w}}{\sqrt{\xi}}+\gamma\|\hat Ax\|_{l^2}+\gamma\|\hat A-A\|\left(\|x_{\Delta}\|_{l^2}+\frac{\|x_{\Delta^c}\|_{l^1_w}}{\min_{k=1,...,r}w_{(k)}}\right).
$$
Rearranging now gives the result.
\end{proof}

The following results are taken from the compressed sensing literature. Lemma \ref{CS_lem1} is Lemma 5.2 from \cite{adcock2019uniform} and Lemma \ref{CS_lem2} is Lemma 5.3 from \cite{adcock2019uniform}.

\begin{lemma}[\textbf{rNSPL implies $l^1_w$ distance bound}]
\label{CS_lem1}
Suppose that $A$ has the weighted rNSPL of order $(\textup{\textbf{s}},\textup{\textbf{M}})$ with constants $0<\rho<1$ and $\gamma>0$. Let $x,z\in\mathbb{C}^N$, then
\begin{equation}
\|z-x\|_{l^1_w}\leq \frac{1+\rho}{1-\rho}\left(2\sigma_{\textup{\textbf{s}},\textup{\textbf{M}}}(x)_{l^1_w}+\|z\|_{l^1_w}-\|x\|_{l^1_w}\right)+\frac{2\gamma}{1-\rho}\sqrt{\xi}\|A(z-x)\|_{l^2}.
\end{equation}
\end{lemma}

\begin{lemma}[\textbf{rNSPL implies $l^2$ distance bound}]
\label{CS_lem2}
Suppose that $A$ has the weighted rNSPL of order $(\textup{\textbf{s}},\textup{\textbf{M}})$ with constants $0<\rho<1$ and $\gamma>0$. Let $x,z\in\mathbb{C}^N$, then
\begin{equation}
\|z-x\|_{l^2}\leq \left(\rho+\frac{(1+\rho)\kappa^{1/4}}{2}\right)\frac{\|z-x\|_{l^1_w}}{\sqrt{\xi}}+\left(1+\frac{\kappa^{1/4}}{2}\right)\gamma\|A(z-x)\|_{l^2}.
\end{equation}
\end{lemma}

\subsection{Preliminary constructions of neural networks}

When constructing NNs, we will make use of the following maps from $\mathbb{C}^M$ to itself, defined for various $M\in\mathbb{N}$ and $\beta\in\mathbb{Q}_{>0}$ by
\begin{equation*}
\psi_\beta^0(x)=\max\left\{0,1-\frac{\beta}{\|x\|_{l^2}}\right\}x,\quad\psi^1(x)=\min\left\{1,\frac{1}{\|x\|_{l^2}}\right\}x.
\end{equation*}

\begin{lemma}
\label{key_prox_lemma0}
Let $M\in\mathbb{N}$, $\beta\in\mathbb{Q}_{>0}$ and $\theta\in\mathbb{Q}_{>0}$. Then there exists neural networks $\phi_{\beta,\theta}^0,\phi_{\theta}^1\in\mathcal{N}_{\textup{\textbf{D}},3,2}$ with $\textup{\textbf{D}}=(M,2M,M+1,M)$ such that
$$
\left\|\phi_{\beta,\theta}^0(x)-\psi_\beta^0(x)\right\|_{l^2}\leq\theta,\quad \left\|\phi_{\theta}^1(x)-\psi^1(x)\right\|_{l^2}\leq\theta, \quad \forall x\in\mathbb{C}^M,
$$
and such that the non-linear maps can be computed from $\mathrm{sqrt}_\theta$ and finitely many arithmetic operations and comparisons.
\end{lemma}

\begin{proof}
We deal only with the case of $\psi_\beta^0$ since the case of $\psi^1$ is nearly identical. Consider the maps
\begin{equation*}
\begin{split}
\phi_{\beta,\theta}^0:x\xrightarrow[]{\text{L}} \begin{pmatrix}
x\\
x
\end{pmatrix} \xrightarrow[]{\text{NL}}\begin{pmatrix}
|x_1|^2\\
|x_2|^2\\
\vdots\\
|x_{M}|^2\\
x
\end{pmatrix}\xrightarrow[]{\text{L}}\begin{pmatrix}
\sum_{j=1}^{M}|x_j|^2\\
x
\end{pmatrix}&\xrightarrow[]{\text{NL}}
\begin{pmatrix}
0\\
\max\left\{0,1-\frac{\beta}{\mathrm{sqrt}_\theta(\|x\|_{l^2}^2)}\right\}x
\end{pmatrix}\\&\xrightarrow[]{\text{L}}
\max\left\{0,1-\frac{\beta}{\mathrm{sqrt}_\theta(\|x\|_{l^2}^2)}\right\}x.
\end{split}
\end{equation*}
The first, third and final arrows are simple affine maps. The second arrow applies pointwise modulus squaring, which can be done using finitely many arithmetic operations. The penultimate arrow applies a non-linear map which can be computed from one application of $\mathrm{sqrt}_\theta$ and finitely many arithmetic operations and comparisons. The bound
$$
\left\|\max\left\{0,1-\frac{\beta}{\|x\|_{l^2}}\right\}x-\phi_{\beta,\theta}^0(x)\right\|_{l^2}\leq \theta,
$$
follows from a simple case by case analysis.
\end{proof}

The final piece of machinery needed will be the approximation, via a NN, of applying a pointwise version of $\psi_\beta^0$.

\begin{lemma}
\label{key_prox_lemma}
Let $s,\theta\in\mathbb{Q}_{> 0}$, $w\in\mathbb{Q}_{>0}^N$ and for $\hat x\in\mathbb{C}^N$ consider the minimisation problem
\begin{equation}
\label{key_prox}
\argmin_{x \in \mathbb{C}^{N}} \|x\|_{l^1_w}+s\|x-\hat x\|_{l^2}^2.
\end{equation}
Let $\tilde x_s(\hat x)$ denote the solution of (\ref{key_prox}). Then, there exists $\phi_{s,\theta}\in\mathcal{N}_{\textup{\textbf{D}},2,1}$ such that
\begin{equation}
\label{key_prox_lemma_EQ1}
\|\phi_{s,\theta}(\hat x)-\tilde x_s(\hat x)\|_{l^2}\leq\theta\|w\|_{l^2}, \quad \forall\hat x\in\mathbb{C}^N
\end{equation}
and $\textup{\textbf{D}}=(N,N,N)$. Each affine map in the neural network is linear and is an arithmetic function of $w$. Moreover, the non-linear maps used can be computed from $\mathrm{sqrt}_\theta$ and finitely many arithmetic operations and comparisons.
\end{lemma}

\begin{proof}
Let $B=\mathrm{diag}(w_1,...,w_N)\in\mathbb{Q}^{N\times N}$ and consider the function
$$
F(y)=\frac{1}{2s}\left\|By\right\|_{l^1}=\frac{1}{2s}\|y\|_{l^1_w}.
$$
Then we can write the minimisation problem (\ref{key_prox}) as $\mathrm{prox}_{F}(\hat x)$. Given $y\in\mathbb{C}^N$, we identify $y=(y_1,y_2)^{\top}\in\mathbb{R}^{2N}$.

First, for $\beta > 0$ and $x\in \mathbb{R}^n$ we recall that the proximal operator of (a multiple of) the $l^2$-norm can be written as
\begin{equation}
\label{djhibiuwf}
\prox_{\beta \|\cdot\|_{l^2}}(x) = \max\left\{0, 1- \frac{\beta}{\|x\|_{l^2}}\right\} x.
\end{equation}
Thus, for $\beta>0$ we define
$$
\varphi_\beta(y)=(v(y,\beta)*y_1,v(y,\beta)*y_2)^{\top},
$$
where $*$ denotes pointwise multiplication and for $j=1,...,N$ we have
$$
v(y,\beta)_j=\max\left\{0,1-\frac{\beta }{\sqrt{y_{1,j}^2+y_{2,j}^2}}\right\}.
$$
The function $\varphi_\beta$ simply corresponds to a proximity map of the $l^2$-norm applied component-wise to the complexified version of $y$. Using (\ref{djhibiuwf}), we have
\begin{align*}
    \prox_{F}(y) &= \argmin_{z \in \mathbb{C}^{N}} \frac{1}{2s}\|Bz\|_{l^1} + \frac{1}{2} \| z-y \|_{l^2}^{2} \\
     &= \argmin_{z \in \mathbb{C}^{N}} \sum_{j=1}^{N} \left( \frac{B_{jj}}{2s}\sqrt{z_{1,j}^2 + z_{2,j}^2} + \frac{1}{2}\left((z_{1,j} - y_{1,j})^2 + (z_{2,j} - y_{2,j})^2 \right) \right)
		\end{align*}
		It follows that (in complex vector form)
		$$
[\prox_{F}(y)]_j= \max\left\{ 0, 1 - \frac{B_{jj}/(2s)}{|y_j|}\right\} y_j, \quad\text{for }j=1,\ldots, N.
$$
{We can therefore write}
$$
\prox_F(y)=B\varphi_{(2s)^{-1}}(B^{-1}y).
$$
We then unroll the computation of ${\prox_{F}}(\hat x)$ via:
$$
\hat x
\xrightarrow[]{\text{L}}
B^{-1}\hat x
\xrightarrow[]{\text{NL}}
 \varphi_{(2s)^{-1}}(B^{-1}\hat x)
\xrightarrow[]{\text{L}}
 B\varphi_{(2s)^{-1}}(B^{-1}y).
$$
The first arrow is a simple linear map, the second applies $\varphi_{(2s)^{-1}}$ and the third is a linear map. We approximate this by replacing $v(y,\beta)_jy_j$ with $\phi^0_{\beta,\theta}(y_{1,j}+y_{2,j}i)$ (denoting the replacement of $\varphi_{(2s)^{-1}}$ by $\varphi_{(2s)^{-1}}^{\theta}$) where $\phi^0_{\beta,\theta}$ is the NN from Lemma \ref{key_prox_lemma0} with $M=1$. This clearly gives $\phi_{s,\theta}\in\mathcal{N}_{\textup{\textbf{D}},2,1}$, so we need to only bound the error. From Lemma \ref{key_prox_lemma0} we have
\begin{align*}
\left\|\mathrm{prox}_{F}(\hat x)-B\varphi_{(2s)^{-1}}^{\theta}( B^{-1}\hat x)\right\|_{l^2}&=\left\|B\left(\varphi_{(2s)^{-1}}(B^{-1}\hat x)- \varphi_{(2s)^{-1}}^{\theta}( B^{-1}\hat x)\right)\right\|_{l^2}\\
&\leq\theta\|w\|_{l^2}.
\end{align*}
The bound \ref{key_prox_lemma_EQ1} now follows.
\end{proof}

The following theorem proves that one can construct NNs with objective function bounds. The proof constructs approximations of unrolled iterations of Chambolle and Pock's primal-dual algorithm \cite{chambolle2011first, chambolle2016ergodic, he2012convergence}. We have used $b$ to denote part of the inputs of the NNs, instead of $y$, to avoid a clash of notation with the usual notation for primal-dual iterations ($y$ is used to denote a dual variable). The bounds in part 2 of Theorem \ref{BP_prop1} will be combined with results from \S \ref{compressed_results} to construct the families of NNs in Theorem \ref{key_CS_prop}.

\begin{theorem}
\label{BP_prop1}
Let $A\in\mathbb{Q}[i]^{m \times N}$ and $\theta\in\mathbb{Q}_{>0}$. Suppose also that $L_A\in\mathbb{Q}_{\geq 1}$ is an upper bound for $\|A\|$, and that $\tau,\sigma\in\mathbb{Q}_{>0}$ are such that $\tau\sigma L_A^2<1$. Let $\lambda\in\mathbb{Q}_{>0}$, $w\in\mathbb{Q}_{>0}^N$ and consider the resulting optimisation problem $(P_3)$. Then there exists an algorithm that constructs a sequence of neural networks $\left\{\phi^{A}_{n,\lambda},\theta\right\}$ with the following properties:
\begin{enumerate}
 \item (Size) Each $\phi^{A}_{n,\lambda}:\mathbb{C}^{m+N}\rightarrow\mathbb{C}^{N}$ takes as input data $b\in\mathbb{C}^m$ and an initial guess $x_0\in\mathbb{C}^{N}$, both of which are completely general. Also, $\phi^{A}_{n,\lambda}\in\mathcal{N}_{\textup{\textbf{D}}_n,3n+1,3}$ with
$$
\textup{\textbf{D}}_n=(m+N,\underbrace{2N+m,2(N+m),2N+m+1}_{\text{repeated $n$ times}},N).
$$
	\item ($\mathcal{O}(n^{-1}+n\theta)$ Error Control) Let
	\begin{equation}\label{eq:C_thm}
	C=\left(1+\|w\|_{l^2}+2\sigma\|A\|\|w\|_{l^2}\right)\sqrt{\frac{\tau +\sigma}{1-\tau\sigma L_A^2}}\sqrt{\frac{\tau+\sigma}{\tau\sigma}},
	\end{equation}
	then for any inputs $b\in\mathbb{C}^m$ and $x_0\in\mathbb{C}^N$, there exists a vector $\psi_n(b,x_0)\in\mathbb{C}^N$ with
		\begin{equation}
		\label{SQRT_LASSO_BD2}
		\left\|\psi_n(b,x_0)-\phi_{n,\lambda}^{A}(b,x_0)\right\|_{l^2}\leq n\theta C
		\end{equation}
such that for any $x\in\mathbb{C}^N$ and $\eta\in[0,1]$, it holds that
		\begin{equation}
		\label{SQRT_LASSO_BD1}
		\lambda \|\psi_n(b,x_0)\|_{l^1_w}-\lambda \|x\|_{l^1_w}+\eta\|A\psi_n(b,x_0)-b\|_{l^2}-\|Ax-b\|_{l^2}\leq \frac{1}{n}\left(\frac{\|x-x_0\|_{l^2}^2}{\tau}+\frac{\eta^2}{\sigma}\right).
		\end{equation}
\end{enumerate}
\end{theorem}

\begin{proof}
We will use the notation $b\in\mathbb{C}^m$ to denote an input vector for our NNs throughout the proof and reserve $y$ to denote dual vectors, consistent with the literature on primal-dual iterative optimisation algorithms for saddle point problems.

\textbf{Step 1:} The first step is to consider an equivalent optimisation problem over $\mathbb{R}$ instead of $\mathbb{C}$, and rewrite the problem as a saddle point problem. For $x\in\mathbb{C}^N$, let $x_1=\mathrm{real}(x)$ and $x_2=\mathrm{imag}(x)$ and consider $x=(x_1,x_2)^{\top}$ as a vector in $\mathbb{R}^{2N}$ (and likewise for the dual variables). With an abuse of notation, we will use the same notation for complex $x\in\mathbb{C}^N$ and the corresponding vector in $\mathbb{R}^{2N}$, though it will be clear from the context whether we refer to the complex vector or it's real counterpart. We let $c=(\mathrm{real}(b),\mathrm{imag}(b))^{\top}$.  Define the matrices
$$
K_1=\begin{pmatrix}
\mathrm{real}(A) & -\mathrm{imag}(A)\\
\mathrm{imag}(A) & \mathrm{real}(A)
\end{pmatrix}\in\mathbb{R}^{2m\times 2N},\quad K_2=\begin{pmatrix}
\mathrm{real}(B) & -\mathrm{imag}(B)\\
\mathrm{imag}(B) & \mathrm{real}(B)
\end{pmatrix}\in\mathbb{R}^{2N\times 2N},
$$
which simply correspond to multiplication by the matrices $A$ and $B\coloneqq \mathrm{diag}(w_1,...,w_N)$ respectively. Let $\widetilde F_1:\mathbb{R}^{2N}\rightarrow\mathbb{R}$ be defined by
$$
\widetilde F_1(x)=\sum_{j=1}^N\sqrt{ (K_2x)_{j}^2+(K_2x)_{j+N}^2}.
$$
and set $\widetilde F_3(x)=\lambda \widetilde F_1(x)$. Then the primal problem $(P_3)$ is equivalent to
$$
\min_{x\in\mathbb{R}^{2N}}\quad \widetilde F_3(x)+\|K_1x-c\|_{l^2}.
$$
It is also straightforward to check that $L_A$ is an upper bound for $\|K_1\|$. The saddle point formulation of the problem is given by
\begin{equation}
\label{saddle2}
\min_{x\in\mathbb{R}^{2N}}\max_{y\in\mathbb{R}^{2m}}\mathcal{L}(x,y)\coloneqq \langle K_1x,y\rangle +\widetilde F_3(x)-f_3^*(y),
\end{equation}
where $f_3^*(y)=\chi_{B_1(0)}(y)+\langle c,y\rangle$, and $\chi_S$ denotes the indicator function of a set $S$, taking the value $0$ on $S$ and $+\infty$ otherwise, and $B_1(0)$ denotes the $l^2$ closed unit ball.

\textbf{Step 2:} We will solve (\ref{saddle2}) by approximating Chambolle and Pock's primal-dual algorithm \cite{chambolle2011first} with a NN. As well as this, we will use an additional shift of updates (considered in \cite{chambolle2016ergodic,he2012convergence}) in comparison with the original paper \cite{chambolle2011first}. We will write the iteration as an instance of the proximal point algorithm \cite{rockafellar1976monotone} and gain a non-expansive map in a norm which we relate to the standard Euclidean norm.

We start by setting $x^0=x_0$ (one of the inputs of the NN) and $y^0=0$. Recall that for a convex function $h$, we have that $x = \prox_{h}(z)$ if and only if $z \in x+\partial h(x)$, where $\partial h$ denotes the subdifferential of $h$, see, for example, \cite[Prop. B.23]{foucart2013mathematical}. Letting $g=\widetilde F_3$ and $f^*=f_3^*$, the exact iterates can be written as
\begin{equation}
\label{eq:iterCP}
\begin{split}
x^{k+1}&=\argmin_{x \in \mathbb{R}^{2N}} g(x)+\langle K_1x,y^k\rangle +\frac{1}{2\tau}\|x-x^k\|_{l^2}^2\\
&=\argmin_{x \in \mathbb{R}^{2N}} g(x) +\frac{1}{2\tau}\|x-(x^k-\tau K_1^* y^k)\|_{l^2}^2\\
&=(I+\tau\partial g)^{-1}(x^k-\tau K_1^* y^k)\\
y^{k+1}&=\argmin_{y \in \mathbb{R}^{2m}} f^*(y)-\langle K_1(2x^{k+1}-x^k),y\rangle +\frac{1}{2\sigma}\|y-y^k\|_{l^2}^2\\
&=\argmin_{y \in \mathbb{R}^{2m}} f^*(y) +\frac{1}{2\sigma}\|y-(y^k+\sigma K_1 (2x^{k+1}-x^k)\|_{l^2}^2\\
&=(I+\sigma\partial f^*)^{-1}\left[y^k+\sigma K_1 (2x^{k+1}-x^k)\right].
\end{split}
\end{equation}
Note that the solutions of these proximal mappings are given by Lemmas \ref{key_prox_lemma0} and \ref{key_prox_lemma} and their proofs, as we describe explicitly below in step 4. The function $f^*$ also depends on the input data $b$.

Let $z=(x,y)^{\top}$ and define the matrix
$$
M_{\tau\sigma}=\begin{pmatrix}
\frac{1}{\tau}I & -K_1^*\\
-K_1 & \frac{1}{\sigma}I
\end{pmatrix}\in\mathbb{R}^{2(m+N)\times 2(m+N)},
$$
which is positive definite by the assumption $\tau\sigma L_A^2<1$ and hence induces a norm denoted by $\|\cdot\|_{\tau\sigma}$. We can write the iterations as (see, for example, \cite[Sec.\ 3]{chambolle2016ergodic})
$$
0\in M_{\tau\sigma}^{-1}\begin{pmatrix}
\partial g & K_1^* \\
-K_1 & \partial f^*
\end{pmatrix}z^{k+1}+(z^{k+1}-z^k)\Rightarrow z^{k+1}=\left[I+M_{\tau\sigma}^{-1}\begin{pmatrix}
\partial g & K_1^* \\
-K_1 & \partial f^*
\end{pmatrix}\right]^{-1}z^k.
$$
The multi-valued operator
$$
M_{\tau\sigma}^{-1}\begin{pmatrix}
\partial g & K_1^* \\
-K_1 & \partial f^*
\end{pmatrix}
$$
is maximal monotone with respect to the inner product induced by $M_{\tau\sigma}$ \cite{rockafellar1976monotone} and hence the iterates are non-expansive in the norm $\|\cdot\|_{\tau\sigma}$. We also have that
\begin{align*}
\|(x,y)^{\top}\|^2_{\tau\sigma}&\leq\frac{\|x\|_{l^2}^2}{\tau}+\frac{\|y\|_{l^2}^2}{\sigma}+2L_A\|x\|_{l^2}\|y\|_{l^2}\\
&\leq \left(\frac{L_A}{\nu}+\tau^{-1}\right)\|x\|_{l^2}^2+\left(L_A\nu+\sigma^{-1}\right)\|y\|_{l^2}^2,
\end{align*}
for any $\nu> 0$ by the generalised AM--GM inequality. Choosing $\nu=\sigma L_A$ and using $\tau\sigma L_A^2<1$, we have that
\begin{equation}
\label{equiv_norm1}
\|(x,y)^{\top}\|^2_{\tau\sigma}\leq(\tau^{-1}+\sigma^{-1})\|(x,y)^{\top}\|_{l^2}^2.
\end{equation}
A similar calculation yields that
\begin{equation}
\label{equiv_norm2}
\|(x,y)^{\top}\|_{l^2}^2\leq \frac{\tau+\sigma}{1-\tau\sigma L_A^2}\|(x,y)^{\top}\|^2_{\tau\sigma}.
\end{equation}
We shall need these inequalities in Step 5.

\textbf{Step 3:} Next, we use convergence guarantees proven in \cite{chambolle2016ergodic} to obtain inequalities that closely resemble (\ref{SQRT_LASSO_BD1}). Define the ergodic averages
$$
X^k=\frac{1}{k}\sum_{j=1}^k x^j,\quad Y^k=\frac{1}{k}\sum_{j=1}^ky^j.
$$
By convexity, the map from $(x^1,y^1)^{\top}$ to $(X^k,Y^k)^{\top}$ is also non-expansive in the norm $\|\cdot\|_{\tau\sigma}$. It also holds (see \cite{chambolle2016ergodic} Theorem 1 and remarks) that
\begin{equation}
\label{gap_conv}
\mathcal{L}(X^k,y)-\mathcal{L}(x,Y^k)\leq\frac{1}{k}\left(\frac{\|x-x_0\|^2_{l^2}}{\tau}+\frac{\|y\|^2_{l^2}}{\sigma}\right),
\end{equation}
for all $x\in\mathbb{R}^{2N}$ and all $y\in\mathbb{R}^{2m}$. Let $y$ be parallel to $KX^k-c$ such that $\|y\|_{l^2}=\eta\leq 1$, and $x$ be general in (\ref{gap_conv}). This gives
$$
\widetilde F_3(X^k)-\widetilde F_3(x)+\langle K_1X^k-c,y\rangle + \langle c-K_1x,Y^k\rangle\leq\frac{1}{k}\left(\frac{\|x-x_0\|_{l^2}^2}{\tau}+\frac{\eta^2}{\sigma}\right).
$$
Since $\|Y^k\|_{l^2}\leq 1$ (otherwise we gain a contradiction in that the left-hand side of (\ref{gap_conv}) is infinite), this implies
\begin{equation}
\label{BoundP3}
\widetilde F_3(X^k)-\widetilde F_3(x)+\eta\|K_1X^k-c\|_{l^2}-\|K_1x-c\|_{l^2}\leq\frac{1}{k}\left(\frac{\|x-x_0\|_{l^2}^2}{\tau}+\frac{\eta^2}{\sigma}\right).
\end{equation}

\textbf{Step 4:} The next step is to unroll the iterations in (\ref{eq:iterCP}) as (complex-valued) NNs that approximate the $X^k$. We unroll via the following steps:
\begin{equation*}
\begin{split}
\begin{pmatrix}
 X^k\\
x^k\\
y^k
\end{pmatrix}&\xrightarrow[]{\text{L}}\begin{pmatrix}
 X^k\\
x^k-\tau A^* y^k\\
y^k-\sigma Ax^k
\end{pmatrix}\xrightarrow[]{\text{NL}}\begin{pmatrix}
 X^k\\
x^{k+1}\\
y^k-\sigma Ax^k
\end{pmatrix}
\xrightarrow[]{\text{L}}\begin{pmatrix}
 X^{k+1}\\
x^{k+1}\\
u^k
\end{pmatrix}\xrightarrow[]{\text{NL}}\begin{pmatrix}
 X^{k+1}\\
x^{k+1}\\
y^{k+1}
\end{pmatrix},
\end{split}
\end{equation*}
with $u^k=y^k+\sigma A(2x^{k+1}-x^k)-\sigma b$. The first arrow is a simple linear map, the second computes
$$
x^{k+1}=\left(I+\tau \lambda\partial F_1^A\right)^{-1}(x^k-\tau A^* y^k)
$$
The third is an affine map and the final arrow applies $\psi^1$ to $u^k$. We now define the approximations $\tilde Z^k$ and $\tilde z^k$ (of $Z^k=(X^k,Y^k)^{\top}$ and $z^k=(x^k,y^k)^{\top}$ respectively) defined by replacing $\psi^1$ with $\phi_{\theta}^1$ (Lemma \ref{key_prox_lemma0}) and the computation of $(I+\tau\lambda\partial F_1^A)^{-1}(x^k-\tau A^* y^k)$ with $\phi_{(2\tau\lambda)^{-1},\theta}(\tilde x^k-\tau A^* \tilde y^k)$ (Lemma \ref{key_prox_lemma}). We initialise the network with $\tilde x^0=x_0$ and $y^0=0$. Since the composition of two affine maps is affine, it follows that the mapping from $(b,x_0)$ to $\tilde X^n$ can be realised by $\phi_{n,\lambda}^{A}\in\mathcal{N}_{\textup{\textbf{D}}_n,3n+1,3}$ with
\begin{equation*}
\textup{\textbf{D}}_n=(m+N,\underbrace{2N+m,2(N+m),2N+m+1}_{\text{repeated $n$ times}},N).
\end{equation*}
Clearly, the sequence of NNs are NNs in the sense of \S \ref{dfojon} and can be constructed by an algorithm (see \S \ref{computational_setup}).

\textbf{Step 5:} Finally, we bound the difference between $Z^k$ and $\tilde Z^k$ to deduce (\ref{SQRT_LASSO_BD2}), and the error bound in the objective function using the inequalities in Step 3. We write
$$
\tilde x^k=x^k+e_1^k,\quad \tilde y^k=y^k+e_2^k
$$
and clearly have that $e_1^0=0$ and $e_2^0=0$. We can write
$$
\tilde x^{k+1}=\phi_{(2\tau\lambda)^{-1}}(\tilde x^k-\tau A^* \tilde y^k)+e_{3}^{k+1},
$$
with $\|e_{3}^{k+1}\|_{l^2}\leq\theta\|w\|_{l^2}$ by Lemma \ref{key_prox_lemma}. We also have that
$$
\tilde y^{k+1}=\psi^1(\tilde y^k+\sigma A(2\tilde x^{k+1}-\tilde x^k)-\sigma b)+e_{4}^{k+1},
$$
with $\|e_{4}^{k+1}\|_{l^2}\leq\theta$ by Lemma \ref{key_prox_lemma0}. Since $\psi^1$ is non-expansive, it follows that
$$
\tilde y^{k+1}=\psi^1(\tilde y^k+\sigma A(2(\tilde x^{k+1}-e_{3}^{k+1})-\tilde x^k)-\sigma b)+e_{5}^{k+1},
$$
with $\|e_{5}^{k+1}\|_{l^2}\leq\theta(1+2\sigma \left\|A\right\|\|w\|_{l^2})$. We can then use the fact that the iterates applied with the exact proximal maps are non-expansive in the norm $\|\cdot\|_{\tau\sigma}$, along with \eqref{equiv_norm2} and \eqref{equiv_norm1}, to conclude that
\begin{equation*}
\begin{split}
\|X^{n}-\tilde X^n\|_{l^2}&\leq  \sqrt{\frac{\tau +\sigma}{1-\tau\sigma L_A^2}}\|Z^{n}-\tilde Z^{n}\|_{\tau \sigma}\\
&\leq \sqrt{\frac{\tau +\sigma}{1-\tau\sigma L_A^2}}\left[\|Z^{n-1}-\tilde Z^{n-1}\|_{\tau \sigma}+\theta\sqrt{\frac{\tau+\sigma}{\tau\sigma}}\Big(1+\|w\|_{l^2}+2\sigma\|A\|\|w\|_{l^2})\Big)\right]\\
&\leq n\theta(1+\|w\|_{l^2}+2\sigma\|A\|\|w\|_{l^2})\sqrt{\frac{\tau +\sigma}{1-\tau\sigma L_A^2}}\sqrt{\frac{\tau+\sigma}{\tau\sigma}}.
\end{split}
\end{equation*}
It follows that (\ref{SQRT_LASSO_BD2}) holds with $\psi_n(b,x_0)=X^n$ and then the complex version of (\ref{BoundP3}) implies (\ref{SQRT_LASSO_BD1}).
\end{proof}

\subsection{Proof of Theorem \ref{key_CS_prop}}
\label{nnoinav9bh}
\textbf{Step 1:} The first step is to derive a bound on the distance between vectors using the square-root LASSO objective function and rNSPL. First, note that for any inputs $A$ (more precisely, the rational approximations $\{A_l\}$), $\rho$ and $\gamma$ described in the theorem, we can compute, using Lemma \ref{rNSPL_perturb}, a positive integer $l$ in finitely many arithmetic operations and comparisons, such that $A_l \in \mathbb{Q}[i]^{m \times N}$ satisfies the rNSPL with constants $(1+\rho)/2\in(0,1)$, $2\gamma>0$. Lemmas \ref{CS_lem1} and \ref{CS_lem2} therefore imply that for any pair $z_1,z_2\in\mathbb{C}^N$ we have
\begin{equation}
\label{1bound}
\|z_1-z_2\|_{l^1_w}\leq \frac{3+\rho}{1-\rho}\left(2\sigma_{\textbf{s},\textbf{M}}(z_2)_{l^1_w}+\|z_1\|_{l^1_w}-\|z_2\|_{l^1_w}\right)+\frac{8\gamma\sqrt{\xi}}{1-\rho}\|A_l(z_1-z_2)\|_{l^2},
\end{equation}
and
\begin{equation}
\label{2bound}
\|z_1-z_2\|_{l^2}\leq \left(\frac{1+\rho}{2}+\frac{(3+\rho)\kappa^{1/4}}{4}\right)\frac{\|z_1-z_2\|_{l^1_w}}{\sqrt{\xi}}+\left(2+{\kappa^{1/4}}\right)\gamma\|A_l(z_1-z_2)\|_{l^2}.
\end{equation}
Combining these two inequalities, we obtain the bound
\begin{equation}
\label{3bound}
\begin{split}
\|z_1-z_2\|_{l^2}&\leq \frac{2C_1}{\sqrt{\xi}}\sigma_{\textbf{s},\textbf{M}}(z_2)_{l^1_w}+\frac{C_1}{\sqrt{\xi}}\left(\|z_1\|_{l^1_w}-\|z_2\|_{l^1_w}\right)+C_2\|A_l(z_1-z_2)\|_{l^2}\\
&\leq \frac{2C_1}{\sqrt{\xi}}\sigma_{\textbf{s},\textbf{M}}(z_2)_{l^1_w}+2C_2\|A_lz_2-y\|_{l^2}\\
&\quad+\frac{C_1}{\lambda\sqrt{\xi}}\left(\lambda \|z_1\|_{l^1_w}-\lambda \|z_2\|_{l^1_w}+\|A_lz_1-y\|_{l^2}-\|A_lz_2-y\|_{l^2}\right),
\end{split}
\end{equation}
where the second inequality follows from the fact that $\|A_l(z_1-z_2)\|_{l^2}\leq\|A_lz_1-y\|_{l^2}+\|A_lz_2-y\|_{l^2}$ and we chose a positive rational $\lambda\leq C_1/(C_2 \sqrt{\xi})$ (we will specify how small $|\lambda-C_1/(C_2 \sqrt{\xi})|$ must be later, and always assume $\lambda\sim C_1/(C_2 \sqrt{\xi})$). For notational convenience, we define
\begin{equation}
\label{G_def}
G(z_1,z_2,y)\coloneqq \lambda \|z_1\|_{l^1_w}-\lambda \|z_2\|_{l^1_w}+\|A_lz_1-y\|_{l^2}-\|A_lz_2-y\|_{l^2},
\end{equation}
which is the difference between the values of the objective function $F_3^A$ for arguments $z_1$ and $z_2$. We also define
\begin{equation}
\label{c_def}
c(z,y)\coloneqq \frac{2C_1}{C_2\sqrt{\xi}}\cdot\sigma_{\textup{\textbf{s}},\textup{\textbf{M}}}(z)_{l^1_w}+2\|A_lz-y\|_{l^2}.
\end{equation}
It follows from (\ref{3bound}) and $\lambda\leq C_1/(C_2 \sqrt{\xi})$ that
\begin{equation}
\label{exp_key1}
\|z_1-z_2\|_{l^2}\leq\frac{C_1}{\lambda\sqrt{\xi}}\left(c(z_2,y)+G(z_1,z_2,y)\right),
\end{equation}
which also implies the bound
\begin{equation}
\label{for_free}
G(z_1,z_2,y)\geq -c(z_2,y).
\end{equation}
Note that these bounds hold for completely general $z_1,z_2$ and $y$.

\textbf{Step 2:} The second step is to apply Theorem \ref{BP_prop1} using a suitable scaling to define a family of parametrised NNs, which we iterate later in the proof (this corresponds to restarting primal-dual iterations with different parameters). Let $\sigma=\tau \in (4\|A_l\|^{-1}/5, 5\|A_l\|^{-1}/6)$ be positive rational numbers, and note that we can compute such parameters by approximating $\|A_l\|$ via any standard algorithm that approximates the largest singular value of a rectangular matrix using finitely many arithmetic operations and comparisons. We now use Theorem \ref{BP_prop1} (with $\theta$ specified below) with input $y/(p \beta)$ and $x_0/(p\beta)$ for a given $p \in \mathbb{N}$, and $\beta\in\mathbb{Q}_{>0}$ (which we explicitly define below).
Given $\phi_{p,\lambda}^{A_l}(y/(p\beta), x_0/(p\beta))$, Theorem \ref{BP_prop1}  ensures the existence of a vector $\psi_p = \psi_{p}(y/(p\beta), x_0/(p\beta))$ satisfying
\[\left\| \psi_{p}\left( \tfrac{y}{p\beta}, \tfrac{x_0}{p\beta}\right) - \phi_{p,\lambda}^{A}
\left(\tfrac{y}{p\beta}, \tfrac{x_0}{p\beta}\right) \right\|_{l^2}  \leq p C \theta
\]
where $C$ is given in (\ref{eq:C_thm}) and
\begin{equation} \label{eq:scaled}
\lambda \| \psi_{p} \|_{l^{1}_{w}} -\lambda \left\| \frac{x}{p\beta} \right\|_{l^{1}_{w}} +\left\| A \psi_p - \frac{y}{p\beta}\right\|_{l^2} - \frac{1}{p\beta}\|Ax-y\|_{l^2}
\leq \frac{1}{p}\left( \frac{\|x(p\beta)^{-1}-x_0(p\beta)^{-1}\|_{l^2}^2}{\tau} + \frac{1}{\sigma}\right)
\end{equation}
for any $x \in \mathbb{C}^{N}$ (and we have taken $\eta = 1$ in (\ref{SQRT_LASSO_BD1})).
Define the map $H_p^\beta:\mathbb{C}^{m}\times\mathbb{C}^{N}\rightarrow \mathbb{C}^N$ by
$$
H_{p}^\beta(y,x_0)=p\beta\phi_{p,\lambda}^{A_l}\left(\frac{y}{p\beta},\frac{x_0}{p\beta}\right).
$$
The additional scaling factors can be incorporated so that $H_{p}^\beta\in \mathcal{N}_{\textup{\textbf{D}}_{p},3p+1,3}$. Rescaling \eqref{eq:scaled} yields the existence of a vector $\hat\psi_p(y,x_0)\in\mathbb{C}^N$ (where the $\hat\cdot$ denotes an appropriate rescaling by multiplying by $p\beta$) such that
\begin{equation}
\label{ProofC1}
G\left(\hat\psi_p(y,x_0),x,y\right)\leq \frac{5}{4}\left(\frac{\|A_l\|}{p^2\beta}{\|x-x_0\|_{l^2}^2}+\|A_l\|{\beta}\right),
\end{equation}
where we have used the fact that $\tau^{-1}=\sigma^{-1}\leq 5\|A_l\|/4$. Moreover, the constant $C$ in Theorem \ref{BP_prop1} is bounded by
\begin{equation} \label{eq:Cbound}
{C =} \left(1+\|w\|_{l^2}+2\sigma {\|A_l\|}\|w\|_{l^2}\right)\sqrt{\frac{\tau +\sigma}{1-\tau\sigma L_A^2}}\sqrt{\frac{\tau+\sigma}{\tau\sigma}}\leq \hat C_1(1+\|w\|_{l^2}),
\end{equation}
for a constant $\hat C_1$ that we can explicitly compute. Hence, upon rescaling (\ref{SQRT_LASSO_BD2}), we arrive at
$$
\left\|\hat\psi_p(y,x_0)-H_{p}^\beta(y,x_0)\right\|_{l^2}\leq p^2\theta \beta\hat C_1(1+\|w\|_{l^2}).
$$
Using H\"older's inequality, this also implies that
$$
\left\|\hat\psi_p(y,x_0)-H_{p}^\beta(y,x_0)\right\|_{l^1_w}\leq p^2\theta \beta\hat C_1(1+\|w\|_{l^2})\|w\|_{l^2}.
$$
It follows from the reverse triangle inequality that
\begin{equation}
\label{G_var}
G\left(H_{p}^\beta(y,x_0),x,y\right)\leq G\left(\hat\psi_p(y,x_0),x,y\right)+p^2\theta\beta \hat C_1(1+\|w\|_{l^2}) \left(\|A_l\|+\lambda\|w\|_{l^2}\right).
\end{equation}
Using this bound in (\ref{ProofC1}), and the fact that $\lambda \lesssim (\gamma\sqrt{\xi})^{-1}$, we can choose $\theta\in\mathbb{Q}_{>0}$ such that
$$
\theta^{-1}\lesssim {p^2(1+\|w\|_{l^2})}\max\left\{1,\frac{\lambda\|w\|_{l^2}}{\|A_l\|}\right\}\lesssim {p^2(1+\|w\|_{l^2})}\max\left\{1,\frac{\|w\|_{l^2}}{\|A\|\gamma \sqrt{\xi}}\right\},
$$
and, simultaneously,
$$
G\left(H_{p}^\beta(y,x_0),x,y\right)\leq \frac{4}{3}\left(\frac{\|A_l\|}{p^2\beta}{\|x-x_0\|_{l^2}^2}+\|A_l\|{\beta}\right).
$$
Combining this with (\ref{exp_key1}), we obtain the key inequality
\begin{equation}
\label{ProofC2}
G\left(H_{p}^\beta(y,x_0),x,y\right)\leq \frac{4C_1^2\|A_l\|}{3p^2\beta\lambda^2\xi}\left[c(x,y)+G(x_0,x,y)\right]^2+\frac{4}{3}\|A_l\|{\beta}.
\end{equation}

\textbf{Step 3:} In this step, we specify the choice of $p$ and $\beta$. So far, we have not used any information regarding the vectors $x$ and $y$. Recall that for our recovery theorem, we restricted to pairs $(x,y)$ such that
$$
\frac{2C_1}{C_2\sqrt{\xi}}\cdot\sigma_{\textbf{s},\textbf{M}}(x)_{l^1_w}+2\|Ax-y\|_{l^2}\leq\delta,\quad \|x\|_{l^2}\leq b_1,\quad \|y\|_{l^2}\leq b_2.
$$
Using this, we can choose $l$ larger if necessary such that for any such $(x,y)$, we have the bound
$$
c(x,y)\leq \frac{2C_1}{C_2\sqrt{\xi}}\cdot\sigma_{\textbf{s},\textbf{M}}(x)_{l^1_w}+2\|Ax-y\|_{l^2}+2\|A-A_l\|\|x\|_{l^2}\leq 2\delta.
$$
The following lemma shows how to choose $\beta$ and $p$ to gain a decrease in $G$ by a factor of $\upsilon\in(0,1)$, up to small controllable error terms.

\begin{lemma}
\label{inbijn}
Let $\upsilon\in(0,1)\cap\mathbb{Q}_{>0}$, $\epsilon_0\in\mathbb{Q}_{>0}$ and choose $\beta\in\mathbb{Q}_{>0}$ such that $8\|A_l\|{\beta}=3\upsilon_0\upsilon(\epsilon_0+2\delta)$ for some $\upsilon_0\in[1,2)$. Then for any $x_0$ with $G(x_0,x,y)\leq \epsilon_0$ and positive integer $p\geq\left\lceil\frac{8C_1\|A_{l}\|}{3\upsilon\lambda\sqrt{\xi}\sqrt{(2-\upsilon_0)\upsilon_0}}\right\rceil$ the following bound holds
\begin{equation}
\label{exp_6363}
G\left(H_{p}^\beta(y,x_0),x,y\right)\leq \upsilon\left(2\delta+\epsilon_0\right).
\end{equation}
\end{lemma}

\begin{proof
The choice of $\beta$ ensures that
$$
\frac{4}{3}\|A_l\|{\beta}\leq\frac{\upsilon_0\upsilon}{2}(2\delta+\epsilon_0).
$$
Using (\ref{ProofC2}), and the fact that $0\leq c(x,y)+G(x_0,x,y)\leq 2\delta+\epsilon_0$, the bound (\ref{exp_6363}) therefore holds if
$$
\frac{32C_1^2\|A_l\|^2}{9p^2\upsilon_0\upsilon\lambda^2\xi}\left(2\delta+\epsilon_0\right)\leq \frac{(2-\upsilon_0)\upsilon}{2}(2\delta+\epsilon_0)
$$
rearranging and taking the square root now gives the result, where we use the ceiling function to ensure $p$ is an integer.
\end{proof}

We denote the choice of $\beta$ in Lemma \ref{inbijn} by $\beta(\upsilon,\epsilon_0)$. Since $8/3<3$, we can, by taking $l$ larger and by making $\lambda$ closer to $C_1/(C_2 \sqrt{\xi})$ if necessary, and through an appropriate choice of $\upsilon_0$, ensure that we can compute (using finitely many arithmetic operations and comparisons) a choice
$$
p(\upsilon)\leq \left\lceil\frac{3C_2\|A\|}{\upsilon}\right\rceil.
$$
such that the conclusion of the lemma holds.

\textbf{Step 4:} We are now ready to construct our NNs. Note first that
$$
G(0,x,y)\leq \|y\|_{l^2}\leq b_2,
$$
for any $y$ in our desired input. Given $n\in\mathbb{N}$, we set $\epsilon_0=b_2$ and for $j=2,...,n$ set
$$
\epsilon_j=\upsilon\left(2\delta+\epsilon_{j-1}\right).
$$
By summing a geometric series, this implies
$$
\epsilon_n\leq \frac{2\upsilon\delta}{1-\upsilon}+\upsilon^nb_2.
$$
We define $\phi_n(y)$ iteratively as follows. We set
$$
\phi_1(y)=H_{p(\upsilon)}^{\beta(\upsilon,\epsilon_0)}(y,0)
$$
and for $j=2,...,n$ we set
$$
\phi_{j}(y)=H_{p(\upsilon)}^{\beta(\upsilon,\epsilon_{j-1})}(y,\phi_{j-1}(y))
$$
Clearly this algorithmically constructs a NN $\phi_n$. We can concatenate (by combining affine maps) the NNs corresponding to the $H_p^\beta$ maps to see that $\phi_{n}\in\mathcal{N}_{\textup{\textbf{D}}_{(n,p)},3np+1,3}$. Moreover, Lemma \ref{inbijn} implies the bound
\begin{equation}
\label{nearly_there}
G\left(\phi_{n}(y),x,y\right)\leq\epsilon_n\leq \frac{2\upsilon\delta}{1-\upsilon}+\upsilon^nb_2.
\end{equation}
Combining this with (\ref{3bound}), we obtain
\begin{equation}
\label{nearly_there2}
\|\phi_{n}(y)-x\|_{l^2}\leq\frac{2C_1}{\sqrt{\xi}}\sigma_{\textbf{s},\textbf{M}}(x)_{l^1_w}+2C_2\|Ax-y\|_{l^2}+2C_2\|A-A_l\|_{l^2}b_1+\frac{C_1}{\lambda\sqrt{\xi}}\left(\frac{2\upsilon\delta}{1-\upsilon}+\upsilon^nb_2\right),
\end{equation}
Again, we can apriori choose $l$ and $\lambda$ to ensure that
$$
2C_2\|A-A_l\|_{l^2}b_1+\frac{C_1}{\lambda\sqrt{\xi}}\left(\frac{2\upsilon\delta}{1-\upsilon}+\upsilon^nb_2\right)\leq C_2\left(\frac{2\upsilon\delta}{1-\upsilon}+\delta+\upsilon^nb_2\right).
$$
Applying this bound to (\ref{nearly_there2}) yields (\ref{best1}).

Finally, we argue for the error in the weighted $l^1_w$-norm. Note that since $\rho< 1$, the choice of $\lambda$ ensures that
$$
\frac{8\gamma\sqrt{\xi}}{1-\rho}< \frac{3+\rho}{1-\rho}\frac{1}{\lambda}.
$$
It follows from (\ref{1bound}), using the same argument for the $l^2$ case, that
\begin{equation}
\label{hihuini}
\|\phi_{n}(y)-x\|_{l^1_w}\leq \frac{3+\rho}{1-\rho}\left(2\sigma_{\textbf{s},\textbf{M}}(x)_{l^1_w}+\frac{2}{\lambda}\|A_lx-y\|_{l^2}+\frac{1}{\lambda}G\left(\phi_{n}(y),x,y\right)\right),
\end{equation}
Again, we can apriori adjust $l$ and $\lambda$ as necessary to obtain the bound
$$
\|\phi_{n}(y)-x\|_{l^1_w}\leq \frac{3+\rho}{1-\rho}\left(2\sigma_{\textbf{s},\textbf{M}}(x)_{l^1_w}+\frac{2C_2\sqrt{\xi}}{C_1}\|Ax-y\|_{l^2}+\frac{C_2\sqrt{\xi}}{C_1}\left(\frac{2\upsilon\delta}{1-\upsilon}+\upsilon^nb_2\right)+\delta\frac{C_2\sqrt{\xi}}{C_1}\right),
$$
where the final term in brackets corresponds to this final approximation. Simplifying this expression yields (\ref{best2}) and completes the proof of the theorem.\qed

To end this section, we provide a brief proof sketch of the bounds in Remark \ref{no_restart}. The argument is similar to the proof of Theorem \ref{key_CS_prop}. We set
$$
\widehat\phi_{n}(y,x_0)=\beta\phi_{n,\lambda}^{A_l}\left(\frac{y}{\beta},\frac{x_0}{\beta}\right),
$$
and the arguments in Theorem \ref{key_CS_prop} show that we can choose $\tau,\sigma,l$ and $\theta_n$ with $\theta_n^{-1}=\mathcal{O}(n^2)$ such that for any $x,x_0\in\mathbb{C}^N$ and $y\in\mathbb{C}^m$,
$$
G\left(\widehat\phi_{n}(y,x_0),x,y\right)\leq \frac{3}{2}\frac{\|A\|}{n}\left(\frac{\|x-x_0\|_{l^2}^2}{\beta}+\beta\right).
$$
If $\|x\|_{l^2}\leq b_1$, then we can choose $l$ such that $2\|A-A_l\|b_2\leq \|A\|\beta/(2n)\min\{C_1/(C_2\lambda\sqrt{\xi}),1\}$ and hence (\ref{old_BOUND1}) follows from (\ref{3bound}). Similarly, we can use the corresponding bound (\ref{hihuini}) to show (\ref{old_BOUND2}).

\section{Proof of Theorem \ref{NN_FW_recov_thm}}
\label{CS_proofs_F}

For the benefit of the reader, we first recall the definition of the orthonormal bases used. We then provide coherence estimates which are used to obtain bounds on the number of samples needed, and end this section with the proof of Theorem \ref{NN_FW_recov_thm}. It will be convenient to sometimes enumerate the vector or tensor elements starting from 0, or negative numbers. That is for $x \in \mathbb{C}^{N}$ with $d=1$ we might denote its elements as $x = (x(0),\ldots, x(N-1))$, or $x = (x(-N/2+1), \ldots, x(N/2))$ and for $d>1$ its $\boldsymbol{k}=(k_1, \ldots, k_d)$'th element is written as $x(\boldsymbol{k})$. It will always be clear from the context, which range of indices we consider. Furthermore, recall from \S \ref{examples_in_CS} that we let $N = K^d$ and $K = 2^r$ for $r \in \mathbb{Z}_{\geq 0}$. This is assumed throughout this section.

\subsection{Setup: the relevant orthonormal bases}
\label{ON_bases_def_sec}

\subsubsection*{Discrete Fourier transform}

\noindent{}For a $d$-dimensional signal $x=\{x(\boldsymbol{t})\}_{t_1,\ldots, t_{d}=0}^{K-1}\in\mathbb{C}^{K\times \cdots\times K}$ we denote its Fourier transform by
$$
[\mathcal{F}x](\boldsymbol{\omega})=\frac{1}{N^{1/2}}\sum_{t_1,\ldots, t_d=0}^{K-1}x(\boldsymbol{t})\exp\left(\frac{2\pi i\boldsymbol{\omega} \cdot \boldsymbol{t}}{K}\right),\quad \boldsymbol{\omega}\in\mathbb{R}^d.
$$
For discrete computations, it is customary to consider this transform at the integers
$\boldsymbol{\omega} \in \{-K/2+1, \ldots, K/2\}^{d}$ and let $F^{(d)} \in \mathbb{C}^{K^d \times K^d}$
denote the corresponding matrix so that
\[ F^{(d)} \textnormal{vec}(x) = \left\{[\mathcal{F}x] (\boldsymbol{\omega})\right\}_{\boldsymbol{\omega} \in \{-K/2+1, \ldots, K/2\}^{d}}\]
for a suitable vectorisation $\textnormal{vec}(x)$ of $x$ and ordering of the $\boldsymbol{\omega}$'s.
Let
$$
\vartheta_{\boldsymbol{\omega}} = \left\{ N^{-1/2} \exp\left( -2\pi i K^{-1}\boldsymbol{\omega}\cdot\boldsymbol{t} \right): \boldsymbol{t} \in \{0, \ldots, K-1\}^d \right\} \subset \mathbb{C}^{K \times \cdots\times K}.
$$
Then it is well known that
\begin{equation}
\label{fourier_basis}
\left\{ \textnormal{vec} (\vartheta_{\boldsymbol{\omega}}) : \boldsymbol{\omega} \in \{-K/2+1, \ldots, K/2\}^d \right\}
\end{equation}
is an orthonormal basis for $\mathbb{C}^{K^d}=\mathbb{C}^N$. Furthermore, recall from \S \ref{examples_in_CS}, that we divide the different frequencies into dyadic bands. For $d=1$ we let $B_1 = \{0, 1\}$ and
$$
B_k=\left\{-2^{k-1}+1,...,-2^{k-2}\right\}\cup\left\{2^{k-2}+1,...,2^{k-1}\right\},\quad k=2,...,r.
$$
In the general $d$-dimensional case we set
$$
B_\textbf{k}^{(d)}=B_{k_1}\times...\times B_{k_d},\quad \textbf{k}=(k_1,...,k_d)\in\mathbb{N}^d.
$$

\subsubsection*{Walsh transform}

\noindent{}\begin{definition}[see, for example, \cite{Beauchamp75}]
The (sequentially-ordered) Walsh functions {$v_n\colon [0,1) \to \{+1,-1\}$} are defined by
\begin{equation}
\label{Walsh}
v_{\omega}(z)=(-1)^{\sum_{j=1}^\infty (\omega^{(j)}+\omega^{(j+1)})z^{(j)}},\quad z\in [0,1),\quad \omega \in\mathbb{Z}_{\geq 0},
\end{equation}
where $(z^{(i)})_{i\in\mathbb{N}}$ denotes the binary expansion of $z$ (terminating if $z$ is a dyadic rational) and we write $\omega=\sum_{j=1}^\infty \omega^{(j)}2^{j-1}$ for $\omega^{(j)} \in \{0,1\}$. For $\boldsymbol{z} \in [0,1)^d$ and
$\boldsymbol{\omega} \in \mathbb{Z}_{\geq 0}^{d}$, we let
\[v_{\boldsymbol{\omega}}(\boldsymbol{z}) = v_{\omega_1}(z_1) \cdots v_{\omega_d}(z_d) \]
be the tensor product of these functions.
\end{definition}

For $x \in \mathbb{C}^{K \times \cdots \times K}$ {and $K=2^r$} we let its $d$-dimensional Walsh transform be denoted by
$$
[\mathcal{W}x] (\boldsymbol{\omega}) = \frac{1}{N^{1/2}} \sum_{t_1,\ldots, t_d=0}^{K-1} x(\boldsymbol{t}) v_{\boldsymbol{\omega}}(\boldsymbol{t}/K), \quad \boldsymbol{\omega} \in \{0, \ldots, 2^{r}-1 \}^d.
$$
As in the Fourier case, we let $W^{(d)} \in \mathbb{C}^{N \times N}$ denote the corresponding matrix,
so that
$$
W^{(d)} \textnormal{vec} (x) = \left\{ [\mathcal{W}x](\boldsymbol{\omega})\right\}_{\boldsymbol{\omega} \in \{0, \ldots, K -1\}^d}
$$
for a suitable vectorisation of $x$ and ordering of the $\boldsymbol{\omega}$'s. We let
$$
\varrho_{\boldsymbol{\omega}} = \left\{ N^{-1/2}v_{\boldsymbol{\omega}}(\boldsymbol{t}/K) : \boldsymbol{t} \in \{0,
\ldots, K-1\}^d \right\} {\subset} \mathbb{C}^{K \times \cdots\times K}
$$
and note that also
\begin{equation}
\label{walsh_basis}
\left\{ \textnormal{vec}(\varrho_{\boldsymbol{\omega}}) : \boldsymbol{\omega} \in \{0, \ldots, K-1\}^d  \right\}
\end{equation}
is an orthonormal basis for $\mathbb{C}^{N}$. As in the Fourier case we {recall the definition of the} frequency bands we introduced in {\S \ref{examples_in_CS}.}  We let $B_1 = \{0,1\}$ and $B_{k} = \{2^{k-1}, \ldots, 2^{k}-1\}$ for $k=2,\ldots, r$ in the one-dimensional case, and
$$
B_\textbf{k}^{(d)}=B_{k_1}\times...\times B_{k_d},\quad \textbf{k}=(k_1,...,k_d)\in\mathbb{N}^d.
$$

Whether we mean the Walsh or Fourier frequency bands will always be clear from the context.

\subsubsection*{Haar-wavelet transform}

\noindent{}On $\mathbb{C}^{K}$ the Haar wavelet vectors are defined as
\begin{equation*}
\psi_{j,p}(i)=\begin{dcases} 2^{\frac{j-r}{2}},\quad &p2^{r-j}\leq i<\left(p+\frac{1}{2}\right)2^{r-j}\\
-2^{\frac{j-r}{2}},\quad &\left(p+\frac{1}{2}\right)2^{r-j}\leq i<(p+1)2^{r-j}\\
0,\quad &\text{otherwise}.
\end{dcases}
\end{equation*}
For $j=0,...,r-1$ and $p=0,...,2^j-1$, and we can define the corresponding scaling vectors as $\varphi_{j,p}(i)=|\psi_{j,p}(i)|$. To simplify the notation we set
$$
\psi_{j,k}^{(0)}=\varphi_{j,k},\quad \psi_{j,k}^{(1)}=\psi_{j,k}
$$
For $d > 1$ and $\boldsymbol{q}=(q_1,...,q_d)\in\{0,1\}^d$, $\boldsymbol{p}{=(p_1,\ldots, p_{d})}\in\mathbb{Z}_{\geq 0}^d$ define the tensor product
$$
\psi_{j,\boldsymbol{p}}^{\boldsymbol{q}}=\psi_{j,p_1}^{(q_1)}\otimes...\otimes \psi_{j,p_d}^{(q_d)}.
$$
Splitting these tensors by scale
\begin{align*}
C_1&=\{\textnormal{vec} (\psi_{0,0}^{\boldsymbol{q}}):\boldsymbol{q}\in\{0,1\}^d\},\\
C_j&=\{\textnormal{vec} (\psi_{j-1,\boldsymbol{p}}^{\boldsymbol{q}}):\boldsymbol{q}\in\{0,1\}^d\backslash\{0\},p_k=0,...,2^{j-1}-1\},
\end{align*}
for $j=2,...,r$, we get that $C_1 \cup \cdots \cup C_{r}$ is an orthonormal
basis for $\mathbb{C}^{N}$. {Next, let the vectors in $C_1 \cup \cdots \cup C_{r}$, form the rows of
a matrix $\Phi \in \mathbb{C}^{N \times N}$. The matrix $\Psi$ is called the
\emph{discrete wavelet transform} (DWT) matrix, and its inverse $\Psi^{-1}$ is
called the \emph{inverse discrete wavelet transform} (IDWT) matrix. Notice that since $C_1 \cup \cdots \cup C_{r}$ is an orthonormal basis, we have the relation
$\Psi^{-1} = \Psi^*$.}

\subsection{{Uniform recovery guarantees and coherence estimates}}

We express $U$ in block form as
$$
U = \left[ U^{(\textbf{k}, j)} \right]_{\textbf{k}=1, j=1}^{\|\textbf{k}\|_{l^\infty} \leq r, r},
$$
where the entries in each $U^{\textbf{k}, j}$ consist of the inner products $\langle \varphi,\rho_{\boldsymbol{\omega}}\rangle$ for $\varphi \in C_{j}$ and {where $\rho_{\boldsymbol{\omega}}$ is an element in either \eqref{fourier_basis} or \eqref{walsh_basis} with $\boldsymbol{\omega} \in B_{\textbf{k}}^{(d)}$, depending on whether we consider Fourier or Walsh sampling}. For this decomposition we define local coherence as follows.

\begin{definition}
\label{def_local_coherence_def}
Let $U = \left[ U^{(\text{\upshape{\textbf{k}}}, j)} \right]_{\text{\upshape{\textbf{k}}}=1, j=1}^{\|\text{\upshape{\textbf{k}}}\|_{l^\infty} \leq r, r}$ be
defined as above. Then the $(\text{\upshape{\textbf{k}}},j)$th local coherence of $U$ is
\[ \mu(U^{\text{\upshape{\textbf{k}}},j}) = \left|B_{\text{\upshape{\textbf{k}}}}^{(d)}\right| \max_{p,q}~ |(U^{\text{\upshape{\textbf{k}}},j})_{pq}|^2 \]
where $\left|B_{\text{\upshape{\textbf{k}}}}^{(d)}\right|$ denotes the cardinality of the set $B_{\text{\upshape{\textbf{k}}}}^{(d)}$.
\end{definition}

Recall from Definition \ref{levels}, that for an $(\mathbf{s},\mathbf{M})$-sparse vector, we let $s = s_1+\ldots+s_r$ denote the total sparsity. Furthermore, (as always) we let $m=\sum_{\mathbf{k}=1}^{\|\mathbf{k}\|_{l^{\infty}} \leq r} m_{\mathbf{k}}$ denote the total number of samples in an $(\mathbf{N}, \mathbf{m})$-multilevel subsampling scheme. The following result appeared in \cite{adcock2019uniform}, and shows that in order to prove Theorem \ref{NN_FW_recov_thm} via Theorem \ref{key_CS_prop}, we need to bound the local coherences of $U$ (we have stated the general result for a scalarised indexing of the multilevel subsampling scheme with $l$ levels).

\begin{proposition}[\cite{adcock2019uniform}]
\label{needed_G_pp}
Let $\epsilon_{\mathbb{P}}\in(0,1)$, $(\textup{\textbf{s}},\textup{\textbf{M}})$ be local sparsities and sparsity levels respectively with $2\leq s\leq N$, and consider the $(\mathbf{N}, \mathbf{m})$-multilevel subsampling scheme to form a subsampled unitary matrix $A$ as in Definitions \ref{msrsUUU} and \ref{msUnitary}. Let
\begin{equation}
\label{t_k_reshape}
t_j=\min\left\{\left\lceil{\frac{\xi(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}},w)}{w_{(j)}^2}}\right\rceil,M_{j}-M_{j-1}\right\},\quad j=1,...,r,
\end{equation}
and suppose that
\begin{equation}
\label{m_G_bdjj}
m_k\gtrsim \mathcal{L}'\cdot\sum_{j=1}^rt_j\mu(U^{k,j}),\quad k=1,...,l
\end{equation}
where
$$
\mathcal{L}' = r \cdot \log(2m) \cdot \log^2(t) \cdot \log(N) + \log(\epsilon_{\mathbb{P}}^{-1}).
$$
Then with probability at least $1-\epsilon_{\mathbb{P}}$, $A$ satisfies the weighted rNSPL of order $(\textup{\textbf{s}},\textup{\textbf{M}})$ with constants $\rho=1/2$ and $\gamma=\sqrt{2}$.
\end{proposition}

The following lemmas bound the local coherences of $U$, with $\mathcal{M}_{\mathcal{F}}(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{k}}})$ and $\mathcal{M}_{\mathcal{W}}(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{k}}})$ defined in (\ref{M1F}) and (\ref{M1W}) respectively.

\begin{lemma}[\textbf{Coherence bound for Fourier case}]
\label{coh_lem1}
Consider the $d$-dimensional Fourier--Haar--wavelet matrix with blocks $U^{\text{\upshape{\textbf{k}}},j}$, then the local coherences satisfy
\begin{equation}
\label{FT_coherence}
\mu(U^{\text{\upshape{\textbf{k}}},j})\lesssim 2^{-2(j-\|\textbf{k}\|_{l^\infty})_+}\prod_{i=1}^d 2^{-\left|k_i-j\right|},
\end{equation}
where for $t\in\mathbb{R}$, $t_+=\max\{0,t\}$. It follows that
\begin{equation}
\label{FT_coherence2}
\sum_{j=1}^rs_j\mu(U^{\text{\upshape{\textbf{k}}},j})\lesssim \sum_{j=1}^{\|\text{\upshape{\textbf{k}}}\|_{l^\infty}}s_j\prod_{i=1}^d 2^{-\left|k_i-j\right|}+\sum_{j=\|\text{\upshape{\textbf{k}}}\|_{l^\infty}+1}^{r}s_j2^{-2(j-\|\text{\upshape{\textbf{k}}}\|_{l^\infty})}\prod_{i=1}^d 2^{-\left|k_i-j\right|}=\mathcal{M}_{\mathcal{F}}(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{k}}}).
\end{equation}
\end{lemma}

\begin{proof}
From the one-dimensional case treated in \cite[{See proof of Lem.\ 1}]{adcock2016note}, we have
\[
\left|\left[\mathcal{F}\psi_{j,p}^{(1)}\right](\omega)\right|^2\lesssim \begin{dcases}
2^{-k}2^{-\left|k-j\right|},\quad&\text{if }j\leq k,\\
2^{-k}2^{-3\left|k-j\right|},\quad&\text{otherwise}
\end{dcases},
\]
We proceed by showing that $\left|\left[\mathcal{F}\psi_{j,p}^{(0)}\right](\omega)\right|^2\lesssim
2^{-k}2^{-|k-j|}$ in the one-dimensional case, before we consider the $d$-dimensional case.

Let $\omega\neq 0$ correspond to a frequency in $B_{k}$, $j\in\{0,...,r-1\}$ and $p\in\{0,...,2^{j}-1\}$. Then
\begin{equation*}
\begin{split}
\left[\mathcal{F}\psi^{(0)}_{j,p}\right](\omega)&=2^{\frac{j}{2}-r}e^{2\pi i \omega p2^{-{j}}}\sum_{t=0}^{2^{r-j}-1}e^{2\pi i \omega t 2^{-r}}\\
&=2^{\frac{j}{2}-r}e^{2^{1-j}\pi i \omega p}\frac{1-e^{2\pi i \omega 2^{-j}}}{1-e^{2\pi i \omega 2^{-r}}}.
\end{split}
\end{equation*}
A simple application of the double angle formula then yields
\[
\left|\left[\mathcal{F}\psi^{(0)}_{j,p}\right](\omega)\right|\lesssim2^{\frac{j}{2}-r}\frac{\left|\sin(\pi \omega 2^{-j})\right|}{\left|\sin(\pi \omega 2^{-r})\right|}=\frac{2^{\frac{j}{2}}}{\left|\omega\right|}\frac{\left|\omega 2^{-r}\right|}{\left|\sin(\pi \omega 2^{-r})\right|}\left|\sin(\pi \omega 2^{-j})\right|\lesssim2^{\frac{j}{2}-k}\left|\sin(\pi \omega 2^{-j})\right|,
\]
where the second inequality follows from the fact that $|\omega 2^{-r}|\leq1/2$ and $2^{k}\lesssim\left|\omega\right|$. If $k>j$ then this implies that $\left|\left[\mathcal{F}\psi^{(0)}_{j,p}\right](\omega)\right|^2\lesssim 2^{-k}2^{-|k-j|}$. If $k\leq j$, we use that $|\sin (\pi t)| \leq \pi |t|, ~\forall t \in \mathbb{R}$
to get
\[
\left|\sin(\pi \omega 2^{-j})\right|\lesssim 2^{k-j}.
\]
This implies
\[
\left|\left[\mathcal{F}\psi^{(0)}_{j,p}\right](\omega)\right|^2\lesssim 2^{j-2k}2^{2k-2j}=2^{-j}=2^{-k}2^{-|k-j|}.
\]
If $\omega=0$ then directly from the definition we have $\left|\left[\mathcal{F}\psi^{(0)}_{j,p}\right](\omega)\right|^2\lesssim
2^{-j}=2^{-k}2^{-|k-j|}$ and hence this bound still holds.

We now consider the general $d$-dimensional case. The above computations give that
\begin{equation*}
\mu(U^{(\textbf{k},1)})\lesssim2^{\sum_{i=1}^{d}k_i}\max_{\boldsymbol{q}\in\{0,1\}^d} \prod_{i=1}^d \max_{w\in B_{k_i}}\left|\left[\mathcal{F}\psi^{(q_i)}_{0,0}\right](\omega)\right|^2\lesssim \prod_{i=1}^d 2^{-\left|k_i-1\right|}.
\end{equation*}
Similarly for $j>1$
\begin{equation*}
\begin{split}
\mu(U^{(\textbf{k},j)})&\lesssim2^{\sum_{i=1}^{d} k_i}\max_{\boldsymbol{q}\in\{0,1\}^d\backslash\{0\}} \prod_{i=1}^d \max_{w\in B_{k_i}}\max_{p_i\in\{0,...,2^{j-1}-1\}}\left|\left[\mathcal{F}\psi^{(q_i)}_{j-1,p_i}\right](\omega)\right|^2\\
&\lesssim \max_{\boldsymbol{q}\in\{0,1\}^d\backslash\{0\}} \prod_{i=1}^d 2^{-\left|k_i-j\right|-2q_i(j-k_i)_+}.
\end{split}
\end{equation*}
The maximum value of this estimate is obtained when the non-zero component of
$\boldsymbol{q}$ corresponds to the maximum value of $k_i$. This gives precisely
(\ref{FT_coherence}).
\end{proof}

Before we proceed with the Walsh--Haar--wavelet case, we note the following lemma, which can be found in, for example, \cite{Antun16}. {For an in-depth treatment of different types of Walsh--Haar systems we refer the reader to \cite{walsh-haar-system}.}
\begin{lemma}\label{lem:wal_const}
    Let $\omega$ and $j\geq 0$ be integers so that $2^{j} \leq \omega < 2^{j+1}$ and let
\[ \Delta_{k}^{j} = [k2^{-j} , (k+1)2^{-j} ), \quad k {\in \mathbb{Z}_{\geq 0}}. \]
Then $v_{\omega}$ is constant on each of the intervals
$\Delta_{k}^{j+1}$, $k \in \{ 0,\ldots, 2^{j+1}-1 \}$. Each of the intervals
$\Delta_{k}^{j}$ can be decomposed into the intervals $\Delta_{2k}^{j+1}$ and
$\Delta_{2k+1}^{j+1}$, where $v_{\omega}$ is equal to 1 on exactly one of them and
equal to $-1$ on the other. {When $\omega = 0$, we have $v_{\omega} \equiv 1$.}
\end{lemma}

\begin{lemma}[\textbf{Coherence bound for Walsh case}]
\label{coh_lem2}
Consider the $d$-dimensional Walsh--Haar--wavelet matrix with blocks $U^{(\text{\upshape{\textbf{k}}},j)}$, then the local coherences satisfy
\begin{equation}
\label{Walsh_coherence}
\mu(U^{(\text{\upshape{\textbf{k}}},j)})\lesssim \begin{dcases} \prod_{i=1}^d 2^{-\left|k_i-j\right|} &\quad \text{if $k_i\leq j$ for $i=1,...,d$ with at least one equality,}\\
0&\quad\text{otherwise}
\end{dcases}.
\end{equation}
It follows that
\begin{equation}
\label{Walsh_coherence2}
\sum_{j=1}^rs_j\mu(U^{(\text{\upshape{\textbf{k}}},j)})\lesssim s_{\|\text{\upshape{\textbf{k}}}\|_{l^\infty}}\prod_{i=1}^d 2^{-\left|k_i-\|\text{\upshape{\textbf{k}}}\|_{l^\infty}\right|}=\mathcal{M}_{\mathcal{W}}(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{k}}}).
\end{equation}
\end{lemma}

\begin{proof}
We begin with some computations in the one-dimensional case. Let $I_{j,p} = \{
p2^{r-j}, \ldots, (p+1)2^{r-j}-1 \}$. We  recall that
that $\text{supp} (\psi^{(0)}_{j,p}) = \text{supp} (\psi^{(1)}_{j,p}) = I_{j,p}$.
Using Lemma \ref{lem:wal_const} it is clear that for $2^{m} \leq \omega < 2^{m+1}$,
$\varrho_{\omega}$ is constant on $I_{m+1, k}$, for
$k \in \{0, \ldots, 2^{m+1}-1\}$ and that for any pair
$I_{m+1,2t}$, $I_{m+1,2t+1}$, $\varrho_{\omega}$ changes sign. For $\omega =0$,
we have that $\varrho_\omega$ is all constant.
Keeping track of the supports gives the relations
\[
 \left|\left\langle \psi_{j,p}^{(0)}, \varrho_{\omega}\right\rangle \right| =
\begin{cases}
    2^{-j/2} &  \text{if } \omega < 2^{j} \\
    0 & \text{otherwise}
\end{cases},
\quad\text{and}\quad
 \left|\left\langle \psi_{j,p}^{(1)}, \varrho_{\omega}\right\rangle \right|
=
\begin{cases}
    2^{-j/2} &  \text{if } 2^{j}\leq \omega < 2^{j+1} \\
    0 & \text{otherwise}
\end{cases}.
\]
We notice in particular that we can rewrite this as
$|\langle \psi_{j,p}^{(0)}, \varrho_{\omega}\rangle |= 2^{-j/2} = 2^{-k/2} 2^{-(j-k)/2}$ if $\omega \in B_{k}$, $k \leq j$
and $0$ otherwise, and that $|\langle \psi_{j,p}^{(1)}, \varrho_{\omega}\rangle |= 2^{-j/2}$
if $\omega \in B_{j+1}$ and $0$ otherwise.

Turning to the general $d$-dimensional case. The above computations immediately give that
\begin{equation*}
\mu(U^{(\textbf{k},1)})\lesssim \prod_{i=1}^d \delta_{k_i,1}
\end{equation*}
where $\delta_{i,j}$ is the Kronecker-delta. Similarly for $j>1$
\begin{equation*}
\begin{split}
\mu(U^{(\textbf{k},j)})&=\left|B_{{\textbf{k}}}^{(d)}\right|\max_{\boldsymbol{q}\in\{0,1\}^d\backslash\{0\}}
\prod_{i=1}^d \max_{\omega_i \in B_{k_i}}\max_{p_i\in\{0,...,2^{j-1}-1\}}\left|\langle \varrho_{\omega_i}, \psi^{(q_i)}_{j-1,p_i} \rangle\right|^2\\
&\lesssim 2^{\sum_{i=1}^{d}k_i} \max_{\boldsymbol{q}\in\{0,1\}^d\backslash\{0\}}
\prod_{i=1}^d \left(\delta_{q_i,0}\delta_{k_i < j}2^{-k_i -\left|k_i-j\right|}+\delta_{q_i,1}\delta_{k_i,j}2^{-k_i}\right) \\
&\lesssim \max_{\boldsymbol{q}\in\{0,1\}^d\backslash\{0\}}
\prod_{i=1}^d \left(\delta_{q_i,0}\delta_{k_i < j}2^{ -\left|k_i-j\right|}+\delta_{q_i,1}\delta_{k_i,j}\right).
\end{split}
\end{equation*}
This estimate is zero unless $k_i\leq j$ and at least one of the $k_i$ is equal to $j$. In this case the maximum corresponds to $q_i=1$ if $k_i=j$ and $q_i=0$ otherwise. This gives precisely (\ref{Walsh_coherence}).
\end{proof}

\subsection{Proof of Theorem \ref{NN_FW_recov_thm}}
For the benefit of the reader, we recall that $A = P_{\mathcal{I}}DV\Psi$. We apply Proposition \ref{needed_G_pp}, noting that the $t_j$ in (\ref{t_k_reshape}) satisfy
\begin{equation}
\label{fibka}
t_j\lesssim \frac{\xi(\textbf{s},\textbf{M},w)}{w_{(j)}^2}\leq s_j\cdot\kappa(\textbf{s},\textbf{M},w),\qquad t\lesssim s\cdot\kappa(\textbf{s},\textbf{M},w).
\end{equation}
It follows that
$$
\sum_{j=1}^rt_j\mu(U^{\textbf{k},j})\lesssim \kappa(\textbf{s},\textbf{M},w)\sum_{j=1}^rs_j{\mu(U^{\textbf{k},j})}.
$$
Combining with (\ref{fibka}), it follows that the bound (\ref{m_G_bdjj}) is satisfied, if
\begin{equation}
\label{needed_bound}
m_\textbf{k}\gtrsim \kappa(\textbf{s},\textbf{M},w) \cdot\left(\sum_{j=1}^rs_j{\mu(U^{\textbf{k},j})}\right)\cdot \mathcal{L},
\end{equation}
where
$$
\mathcal{L}= \frac{r\cdot \log(2m)}{\log(2)}  \cdot \log^2\left(s\cdot\kappa(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}},w)\right) \cdot \log(N) + \log(\epsilon_{\mathbb{P}}^{-1})=d\cdot r^2 \cdot \log(2m) \cdot \log^2\left(s\cdot\kappa(\text{\upshape{\textbf{s}}},\text{\upshape{\textbf{M}}},w)\right) + \log(\epsilon_{\mathbb{P}}^{-1}),
$$
since $N=2^{r\cdot d}$. In the Fourier sampling case, note that, by Lemma \ref{coh_lem1}, (\ref{needed_bound}) holds if (\ref{needed_bounda}) holds. Similarly, in the Walsh sampling case, note that, by Lemma \ref{coh_lem2}, (\ref{needed_bound}) holds if (\ref{needed_boundb}) holds. By Proposition \ref{needed_G_pp}, with probability at least $1-\epsilon_{\mathbb{P}}$, $A$ satisfies the weighted rNSPL of order $(\textup{\textbf{s}},\textup{\textbf{M}})$ with constants $\rho=1/2$ and $\gamma=\sqrt{2}$. The conclusion of Theorem \ref{key_CS_prop} then holds for the uniform recovery of the Haar wavelet coefficients $x=\Psi c\in\mathbb{C}^N.$

For the final part, we use the recovery result of Theorem \ref{key_CS_prop}. The only difference is that we have to compose the NNs with (an approximation of) the matrix $\Psi^*$ to recover approximations of $c$ from approximations of $x=\Psi c$. Recall that
$$
\mathcal{Z}=\max\left\{1,\frac{\max_{j=1,...,r}w_{(j)}\sqrt{(M_j-M_{j-1})}}{\sqrt{\xi(\textup{\textbf{s}},\textup{\textbf{M}},w)}}\right\}
$$
and set
$$
n_0=\left\lceil{\log\left({\delta^{-1}}\mathcal{Z}\right)\kappa^{1/4}\mathcal{Z}}\right\rceil.
$$
Let $p$ be as in Theorem \ref{key_CS_prop} and let $n_1\in\mathbb{Z}_{\geq 0}$ such that $n_0=n_1p+n_2$ for $n_1\in\{0,...,p-1\}$ (the $n$ from the statement of the theorem corresponds to $n_1p$). Set
$$
\phi(y)=\Psi^*\left[\phi_{n_1}(y,0)\right],
$$
where $\phi_{n_1}$ denotes the NN from Theorem \ref{key_CS_prop} with $b_1=1$, $b_2=\|A\|+\delta$ and $\upsilon=e^{-1}$. Strictly speaking, we need to approximate $\|A\|$ and $e^{-1}$, and also apply a rational approximation of the matrix $\Psi^*$ instead of $\Psi^*$, but we have avoided this extra notational clutter (the associated approximation errors can be made smaller than $\kappa^{1/4}\delta$ since the vectors we apply the matrix to are uniformly bounded). Now suppose that $y=P_{\mathcal{I}}DVc+e\in\mathcal{J}({\delta,\mathbf{s},\mathbf{M},w})$, and notice that for $\|c\|_{l^2} \leq 1$ we have that $\|y\|_{l^2}\leq \|A\|+\|e\|_{l^2}\leq b_2$ since $\Psi$ is an isometry. Then, since $C_1,C_2\sim \kappa^{1/4}$ (using that $\kappa\geq 1$), (\ref{best1}) implies that
$$
\left\|\phi(y)-c\right\|_{l^2}=\left\|\phi_{n_0}(y,0)-\Psi c\right\|_{l^2}\lesssim\kappa^{1/4}\delta+b_2\kappa^{1/4}e^{-n_1}.
$$
The theorem follows if we can prove that $b_2e^{-n_1}\lesssim\delta$.

{Let $\mathbf{t}$ be as in \eqref{t_k_reshape} and} let $\Delta_1,\Delta_2,...$ be a partition of $\{1,...,N\}$ such that each support set is $(\textbf{t},\textbf{M})$-sparse. We can choose such as partition with at most
$$
\max_{j=1,...,r}\left\lceil{\frac{M_j-M_{j-1}}{t_j}}\right\rceil\lesssim \max_{j=1,...,r}\left\lceil{\frac{M_j-M_{j-1}}{\min\{\xi(\textbf{s},\textbf{M},w)/w_{(j)}^2,M_j-M_{j-1}\}}}\right\rceil
$$
sets. The proof of Proposition \ref{needed_G_pp} shows that $A$ satisfies the RIPL of order $(\textbf{t},\textbf{M})$ and hence we have for any $x\in\mathbb{C}^N$ that
$$
\|Ax\|_{l^2}\leq \sum_{i}\|A(x_{\Delta_i})\|_{{l^2}} \lesssim \sum_{i}\|x_{\Delta_i}\|_{{l^2}} \lesssim \max_{j=1,...,r}\sqrt{\left\lceil{\frac{M_j-M_{j-1}}{\min\{\xi(\textbf{s},\textbf{M},w)/w_{(j)}^2,M_j-M_{j-1}\}}}\right\rceil}\|x\|_{l^2},
$$
where we have used H\"older's inequality in the last step. It follows that $\|A\|\lesssim\mathcal{Z}$ and hence that $p\lesssim \kappa^{1/4}\mathcal{Z}$ and $b_2\lesssim\mathcal{Z}$. This implies that
$$
n_1\gtrsim\log\left({\delta^{-1}}\mathcal{Z}\right)\quad \text{and} \quad b_2e^{-n_1}\lesssim\delta
$$
and completes the proof of the theorem.\qed

\section{Code and Data}\label{s:code_and_data}
All the code and data used to produce the figures in this manuscript are available from
\begin{displayquote}
\centering
\url{www.github.com/Comp-Foundations-and-Barriers-of-AI/firenet}
\end{displayquote}
The original image, used in Figures \ref{fig:automap_pert}, \ref{fig:automap_pert3} and \ref{fig:stabilisation}, is taken from the MGH--USC HCP dataset\cite{fan2016mgh}. The AUTOMAP network has been trained by authors of \cite{Bo-18}, and the weights and sampling procedure are made available via our GitHub page. The images used in Figure \ref{fig_FIRENET_gen}, are taken from \url{www.pixabay.com} and are free for commercial use without any attribution. In Figure \ref{fig:automap_pert4} we have used the code from \cite{genzel2020solving} and the Operator Discretization Library (ODL) (\url{https://odlgroup.github.io/odl/}) to create the ellipses images. The image used in Figure \ref{fig:exp_conv1} are from the authors' own collection. The FIRENETs have been implemented in both TensorFlow v1.14 and \textsc{MatLab}. The code used to train and test the U-net networks in Figure \ref{fig:automap_pert4} is implemented in TensorFlow v2.3.

\section*{Acknowledgments}
An unrolled version of Chambolle and Pock's primal-dual algorithm \cite[Ch.\ 15.3]{foucart2013mathematical} was developed in TensorFlow by Krisitian M.\ Haug, prior to this work. We are grateful to Kristian, for sharing his code and allowing us to use parts of in when implementing the FIRENETs. We would also like to thank Ben Adcock and Alex Townsend for useful discussions and comments. This work was supported by a Research Fellowship at Trinity College, Cambridge (M.J.C.), and a Leverhulme Prize and a Royal Society University Research Fellowship (A.C.H.).

\bibliographystyle{abbrv}

Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Neural and Evolutionary Computing (cs.NE); Numerical Analysis (math.NA)