Optimal Nonparametric Inference via Deep Neural Network
Deep neural network is a state-of-art method in modern science and technology. Much statistical literature have been devoted to understanding its performance in nonparametric estimation, whereas the results are suboptimal due to a redundant logarithmic sacrifice. In this paper, we show that such log-factors are not necessary. We derive upper bounds for the $L^2$ minimax risk in nonparametric estimation. Sufficient conditions on network architectures are provided such that the upper bounds become optimal (without log-sacrifice). Our proof relies on an explicitly constructed network estimator based on tensor product B-splines. We also derive asymptotic distributions for the constructed network and a relating hypothesis testing procedure. The testing procedure is further proven as minimax optimal under suitable network architectures.
\section{Introduction}

With the remarkable development of modern technology, difficult learning problems can nowadays be tackled smartly via deep learning architectures.
For instance, deep neural networks have led to impressive performance  
in fields such as computer vision, natural language processing, image/speech/audio recognition, social network filtering, machine translation, bioinformatics, drug design, medical image analysis,
where they have demonstrated superior performance to human experts. 
The success of deep networks hinges on their rich expressiveness
(see \cite{db11} , \cite{rpkgdj17}, \cite{mpcb14}, \cite{bs14}, \cite{t16}, \cite{l17} and \cite{y17, y18}).
Recently, deep networks have played an increasingly important role in statistics
particularly in nonparametric curve fitting (see \cite{kk05, hk06, kk17, km11, s17}).
Applications of deep networks in other fields such as image processing or pattern recgnition include, to name a few, \cite{lbh15}, \cite{dlhyysszhw13}, \cite{wwhwzzl14}, \cite{gg16}, etc.

A fundamental problem in statistical applications of deep networks is how accurate they can estimate a nonparametric regression function. 
To describe the problem, let us consider i.i.d. observations $(Y_i, \mathbf{X}_i)$, $i=1,2,\ldots, n$ generated from the following nonparametric model:
\begin{eqnarray}
	Y_i=f_0(\mathbf{X}_i)+\epsilon_i, \label{eq:model}
\end{eqnarray}
where $\mathbf{X}_i \in [0,1]^d$ are i.i.d. $d$-dimensional predictors for a fixed $d\ge1$, 
$\epsilon_i$ are i.i.d. random noise with $E(\epsilon_i)=0$ and $Var(\epsilon_i)=\tau^2$,
$f_0\in\mathcal{H}$ is an unknown function. 
For any $L\in\mathbb{N}$ and $\mathbf{p}=(p_1,\ldots,p_L)\in\mathbb{N}^L$,
let $\mathcal{F}(L,\mathbf{p})$ denote the collection of network functions from $\mathbb{R}^d$ to $\mathbb{R}$ consisting
of $L$ hidden layers with the $l$th layer including $p_l$ neurons. 
The problem of interest is to find an order $R_n$ that controls the $L^2$ minimax risk:
\begin{equation}\label{eqn:goal}
	\inf_{\widehat{f}\in\mathcal{F}(L,\mathbf{p})}\sup_{f_0\in \mathcal{H}}\mathbb{E}_{f_0}\bigg(\|\widehat{f}-f_0\|_{L^2}^2\bigg|\mathbb{X}\bigg)=O_P(R_n),
\end{equation}
where $\mathbb{X}=\{\mathbf{X}_1,\ldots,\mathbf{X}_n\}$ and the infimum is taken over all estimators 
$\widehat{f}\in\mathcal{F}(L,\mathbf{p})$. In other words, we are interested in the performance of the ``best'' network estimator
in the ``worst'' scenario.

Existing results regarding (\ref{eqn:goal}) are sub-optimal.
For instance, when $\mathcal{H}$ is a $\beta$-smooth H\"{o}lder class and $L,\mathbf{p}$ are properly selected, 
it has been argued that $R_n=n^{-\frac{2\beta}{2\beta+d}}(\log{n})^s$ for some constant $s>0$;
see \cite{kk05, hk06, kk17, km11, s17, su18, flm18}. Such results are mostly proved based on empirical processes techniques
in which the logarithmic factors
arise from the entropy bound of the neural network class.
The aim of this paper is to fully remove the redundant logarithmic factors, i.e., 
under proper selections of $L,\mathbf{p}$ one actually has $R_n=n^{-\frac{2\beta}{2\beta+d}}$ in (\ref{eqn:goal}).
This means that neural network estimators can exactly achieve minimax estimation rate.
Our proof relies on an explicitly constructed neural network which is proven minimax optimal.

Some interesting byproducts are worth mentioning.
First, the rate $R_n$ can be further improved when $f_0$ satisfies additional structures.
Specifically, we will show that $R_n=n^{-\frac{2\beta}{2\beta+1}}$ if $f_0$ satisfies
additive structure, i.e., $f_0$ is a sum of univariate $\beta$-H\"{o}lder functions.
Such rate is minimax according to \cite{s85}.
Second, we will derive the pointwise asymptotic distribution of the constructed neural network estimator 
which will be useful to establish pointwise confidence interval.
Third, the constructed neural network estimator will be further used 
as a test statistic which is proven optimal when $L,\mathbf{p}$ are properly selected. 
As far as we know, these are the first provably valid confidence interval and test statistic
based on neural networks in nonparametric regression.

This paper is organized as follows. 
Section \ref{sec:prelim} includes some preliminaries on
deep networks and function spaces.
In Section \ref{sec:optimal:rate:convergence}, we derive
upper bounds for the minimax risk and investigate their optimality. 
Both multivariate regression and additive regression are considered.
Section \ref{sec:basis:approximation} contains the main proof strategy,
which covers the construction of (optimal) network and relates results on network approximation
of tensor product B-splines.
As by products, we also provide limit distributions and optimal testing results in Section \ref{sec:byproducts}. 
The proofs of some of the main results and technical lemmas are deferred to Appendix A-D.

\section{Preliminaries}\label{sec:prelim}
In this section, we review some notion about deep networks and function spaces.
Throughout let $\sigma$ denote the rectifier linear unit (ReLU) activation function,
i.e., $\sigma(x)=(x)_+$ for $x\in\mathbb{R}$.
For any real vectors $\mathbf{v}=(v_1,\ldots,v_r)^T$ and $\mathbf{y}=(y_1,\ldots,y_r)^T$, define the shift activation function
$\sigma_\mathbf{v}(\mathbf{y})=(\sigma(y_1-v_1),\ldots,\sigma(y_r-v_r))^T$.
Let $\mathbf{p}=(p_1,\ldots,p_L)\in\mathbb{N}^L$.
Any $f\in\mathcal{F}(L,\mathbf{p})$ has an expression
\[
f(\mathbf{x})=W_{L+1}\sigma_{\mathbf{v}_{L}}W_{L}\sigma_{\mathbf{v}_{L-1}}\ldots W_2\sigma_{\mathbf{v}_1}W_1 \mathbf{x},\,\,\mathbf{x} \in \mathbb{R}^{d},
\]
where $\mathbf{v}_{l}\in \mathbb{R}^{p_l}$ is a shift vector and $W_{l}\in \mathbb{R}^{p_{l}\times p_{l-1}}$ is a weight matrix. 
Here we have adopted the convention $p_0=d$ and $p_{L+1}=1$.
For simplicity, we only consider fully connected networks and do not make any sparsity assumptions on the entries of $\mathbf{v}_l$ and $W_l$.

Next let us introduce various function spaces under which the estimation rates will be derived.
We will consider two types of function spaces: H\"{o}lder space and additive space.
Let $\Omega=[0,1]^d$ denote the domain of the functions. 
For $f$ defined on $\Omega$, define the supnorm and $L^2$-norm of $f$ by $\|f\|_{\sup}=\sup_{\mathbf{x}\in\Omega}|f(\mathbf{x})|$ 
and $\|f\|_{L^2}^2=\int_\Omega f(\mathbf{x})^2Q(\mathbf{x})d\mathbf{x}$ respectively.
Here $Q(\cdot)$ is the probability density for the predictor $\mathbf{X}_i$'s.
For any $\bm{\alpha}=(\alpha_1, \alpha_2, \ldots, \alpha_d)\in \mathbb{N}^d$, define $|\bm{\alpha}|=\sum_{j=1}^d\alpha_j$ and
\begin{eqnarray*}
	\partial^{\bm{\alpha}}f=\frac{\partial^{|\bm{\alpha}|}f}{\partial x_1^{\alpha_1}\ldots \partial x_d^{\alpha_d}},
\end{eqnarray*}
whenever the partial derivative exists. 
For any $\beta>1$ and $F>0$, let $\Lambda^\beta(F, \Omega)$ denote the ball of $\beta$-H\"{o}lder functions with radius $F$, i.e.,
\begin{eqnarray}
	\Lambda^\beta(F, \Omega)=\bigg\{f: \Omega\to \mathbb{R} \bigg|\;\sum_{\bm{\alpha}:|\bm{\alpha}|\leq \floor{\beta}}\|\partial^{\bm{\alpha}}f\|_{\sup}+\sum_{\bm{\alpha}:|\bm{\alpha}|=\floor{\beta}}\sup_{\mathbf{x}_1\neq \mathbf{x}_2\in \Omega}\frac{|\partial^{\bm{\alpha}}f(\mathbf{x}_1)-\partial^{\bm{\alpha}}f(\mathbf{x}_2)|}{\|\mathbf{x}_1-\mathbf{x}_2\|_2^{\beta-\floor{\beta}}}\leq F\bigg\},\nonumber
\end{eqnarray}
in which $\floor{\beta}$ is the largest integer smaller than $\beta$ and 
$\|v\|_2$ denotes the Euclidean norm of a real vector $v$. 
For any $F>0$ and $\boldsymbol\beta=(\beta_1, \ldots, \beta_d)\in (1,\infty)^d$,
define
\begin{eqnarray}
	\Lambda^{\boldsymbol\beta}_+(F, \Omega)=\left\{f: [0,1]^d \to \mathbb{R}|\; f(\mathbf{x})=a+\sum_{j=1}^dg_{j}(x_j) \textrm{ with } g_j \in \Lambda^{\beta_j}(F, [0,1]), \textrm{ for } j=1,\ldots, d\right\}. \nonumber
\end{eqnarray}
Clearly, any $f\in\Lambda^{\boldsymbol\beta}_+(F, \Omega)$ has an expression $f(\mathbf{x})=a+\sum_{j=1}^dg_{j}(x_j)$
with the $j$th additive component belonging to the ball of univariate $\beta_j$-H\"{o}lder functions with radius $F$.

\section{Minimax Neural Network Estimation}\label{sec:optimal:rate:convergence}

In this section, we derive an upper bound for the $L^2$ minimax risk in the problem (\ref{eqn:goal}).
The risk bound will be proven optimal under suitable circumstances.
To simplify the expressions, we only consider networks with architecture $(L,\mathbf{p}(T))$,
where $\mathbf{p}(T):=(T,\ldots,T)\in\mathbb{N}^L$ for any $T\in\mathbb{N}$.
In other words, we focus on networks whose $L$ layers each have $T$ neurons. 
Our results hold under suitable conditions on $L$ and $T$ as well as the following assumption on the design and model error.
\begin{Assumption}\label{A0}
The probability density $Q(\mathbf{x})$ of $\mathbf{X}$ is supported on $\Omega$.
There exists a constant $c>0$ such that $c^{-1}\le Q(\mathbf{x})\le c$ for any $\mathbf{x}\in\Omega$.
The error terms $\epsilon_i$'s are independent of $\mathbf{X}_i$'s. 
\end{Assumption}

\begin{theorem}\label{thm:neural:estimator:rate:of:convergence:main:text}
Let Assumption \ref{A0} be satisfied and $F>0$ be a fixed constant.
Suppose that $T\to\infty$ and $T\log T=o(n)$ as $n\to\infty$, then it follows that
\begin{eqnarray}\label{thm1:risk:bound}
	\inf_{\widehat{f}\in \mathcal{F}(L,\mathbf{p}(T))}\sup_{f_0\in \Lambda^\beta(F, \Omega)}
	\mathbb{E}_{f_0}\bigg(\|\widehat{f}(\mathbf{x})-f_0(\mathbf{x})\|_{L^2}^2\bigg|\mathbb{X}\bigg)=O_P\bigg(\frac{1}{T^{\frac{2\beta}{d}}}+\frac{T}{n}+\frac{T^2}{2^{\frac{L}{d+k}}}\bigg).
\end{eqnarray}
As a consequence, if $T\asymp n^{\frac{d}{2\beta+d}}$ and $n^{\frac{2\beta+2d}{2\beta+d}}=o(2^{\frac{L}{d+k}})$, then the following holds:
\begin{eqnarray*}
	\inf_{\widehat{f}\in \mathcal{F}(L,\mathbf{p}(T))}\sup_{f_0\in \Lambda^\beta(F, \Omega)}
	\mathbb{E}_{f_0}\bigg(\|\widehat{f}(\mathbf{x})-f_0(\mathbf{x})\|_{L^2}^2\bigg|\mathbb{X}\bigg)=O_P(n^{-\frac{2\beta}{2\beta+d}}).
\end{eqnarray*}
\end{theorem}

Proof of Theorem \ref{thm:neural:estimator:rate:of:convergence:main:text} 
relies on an explicitly constructed network estimator
based on tensor product B-splines; see Section \ref{sec:basis:approximation}.
The minimax risk bound in (\ref{thm1:risk:bound}) consists of three components ${T^{-\frac{2\beta}{d}}}, n^{-1}T, 2^{-\frac{L}{d+k}}T^2$
corresponding to the bias, variance and approximation error of the constructed network.
The optimal risk bound is achieved through balancing the three terms.
The approximation error of the constructed network decreases exponentially along with $L$. 
Networks constructed based on other methods
such as local Taylor approximations (\cite{y17}, \cite{y18} and \cite{s17} have similar approximation performance.
However, their statistical properties are more challenging to deal with due to the unbalanced eigenvalues of the corresponding basis matrix.
In contrast, the eigenvalues of the tensor product B-spline basis matrix are known to have balanced orders, e.g., see \cite{h98},
which plays an important role in deriving the risk bounds.
Also notice that the risk bounds will blow out when $L$ is fixed,
which partially explains the superior performance of deep networks compared with shallow ones; see \cite{es16}.

The optimal rate in Theorem \ref{thm:neural:estimator:rate:of:convergence:main:text} suffers from the `curse' of dimensionality. 
The following theorem demonstrates that this issue can be addressed when $f_0$ has an additive structure.
For $\boldsymbol\beta=(\beta_1, \ldots, \beta_d)\in (1,\infty)^d$, let $\beta^*=\min_{1\le j\le d}\beta_j$.
\begin{theorem}\label{thm:rate:convergence:additive:main:text}
Let Assumption \ref{A0} be satisfied and $F>0$ be a fixed constant. Suppose that $T\to\infty$ and $T\log T=o(n)$ as $n\to\infty$, 
then it follows that
\begin{eqnarray*}
	\inf_{\widehat{f}\in\mathcal{F}(L,\mathbf{p}(T))}\sup_{f_0\in \Lambda^{\boldsymbol\beta}_+(F, \Omega)}
	\mathbb{E}_{f_0}\bigg(\|\widehat{f}(\mathbf{x})-f_0(\mathbf{x})\|_{L^2}^2\bigg|\mathbb{X}\bigg)=O_P\bigg(\frac{1}{T^{2\beta^*}}+\frac{T}{n}+\frac{T^2}{2^{\frac{L}{1+k}}}\bigg).
\end{eqnarray*}
As a consequence, if $T\asymp n^{\frac{1}{2\beta^*+1}}$ and $n^{\frac{2\beta^*+2}{2\beta^*+1}}=o(2^{\frac{L}{1+k}})$, then
\begin{eqnarray*}
\inf_{\widehat{f}\in\mathcal{F}(L,\mathbf{p}(T))}\sup_{f_0\in \Lambda^{\boldsymbol\beta}_+(F, \Omega)}\mathbb{E}_{f_0}\bigg(\|\widehat{f}(\mathbf{x})-f_0(\mathbf{x})\|_{L^2}^2\bigg|\mathbb{X}\bigg)=O_P\left(n^{-\frac{2\beta^*}{2\beta^*+1}}\right).
\end{eqnarray*}
\end{theorem}
The rate $n^{-\frac{2\beta^*}{2\beta^*+1}}$ in Theorem \ref{thm:rate:convergence:additive:main:text} is optimal 
in nonparmetric additive estimation. 
When $\beta_1=\cdots=\beta_d=\beta$, the rate simply becomes $n^{-\frac{2\beta}{2\beta+1}}$
whose optimality has been proven by \cite{s85}.
Otherwise, the optimal rate relies on the least order of smoothness of the $d$ univariate functions.
The proof of Theorem \ref{thm:rate:convergence:additive:main:text} 
is deferred to Appendix C.

\section{Construction of Optimal Networks}\label{sec:basis:approximation}
In this section, we explicitly construct a network estimator $\widehat{f}_{\textrm{net}}\in\mathcal{F}(L,\mathbf{p}(T))$ and derive its risk bound.
Theorems \ref{thm:neural:estimator:rate:of:convergence:main:text} and \ref{thm:rate:convergence:additive:main:text} 
will immediately follow due to the following trivial fact
\begin{equation}
\inf_{\widehat{f}\in\mathcal{F}(L,\mathbf{p}(T))}\sup_{f_0}
	\mathbb{E}_{f_0}\bigg(\|\widehat{f}(\mathbf{x})-f_0(\mathbf{x})\|_{L^2}^2\bigg|\mathbb{X}\bigg)\le \sup_{f_0}
	\mathbb{E}_{f_0}\bigg(\|\widehat{f}_{\textrm{net}}(\mathbf{x})-f_0(\mathbf{x})\|_{L^2}^2\bigg|\mathbb{X}\bigg).\label{eq:basic:inequality}
\end{equation}
The construction process starts from a pilot estimator $\widehat{f}_{\textrm{pilot}}$ obtained under 
tensor product B-splines.
The tensor product B-spline basis functions are further approximated through explicitly constructed multi-layer networks,
which will be aggregated to obtain the network estimator $\widehat{f}_{\textrm{net}}$.
The key step is to show that the discrepancies between the tensor product B-spline basis functions and the corresponding network approximations are reasonably small
such that $\widehat{f}_{\textrm{net}}$ will perform similarly as $\widehat{f}_{\textrm{pilot}}$, and thus, optimally. 

Our construction is different from \cite{y17} and \cite{s17}, where the basis functions are obtained through local Taylor approximation. 
We find that the eigenvalue performance of the local Taylor basis matrix is difficult to quantify so that the corresponding pilot estimator 
cannot be used effectively. Instead, the pilot estimator based on tensor product B-splines is more convenient to deal with.
Other basis such as wavelets or smoothing splines may also work but this will be explored elsewhere. 
\subsection{A Pilot Estimator Through Tensor Product B-splines}
In this subsection, we review tensor product $B$-splines and construct the corresponding pilot estimator.
For any integer $M\geq 2$, 
let $0=t_0<t_1<\cdots<t_{M-1}<t_M=1$ be knots that form a partition of the unit interval.
The definition of univariate B-splines of order $k\ge2$ depends on 
additional knots $t_{-k+1}<t_{-k+2}<\ldots<t_{-1}<0$ and $1<t_{M+1} <\ldots< t_{M+k-1}$. Given knots $t=(t_{-k+1},\ldots, t_{M+k-1})\in \mathbb{R}^{M+2k-1}$, the univariate $B$-spline basis functions of order $k$, denoted $B_{i,k}(x)$, $i=-k+1,-k+2,\ldots, M-1$, can be defined inductively by $B_{i,s}(x)$ for $s=2,3,\ldots, k$. For $s=2$ and $-k+1\le i\le M+k-3$, define
\begin{eqnarray*}
	B_{i, 2}(x)=\begin{cases}
		\frac{x-t_i}{t_{i+1}-t_i}, & \textrm{if } x\in [t_i, t_{i+1}]\\
		\frac{t_{i+2}-x}{t_{i+2}-t_{i+1}},  & \textrm{if } x\in [t_{i+1}, t_{i+2}]\\
		0,& \textrm{elsewhere}
	\end{cases}.
\end{eqnarray*}
Suppose that $B_{i,s}(x)$, $i=-k+1,\ldots, M+k-s-1$ have been defined. Define
\begin{eqnarray}\label{induction:formula}
	B_{i, s+1}=a_{i,s}B_{i,s,t}+b_{i,s}B_{i+1,s, t},\,\,\textrm{for $i=-k+1,-k+2,\ldots, M+k-s-2$,}
\end{eqnarray}
where 
\begin{eqnarray*}
	{a}_{i,s}(x)=\begin{cases}
	0, & \textrm{if } x<t_i\\
	\frac{x-t_i}{t_{i+s}-t_i}, &\textrm{if } t_i\leq x \leq t_{i+s}\\
	0,&\textrm{if } x> t_{i+s}\\
	\end{cases},\;\;\;{b}_{i,s}(x)=\begin{cases}
	0, & \textrm{if } x<t_{t+1}\\
	\frac{t_{i+s+1}-x}{t_{i+s+1}-t_{i+1}}, &\textrm{if } t_{i+1}\leq x \leq t_{i+s+1}\\
	0,&\textrm{if } x> t_{i+s+1}\\
	\end{cases}.
\end{eqnarray*}
Proceeding with this construction, we can obtain $B_{i,k}(x)$.

To approximate a multivariate function, we adopt the tensor product $B$-splines.
Define $\Gamma=\{-k+1, -k+2, \ldots, 0, 1, \ldots, M-1\}^d$ and $q=|\Gamma|=(M+k-1)^d$. 
For $\mathbf{i}=(i_1, i_2, \ldots, i_d)\in \Gamma$, define 
$D_{\mathbf{i},k}(\mathbf{x})=\prod_{j=1}^dB_{i_j,k}(x_j)$
and obtain the corresponding pilot estimator 
\begin{eqnarray}
	\widehat{f}_{\textrm{pilot}}(\mathbf{x})=\sum_{\mathbf{i} \in \Gamma}\widehat{b}_{\mathbf{i}}D_{\mathbf{i}, k}(\mathbf{x}),\label{eq:pilot:estimator}
\end{eqnarray}
where $\widehat{b}_{\mathbf{i}}, \mathbf{i}\in \Gamma$ are the basis coefficients obtained 
by the following least square estimation:
\begin{equation}\label{LSE:eqn}
\widehat{C}:=[\widehat{b}_{\mathbf{i}}]_{\mathbf{i}\in\Gamma}=\argmin_{b_\mathbf{i}\in\mathbb{R}^q}\sum_{i=1}^n \left(Y_i-\sum_{\mathbf{i}\in\Gamma}b_\mathbf{i} D_{\mathbf{i},k}(\mathbf{X}_i)\right)^2.
\end{equation}

\subsection{Network Approximation of Tensor Product B-splines}
In this subsection, we approximate $B_{i,k}$'s through multilayer neural networks.
We first construct networks that approximate the univariate B-spline basis functions,
and then multiply these networks through a product network $\xmark_s$ introduced by \cite{y17} to approximate the tensor product B-spline basis.
Unlike \cite{y17} and \cite{s17}, our construction proceeds in an inductive manner due to the intrinsic induction structure of B-splines.
For any $s\ge1$, the product network
$\xmark_s(x_1, x_2, \ldots, x_s)$ is constructed to approximate the monomials $\prod_{j=1}^s x_j$.  
The following Proposition \ref{proposition:appriximation:product:k} which is due to \cite{y17} provides guarantees for $\xmark_s$.
\begin{proposition}\label{proposition:appriximation:product:k}
For any integers $m\ge1$ and $s\geq 2$, there exists a neural network function $\xmark_s$ with $(s-1)(2m+3)-1$ hidden layers and $10+s$ nodes in each hidden layer such that
for all $x_1, x_2, \ldots, x_s\in [0, 1]$, $0\leq \xmark_s(x_1, x_2, \ldots, x_s)\leq 1$ and
\begin{eqnarray*} 
\bigg|\xmark_s(x_1, x_2, \ldots, x_s)-\prod_{j=1}^s x_j\bigg|\leq (s-1)4^{-m+1}.
\end{eqnarray*}
As a consequence, if $|\widetilde{x}_j-x_j|\leq \delta$ and $\widetilde{x}_j \in [0,1]$ for $j=1,2,\ldots,s$, then
\begin{eqnarray*} 
\bigg|\xmark_s(\widetilde{x}_1, \widetilde{x}_2, \ldots, \widetilde{x}_s)-\prod_{j=1}^s x_j\bigg|\leq (s-1)4^{-m+1}+s\delta.
\end{eqnarray*}
\end{proposition}

In what follows, we will approximate the $k$th order univariate B-spline basis $B_{i,k}$. Fixing integer $m\geq 1$,
our method is based on the induction formula (\ref{induction:formula}) which allows us to start from approximating $B_{i,2}$.
Specifically, we approximate $B_{i,2}$ by $\widetilde{B}_{i, 2}$ defined as
\begin{eqnarray*}
	\widetilde{B}_{i, 2}(x)=c_1\sigma(x-t_i)+c_2\sigma(x-t_{i+1})+c_3\sigma(x-t_{i+2}),
\end{eqnarray*}
where
\begin{eqnarray}
	c_1=\frac{1}{t_{i+1}-t_i}, \;\;c_2=-\frac{t_{i+2}-t_{i}}{t_{i+2}-t_{i+1}}c_1, \;\;c_3=-(t_{i+2}-t_{i}+1)c_1-(t_{i+2}-t_{i+1}+1)c_2.\label{eq:definition:c1c2c3}
\end{eqnarray}
The piecewise linear function $\widetilde{B}_{i,2}$ is exactly a neural network with one hidden layer consisting of three nodes. 
Suppose that we have
constructed $\widetilde{B}_{i,s}(x)$, a neural network approximation of $B_{i,s}$.
Next we will approximate $B_{i,s+1}$. For $-k+1\leq i\leq M+k-s-1$, define
\begin{eqnarray*}
	\widetilde{a}_{i,s}(x)=\begin{cases}
	0, & \textrm{if } x<t_i\\
	\frac{x-t_i}{t_{i+s}-t_i}, &\textrm{if } t_i\leq x \leq t_{i+s}\\
	1,&\textrm{if } x> t_{i+s}\\
	\end{cases},
	\;\;\;\widetilde{b}_{i,s}(x)=\begin{cases}
	1, & \textrm{if } x<t_{i+1}\\
	\frac{t_{i+s+1}-x}{t_{i+s+1}-t_{i+1}}, &\textrm{if } t_{i+1}\leq x \leq t_{i+s+1}\\
	0, &\textrm{if } x> t_{i+s+1}\\
	\end{cases}.
\end{eqnarray*}
In terms of ReLU activation function, we can rewrite the above as 
$\widetilde{a}_{i,s}(x)=\frac{1}{t_{i+s}-t_i}\sigma(x-t_i)-\frac{1}{t_{i+s}-t_i}\sigma(x-t_{i+s})$ and $\widetilde{b}_{i,s}(x)=-\frac{1}{t_{i+s+1}-t_{i+1}}\sigma(x-t_{i+1})+\frac{1}{t_{i+s+1}-t_i}\sigma(x-t_{i+s+1})+1$, which implies that $\widetilde{a}_{i,s}$ and $\widetilde{b}_{i,s}$
are exactly neural networks with one hidden layer consisting of two nodes (see Figure \ref{figure:tildeBi2:to:tildeBi3}). 
For $i=-k+1,\ldots, M+k-s-2$, define
\begin{eqnarray*}
	\widetilde{B}_{i, s+1}(x)=\frac{\xmark_2(\widetilde{a}_{i,s}(x), \widetilde{B}_{i,s}(x))+\xmark_2(\widetilde{b}_{i,s}(x), \widetilde{B}_{i+1,s}(x))+2\times 4^{-m+1}+\frac{8^{s}}{7}4^{-m}}{1+4\times 4^{-m+1}+\frac{8^{s}}{14}4^{-m+1}}, x\in[0,1].
\end{eqnarray*}
The `seemingly strange' normalizing constant forces $\widetilde{B}_{i, s+1}(x)$ to take values in $[0,1]$.
We repeat the above steps until we reach the construction of $\widetilde{B}_{i, k}$
(see Figure \ref{figure:tildeBi2:to:tildeBi3} for an illustration of such induction). 
We then approximate  ${B}_{i,k}$ by $\widetilde{B}_{i,k}$.
Note that $\widetilde{B}_{i,k}$ has $(2m+4)(k-2)+1$ hidden layers and $8(M+k-3)$ nodes on each hidden layer.

\begin{figure}[ht!]
\centering
\subfigure[]{\includegraphics[width=2 in, height=1.5 in]{p1.pdf}}
\hspace{1cm}
\subfigure[]{\includegraphics[width=2 in, height=1.5 in]{p2.pdf}}
\vspace{0.5cm}

\subfigure[]{\includegraphics[width=2 in, height=1.5 in]{p3.pdf}}
\hspace{1cm}
\subfigure[]{\includegraphics[width=2 in, height=1.5 in]{p4.pdf}}
\caption{Construction of $\widetilde{B}_{i,3}$ through induction. 
(a) and (b)  demonstrate the architectures of the networks $\widetilde{a}_{i,2}$ and $\widetilde{b}_{i,2}$. 
(c) demonstrates the architecture of the network $\widetilde{B}_{i,2}$ with $c_1, c_2, c_3$ defined in (\ref{eq:definition:c1c2c3}). 
(d) demonstrates the induction relationship between $\widetilde{B}_{i,3}$ and $\widetilde{B}_{i,2}$.
}
\label{figure:tildeBi2:to:tildeBi3}
\end{figure}



We next approximate the tensor product B-spline basis $D_{\mathbf{i}}(\mathbf{x})=\prod_{j=1}^dB_{i_j,k}(x_j)$ by
\begin{eqnarray*}
	\widetilde{D}_{\mathbf{i}, k}(\mathbf{x})=\xmark_d(\widetilde{B}_{i_1,k}(x_1),\widetilde{B}_{i_2,k}(x_2),\ldots, \widetilde{B}_{i_d,k}(x_d)), \textrm{ for each } \mathbf{i}=(i_1,\ldots, i_d) \in \Gamma.
\end{eqnarray*}
Note that
$\widetilde{D}_{\mathbf{i}, k}$ has $(2m+3)(k+d-3)+k-1$ hidden layers each consisting of $(d+7)(M+2k-3)^d$ nodes.
Finally, paralellizing $\widetilde{D}_{\mathbf{i}, k}(\mathbf{x}), I \in \Gamma$ according to (\ref{eq:pilot:estimator}), 
we construct $\widehat{f}_{\textrm{net}}$ as 
\begin{eqnarray}
	\widehat{f}_{\textrm{net}}(\mathbf{x})=\sum_{\mathbf{i}\in \Gamma}\widehat{b}_{\mathbf{i}}\widetilde{D}_{\mathbf{i}, k}(\mathbf{x}),\,\,\,\,
	x\in\Omega.\label{eq:optimal:DNN:estimator}
\end{eqnarray}

In comparing (\ref{eq:pilot:estimator}) with (\ref{eq:optimal:DNN:estimator}), if 
we can show that ${D}_{\mathbf{i}, k}$ and $\widetilde{D}_{\mathbf{i},k}$ are close enough, 
then one can expect that $\widehat{f}_{\text{net}}$ performs similarly to $\widehat{f}_{\textrm{pilot}}$. 
A rich class of statistical results in literature enable us to efficiently analyze $\widehat{f}_{\textrm{pilot}}$.
In the rest of our analysis, we focus on cardinal B-splines for convenience. 
\begin{Assumption}\label{Assumption:A1}
\label{A1:c} 
The knots $\{t_i, i=-k+1,\ldots, M+k-1\}$ have constant separation $h=M^{-1}$.
\end{Assumption}

\begin{remark}
Assumption \ref{Assumption:A1} can be relaxed to $\max_{i}(t_{i+1}-t_{i})/\min_{i}(t_{i+1}-t_{i})\leq c$ for some constant $c>0$,
under which one needs to redefine the separation
$h=\max_{i}(t_{i+1}-t_{i})$. Results in this section continue to hold.
This is a standard assumption for $B$-spline literature; see \cite{h98}.
\end{remark}
The following Theorem \ref{thm:approximation:sieve:DNN}
is the main technical result of this paper,
based on which Theorems \ref{thm:neural:estimator:rate:of:convergence:main:text},
\ref{thm:rate:convergence:additive:main:text}, \ref{thm:asymptotic:normality:neural:main:text}
and \ref{thm:optimal:test:neural:main:text} will be proved.

\begin{theorem}\label{thm:approximation:sieve:DNN}
For fixed positive integer $m$,  
$\widehat{f}_{\textrm{net}}\in \mathcal{F}(L,\mathbf{p}(T))$ with $L=(2m+3)(k+d-1)+1$ and $T=3(M+2k)^d$. Under Assumption \ref{A0} and \ref{Assumption:A1}, 
if $k>\beta$ and $F>0$, then it holds that 
\begin{eqnarray*}
	\sup_{f_0\in\Lambda^\beta(F, \Omega)}\mathbb{E}_{f_0}\left\{\sup_{\mathbf{x} \in \Omega}|\widehat{f}_{\text{net}}(\mathbf{x})-\widehat{f}_{\textrm{pilot}}(\mathbf{x})|^2
	\bigg|\mathbb{X}\right\}=O_P(h^{-2d}4^{-2m}).
\end{eqnarray*}
\end{theorem}
Theorem \ref{thm:approximation:sieve:DNN} says that $\widehat{f}_{\text{net}}$ 
is a neural network with $L=(2m+3)(k+d-3)+k$ hidden layers each consisting of $T=(d+2)(M+k-1)^d$ nodes. 
The theorem also provides an explicit upper bound in terms of $(h,d,m)$ for the difference between $\widehat{f}_{\textrm{net}}$ and $\widehat{f}_{\textrm{pilot}}$.
The proof of Theorem \ref{thm:approximation:sieve:DNN} relies on following Lemma \ref{lemma:approximation:b:spline:d:1},
\ref{lemma:approximation:b:spline:d:d}, \ref{lemma:spline:approximation} and \ref{lemma:difference:c:hat:c:0:main:text}.
Let $\mathbf{i}_1, \mathbf{i}_2,\ldots, \mathbf{i}_q$ be the elements of $\Gamma$.
Define
\begin{eqnarray*}
     \mathbf{B}_k(x)&=&(B_{-k+1,k}(x), B_{-k+2,k}(x),\ldots, B_{0,k}(x),B_{1,k}(x),\ldots, B_{M-1,k}(x))^T\in \mathbb{R}^{M-k+1},\\
     \mathbf{D}_{k}(\mathbf{x})&=&(D_{\mathbf{i}_1, k}(\mathbf{x}),D_{\mathbf{i}_2, k}(\mathbf{x}),\ldots, D_{\mathbf{i}_q, k}(\mathbf{x}))^T\in \mathbb{R}^{q},\\ 
     \widetilde{\mathbf{B}}_k(x)&=&(\widetilde{B}_{-k+1,k}(x), \widetilde{B}_{-k+2,k}(x),\ldots, \widetilde{B}_{M-1,k}(x))^T\in \mathbb{R}^{M-k+1},\\
	\widetilde{\mathbf{D}}_k(\mathbf{x})&=&(\widetilde{D}_{\mathbf{i}_1,k}(\mathbf{x}), \widetilde{D}_{\mathbf{i}_2,k}(\mathbf{x}),\ldots, \widetilde{D}_{\mathbf{i}_q,k}(\mathbf{x}))^T\in \mathbb{R}^q.
\end{eqnarray*}
Lemma \ref{lemma:approximation:b:spline:d:1} quantifies the differences between $\widetilde{\mathbf{B}}_k(\cdot)$ and $\mathbf{B}_k(\cdot)$.
For convenience, for $L,p_0,\ldots,p_{L+1}\in\mathbb{N}$, let $\mathcal{N}\mathcal{N}(L,(p_0,p_1,\ldots,p_L,p_{L+1}))$
denote the class of $p_0$-input-$p_{L+1}$-output ReLU neural network functions of $L$ hidden layers,
with the $j$th layer consisting of $p_j$ nodes, for $j=1,\ldots,L$. For any $v=(v_1, \ldots, v_p)\in \mathbb{R}^p$, let $\|v\|_\infty=\max_{1\leq i \leq p}|v_i|$.

\begin{lemma}\label{lemma:approximation:b:spline:d:1}
Given integers $k, M\geq 2$ and knots $t_{-k+1}<t_{-k+2}<\ldots< t_0<t_1< \ldots<t_M< t_{M+1} <\ldots< t_{M+k-1}$ such that $t_0=0, t_M=1$,
there exists a $\widetilde{\mathbf{B}}_k\in \mathcal{N}\mathcal{N}(k(2m+3), (1, 3(M+2k), \ldots, 3(M+2k), M+k-1))$
taking values in $[0,1]$, such that
\begin{eqnarray*}
	\sup_{x\in[0,1]}\|\widetilde{\mathbf{B}}_k(x)-\mathbf{B}_{k}(x)\|_\infty\leq \frac{8^{k}}{14}4^{-m}.
\end{eqnarray*}
\end{lemma}
The proof of Lemma \ref{lemma:approximation:b:spline:d:1} is given in Appendix A. Based on Lemma \ref{lemma:approximation:b:spline:d:1}, we can bound the approximation error between $\widetilde{\mathbf{D}}_k$ and $\mathbf{D}_k$, which is summarized as Lemma \ref{lemma:approximation:b:spline:d:d}.

\begin{lemma}\label{lemma:approximation:b:spline:d:d}
Given integers $k, M\geq 2$ and knots $t_{-k+1}<t_{-k+2}<\ldots< t_0<t_1< \ldots<t_M< t_{M+1} <\ldots< t_{M+k-1}$ with 
$t_0=0, t_M=1$, there exist a
$\widetilde{\mathbf{D}}_k\in \mathcal{N}\mathcal{N}((2m+3)(k+d-1), (d, 3(M+2k)^d, \ldots,  3(M+2k)^d, (M+k-1)^d))$ such that
\begin{eqnarray*}
	\bigg\|\widetilde{\mathbf{D}}_k(\mathbf{x})-\mathbf{D}_{k}(\mathbf{x})\bigg\|_{\infty}\leq [4(d-1)+8^k]4^{-m},\;\textrm{ for all } \mathbf{x} \in \Omega.
\end{eqnarray*}
Furthermore, each element of $\widetilde{\mathbf{D}}_k$ is in $[0,1]$.
\end{lemma}
{\it Proof of Lemma \ref{lemma:approximation:b:spline:d:d}.}
Let $\widetilde{\mathbf{B}}_k(x_1), \widetilde{\mathbf{B}}_k(x_1), \ldots, \widetilde{\mathbf{B}}_k(x_d)$ be the neural networks provided 
in Lemma \ref{lemma:approximation:b:spline:d:1}, which satisfy $|\widetilde{B}_{i, k}(x)-{B}_{i, k}(x)|\leq \delta_m$, where
$\delta_m=8^k4^{-m}/14$. For each $(i_1, i_2, \ldots, i_d)\in \{-k+1, -k+2, \ldots, 1,2, \ldots, M-1\}^d$, we apply the
product network $\xmark_d$ given in Proposition \ref{proposition:appriximation:product:k} to 
$(\widetilde{B}_{i_1,k}(x_1), \widetilde{B}_{i_2,k}(x_2),\ldots, \widetilde{B}_{i_d,k}(x_d))$. According to Proposition \ref{proposition:appriximation:product:k}, we have
\begin{eqnarray*}
	\bigg|\xmark_d(\widetilde{B}_{i_1,k}(x_1), \widetilde{B}_{i_2,k}(x_2),\ldots, \widetilde{B}_{i_d,k}(x_d))-\prod_{j=1}^d B_{i_j,k}(x_j)\bigg|&\leq& (d-1)4^{-m+1}+d\delta_m\\
	&\leq& [4(d-1)+8^k]4^{-m}.
\end{eqnarray*}
Now we deploy $\xmark_d(\widetilde{B}_{i_1,k}(x_1), \widetilde{B}_{i_2,k}(x_2),\ldots, \widetilde{B}_{i_d,k}(x_d))$ parallelly to construct the network $\widetilde{\mathbf{D}}_k$. Since we apply neural network $X_d$ to output of $\widetilde{\mathbf{B}}_k$, so the total number of hidden layers is at most $k(2m+3)+1+(d-1)(2m+3)-1=(2m+3)(d+k-1)$. Moreover, the number nodes in each hidden layer is not greater than the number of nodes in the
output layer, which is further bounded by $3(M+2k)^d$. This completes the proof.

The following Lemma \ref{lemma:spline:approximation} 
is consequence of \cite[Theorem 15.1 and Theorem 15.2]{gkkw06} and \cite[Theorem 12.8 and (13.69)]{s07}, which
quantifies an approximation error
of tensor product B-spline.


\begin{lemma}\label{lemma:spline:approximation}
Suppose that Assumption \ref{Assumption:A1} is satisfied. 
For any $f\in \Lambda^\beta(F, \Omega)$ and any integer $k\geq \beta$, there exists a real sequence $c_i$ such that
	$\sup_{\mathbf{x} \in \Omega}\bigg|\sum_{\mathbf{i}\in \Gamma}c_i D_{\mathbf{i}}(\mathbf{x})-f(\mathbf{x})\bigg|\leq A_f h^\beta$
where $A_f>0$ which only depends on partial derivatives of $f$ upto order $k$. Moreover, the sequence $c_i$ satisfy $|c_i|\leq A_f$. Moreover, $sup_{f\in \Lambda^\beta(F, \Omega)}A_f<\infty$, where the upper bound only depends on $F$ and $\beta$.
\end{lemma}

Define $\Phi=(\mathbf{D}_k(\mathbf{x}_1),\ldots, \mathbf{D}_k(\mathbf{x}_n))^T\in \mathbb{R}^{n\times q}$ and $\mathbf{Y}=(Y_1,\ldots Y_n)^T$.
The following Lemma \ref{lemma:difference:c:hat:c:0:main:text} 
quantifies the magnitude of $\widehat{C}=(\Phi^T\Phi)^{-1}\Phi^T\mathbf{Y}$;
recalling that such $\widehat{C}$ is the solution to the least square problem (\ref{LSE:eqn}).
Its proof is provided in Appendix B.

\begin{lemma}\label{lemma:difference:c:hat:c:0:main:text}
Under Assumptions \ref{A0} and \ref{Assumption:A1}, 
if $h=o(1)$ and $\log(h^{-1})=o(nh^d)$, then 
\[
\sup_{f_0\in\Lambda^\beta(F,\Omega)}
\mathbb{E}_{f_0}\left(\widehat{C}^T\widehat{C}\big|\mathbb{X}\right)=O_P(h^{-d}).
\]
\end{lemma}

We are now ready to provide the Proof of Theorem \ref{thm:approximation:sieve:DNN}.

{\it Proof of Theorem \ref{thm:approximation:sieve:DNN}.}
For any $f_0\in\Lambda^\beta(F, \Omega)$, let $\mathbf{f}_0=(f_0(\mathbf{x}_1),\ldots,f_0(\mathbf{x}_n))^T$.
Also let 
$\bm{\epsilon}=(\epsilon_1,\ldots,\epsilon_n)^T$,
$\widehat{\mathbf{f}}_{\textrm{pilot}}=(\widehat{f}_{\textrm{pilot}}(\mathbf{x}_1),
\ldots,\widehat{f}_{\textrm{pilot}}(\mathbf{x}_n))^T$. 
According to Lemma \ref{lemma:spline:approximation} and by $k\geq \beta$, 
there exists a $C=(c_1, c_2,\ldots, c_q)^T\in \mathbb{R}^q$ 
such that for any $\mathbf{x}\in\Omega$,
$|C^T\mathbf{D}_k(\mathbf{x})-f_0(\mathbf{x})|\leq A_{f_0}h^\beta$. By least square algorithm (\ref{LSE:eqn}), we have
\begin{eqnarray*}
	\widehat{\mathbf{f}}_{\textrm{pilot}}=\Phi(\Phi^T \Phi)^{-1}\Phi^T \mathbf{Y}&=&\Phi(\Phi^T \Phi)^{-1}\Phi^T (\Phi C+\mathbf{E}+\bm{\epsilon})\\
	&=&\Phi C+\Phi(\Phi^T \Phi)^{-1}\Phi^T \mathbf{E}+\Phi(\Phi^T \Phi)^{-1}\Phi^T \bm{\epsilon}\\
	&=&\mathbf{f}_0-(I-\Phi(\Phi^T \Phi)^{-1}\Phi^T)\mathbf{E}+\Phi(\Phi^T \Phi)^{-1}\Phi^T \bm{\epsilon},
\end{eqnarray*}
where $\mathbf{E}=(E_1, E_2, \ldots, E_n)^T\in \mathbb{R}^n$ with $E_i=f_0(\mathbf{x}_i)-C^T\mathbf{D}_k(\mathbf{x}_i)$. 
It follows from (\ref{eq:pilot:estimator}), (\ref{eq:optimal:DNN:estimator}) and (\ref{LSE:eqn}) that
$\widehat{f}_{\textrm{pilot}}(\mathbf{x})=\widehat{C}^T\mathbf{D}_k(\mathbf{x})$ and $\widehat{f}_{\textrm{net}}(\mathbf{x})=\widehat{C}^T\widetilde{\mathbf{D}}_k(\mathbf{x})$.
Therefore, for any $\mathbf{x}\in\Omega$, we have
\begin{eqnarray*}
	|\widehat{f}_{\textrm{pilot}}(\mathbf{x})-\widehat{f}_{\textrm{net}}(\mathbf{x})|^2&=&\big\|\widehat{C}^T\left(\mathbf{D}_k(\mathbf{x})-\widetilde{\mathbf{D}}_k(\mathbf{x})\right)\big\|_2^2\\
	&=& \widehat{C}^T\widehat{C}\left(\mathbf{D}_k(\mathbf{x})-\widetilde{\mathbf{D}}_k(\mathbf{x})\right)^T\left(\mathbf{D}_k(\mathbf{x})-\widetilde{\mathbf{D}}_k(\mathbf{x})\right)\\
	&\leq&q\widehat{C}^T\widehat{C} \sup_{\mathbf{x} \in [0,1]^d}\big\|\mathbf{D}_k(\mathbf{x})-\widetilde{\mathbf{D}}_k(\mathbf{x})\big\|_{\infty}^2
	\le q\widehat{C}^T\widehat{C}[4(d-1)+8^k]^24^{-2m},
\end{eqnarray*}
where the last inequality follows from Lemma \ref{lemma:approximation:b:spline:d:d}.
Following Lemma \ref{lemma:difference:c:hat:c:0:main:text} and the fact $q\asymp h^{-d}$, we have
\begin{eqnarray*}
	\sup_{f_0\in\Lambda^\beta(F,\Omega)}
	\mathbb{E}_{f_0}\bigg(\sup_{\mathbf{x} \in \Omega}|\widehat{f}_{\textrm{pilot}}(\mathbf{x})-\widehat{f}_{\textrm{net}}(\mathbf{x})|^2 \bigg|\mathbb{X}\bigg)&\leq&
	q[4(k-1)+8k]^2 4^{-2m}\sup_{f_0\in\Lambda(F,\Omega)}\mathbb{E}\left(\widehat{C}^T\widehat{C}\big|\mathbb{X}\right)\\
	&=&O_P(h^{-2d}4^{-2m}),
\end{eqnarray*}
which completes the proof.

Theorem \ref{thm:neural:estimator:rate:of:convergence:main:text} is a simple consequence of Theorem \ref{thm:approximation:sieve:DNN}
and Lemma \ref{lemma:rate:of:convergence:sieve} below with $h\asymp T^{-1/d}$ and $m\asymp \frac{L}{3(k+d)}$. 
Proof of Lemma \ref{lemma:rate:of:convergence:sieve} is deferred to Appendix B.
\begin{lemma}\label{lemma:rate:of:convergence:sieve}
Under the Assumptions \ref{A0} and \ref{Assumption:A1}, if $h=o(1)$ and $\log(h^{-1})=o(nh^d)$ hold, then it holds that
\begin{eqnarray*}
	\sup_{f_0\in\Lambda^\beta(F,\Omega)}
	\mathbb{E}_{f_0}\bigg\{\|\widehat{f}_{\textrm{pilot}}-f_0\|_{L^2}^2\bigg|\mathbb{X}\bigg\}=O_P\bigg(h^{2\beta}+\frac{1}{nh^d}\bigg).
\end{eqnarray*}
\end{lemma}

\section{Asymptotic Distribution and Optimal Testing}\label{sec:byproducts}
In this section,  
we derive asymptotic distributions for $\widehat{f}_{\textrm{net}}$ and a 
corresponding hypothesis testing procedure. 
The results are simply byproducts of Theorem \ref{thm:approximation:sieve:DNN}.
Theorem \ref{thm:asymptotic:normality:neural:main:text} below establishes a pointwise asymptotic distribution for
$\widehat{f}_{\textrm{net}}(\mathbf{x})$ for any $\mathbf{x} \in \Omega$. The result is a direct consequence of Theorem \ref{thm:approximation:sieve:DNN} 
and the asymptotic distribution of $\widehat{f}_{\textrm{pilot}}(\mathbf{x})$. 
\begin{theorem}\label{thm:asymptotic:normality:neural:main:text}
Under the Assumptions \ref{A0} and \ref{Assumption:A1}, if $hn^{\frac{1}{2\beta+d}}=o(1)$, $\log(h^{-1})=o(nh^d)$ and $n^{1/2}h^{-d/2}=o(4^{m})$, then
 for any $\mathbf{x} \in \Omega$, we have
\begin{eqnarray*}
	\frac{\widehat{f}_{\textrm{net}}(\mathbf{x})-f_0(\mathbf{x})}{\sqrt{\mathbf{D}^T_k(\mathbf{x})(\Phi^T\Phi)^{-1}\mathbf{D}_k(\mathbf{x})}}\xrightarrow[]{D}N(0,1),
\end{eqnarray*}
where $\Phi=(\mathbf{D}_k(\mathbf{x}_1),\mathbf{D}_k(\mathbf{x}_2),\ldots, \mathbf{D}_k(\mathbf{x}_n))^T\in \mathbb{R}^{n\times q}$.
\end{theorem}
{\it Proof of Theorem \ref{thm:asymptotic:normality:neural:main:text}.}
For fixed $\mathbf{x}\in [0,1]^d$, let $V(\mathbf{x})=\mathbf{D}^T_k(\mathbf{x})(\Phi^T\Phi)^{-1}\mathbf{D}_k(\mathbf{x})$.
By \cite[Theorems 3.1 and 5.2]{h03}, it follows that
\begin{eqnarray}\label{thm11:eqn:1}
	\frac{\widehat{f}_{\textrm{pilot}}(\mathbf{x})-f_0(\mathbf{x})}{\sqrt{V(\mathbf{x})}}\xrightarrow[]{D}N(0,1).
\end{eqnarray}
By Lemma \ref{lemma:empirical:eigen:value} in Appendix B, with probability approaching 1, we have 
$V(\mathbf{x})\geq \frac{2}{a_1nh^d}\mathbf{D}_k(\mathbf{x})^T\mathbf{D}_k(\mathbf{x})\geq  \frac{2b^2}{a_1nh^d}$.
By the proof of Theorem \ref{thm:approximation:sieve:DNN} we get that
$|\widehat{f}_{\textrm{pilot}}(\mathbf{x})-\widehat{f}_{\textrm{net}}(\mathbf{x})|^2=O_P(h^{-2d}4^{-2m})$.
Therefore, 
\begin{equation}\label{thm11:eqn:2}
\frac{\widehat{f}_{\textrm{pilot}}(\mathbf{x})-\widehat{f}_{\textrm{net}}(\mathbf{x})}{\sqrt{V(\mathbf{x})}}=O_P(n^{1/2}h^{-d/2}4^{-m})=o_P(1).
\end{equation}
Theorem \ref{thm:asymptotic:normality:neural:main:text} follows by (\ref{thm11:eqn:1}) and (\ref{thm11:eqn:2}). This completes the proof.

In what follows, we consider a hypothesis testing problem: $H_0: f_0=0$ vs. $H_1: f\neq0$.
Consider a test statistic $T_n=\|\widehat{f}_{\textrm{net}}\|_n^2$, where $\|f\|_n^2=\sum_{i=1}^n f(\mathbf{x}_i)^2/n$
is the empirical norm.
The following Theorem \ref{thm:optimal:test:neural:main:text}
derives null distribution of $T_n$ and analyzes its power under a sequence of local alternatives.
Again, this result is a byproduct of Theorem \ref{thm:approximation:sieve:DNN}.

\begin{theorem}\label{thm:optimal:test:neural:main:text}
Suppose $n^{\frac{4\beta+2d}{4\beta+d}}=O(4^m)$ and $h\asymp n^{-\frac{2}{4\beta+d}}$, then the following hold:
\begin{enumerate}
\item Under $H_0: f_0=0$, it follows that 
\begin{eqnarray}\label{thm:11:1}
	\frac{nT_n-q}{\sqrt{2q}}\xrightarrow[]{D}N(0,1).
\end{eqnarray}
\item For any $\delta>0$, there exists a $C_\delta>0$ such that, under $H_1: f=f_0$ with 
$\|f_0\|_n\geq C_\delta n^{-\frac{2\beta}{4\beta+d}}$, it holds that
\begin{eqnarray}\label{thm:11:2}
	\mathbb{P}\bigg(\bigg|\frac{nT_n-q}{\sqrt{2q}}\bigg|>z_{\alpha/2}\bigg)\geq 1-\delta,
\end{eqnarray}
where $z_{\alpha/2}$ is the $1-\alpha/2$ upper percentile of standard normal variable.
\end{enumerate}
\end{theorem}

Part (\ref{thm:11:1}) of Theorem \ref{thm:optimal:test:neural:main:text} suggests a testing rule at significance $\alpha$:
reject $H_0$ if and only if 
\[
\bigg|\frac{nT_n-q}{\sqrt{2q}}\bigg|\ge z_{\alpha/2}.
\]
Part (\ref{thm:11:2}) of Theorem \ref{thm:optimal:test:neural:main:text}
says that the power of $T_n$ is at least $1-\delta$ provided that the
null and alternative hypotheses are separated by $C_\delta n^{-\frac{2\beta}{4\beta+d}}$
in terms of $\|\cdot\|_n$-norm.
The separation rate is optimal in the sense of \cite{Ingster93}.

{\it Proof of Theorem \ref{thm:optimal:test:neural:main:text}.}
Observe that
\begin{eqnarray}\label{eq:them:optimal:test:neural}
	\frac{n\|\widehat{f}_{\textrm{net}}\|_n^2-q}{\sqrt{2q}}=\frac{n\|\widehat{f}_{\textrm{pilot}}\|_n^2-q}{\sqrt{2q}}+\frac{n\|\widehat{f}_{\textrm{net}}\|_n^2-n\|\widehat{f}_{\textrm{pilot}}\|_n^2}{\sqrt{2q}}.
\end{eqnarray}
By Theorem \ref{thm:approximation:sieve:DNN} and Lemma \ref{lemma:rate:of:convergence:sieve} (see Appendix B for its proof),
both $\|\widehat{f}_{\textrm{net}}-\widehat{f}_{\textrm{pilot}}\|_n$ and $\|\widehat{f}_{\textrm{pilot}}-f_0\|_n$
are $O_P(1)$, we have
\begin{eqnarray*}
	|\|\widehat{f}_{\textrm{net}}\|_n^2-\|\widehat{f}_{\textrm{pilot}}\|_n^2|&=&
	|\|\widehat{f}_{\textrm{net}}\|_n-\|\widehat{f}_{\textrm{pilot}}\|_n|\times \left(\|\widehat{f}_{\textrm{net}}\|_n+\|\widehat{f}_{\textrm{pilot}}\|_n\right)\\
	&\leq&\|\widehat{f}_{\textrm{net}}-\widehat{f}_{\textrm{pilot}}\|_n\times \left(\|\widehat{f}_{\textrm{net}}-\widehat{f}_{\textrm{pilot}}\|_n+2\|\widehat{f}_{\textrm{pilot}}\|_n\right)\\
	&\leq&\|\widehat{f}_{\textrm{net}}-\widehat{f}_{\textrm{pilot}}\|_n\times 
	\left(\|\widehat{f}_{\textrm{net}}-\widehat{f}_{\textrm{pilot}}\|_n+2\|\widehat{f}_{\textrm{pilot}}-f_0\|_n+2\|f_0\|_n\right)\\
	&=&\|\widehat{f}_{\textrm{net}}-\widehat{f}_{\textrm{pilot}}\|_n\times O_P(1)\\
	&=&O_P(h^{-d}4^{-m}).
\end{eqnarray*}
Therefore, the second term in (\ref{eq:them:optimal:test:neural})
is of order $O_P(nh^{-d}4^{-m}q^{-1/2})=O_P\left(n^{\frac{4\beta+2d}{4\beta+d}}4^{-m}\right)=o_P(1)$, 
where we have used the fact $q\asymp h^{-d}$.
The result then follows by Lemma \ref{lemma:optimal:test:sieve} in Appendix D. This completes the proof.

\newpage

2019-02-06T02:12:29.000
1902
1902.01687
en
https://arxiv.org/abs/1902.01687
Deep neural network is a state-of-art method in modern science and technology. Much statistical literature have been devoted to understanding its performance in nonparametric estimation, whereas the results are suboptimal due to a redundant logarithmic sacrifice. In this paper, we show that such log-factors are not necessary. We derive upper bounds for the $L^2$ minimax risk in nonparametric estimation. Sufficient conditions on network architectures are provided such that the upper bounds become optimal (without log-sacrifice). Our proof relies on an explicitly constructed network estimator based on tensor product B-splines. We also derive asymptotic distributions for the constructed network and a relating hypothesis testing procedure. The testing procedure is further proven as minimax optimal under suitable network architectures.
Machine Learning (cs.LG); Machine Learning (stat.ML)
Optimal Nonparametric Inference via Deep Neural Network
Qwen/Qwen-72B
1. YES
2. YES