Title: Nonparametric regression using deep neural networks with ReLU activation function,
Abstract: Consider the multivariate nonparametric regression model. It is shown that
estimators based on sparsely connected deep neural networks with ReLU
activation function and properly chosen network architecture achieve the
minimax rates of convergence (up to log n-factors) under a general composition
assumption on the regression function. The framework includes many well-studied
structural constraints such as (generalized) additive models. While there is a
lot of flexibility in the network architecture, the tuning parameter is the
sparsity of the network. Specifically, we consider large networks with number
of potential network parameters exceeding the sample size. The analysis gives
some insights why multilayer feedforward neural networks perform well in
practice. Interestingly, the depth (number of layers) of the neural network
architectures plays an important role and our theory suggests that for
nonparametric regression scaling the network depth with the logarithm of the
sample size is natural. It is also shown that under the composition assumption
wavelet estimators can only achieve suboptimal rates.