## A NON-PARAMETRIC REGRESSION VIEWPOINT : GENERALIZATION OF OVERPARAMETRIZED DEEP RELU NETWORK UNDER NOISY OBSERVATIONS 29 Sept 2021, 00:31 (modified: 05 May 2022, 20:19)ICLR 2022 PosterReaders: Everyone
Keywords: Overparametrized Deep Neural Network, Neural Tangent Kernel, Minimax, Non-parametric regression
Abstract: We study the generalization properties of the overparameterized deep neural network (DNN) with Rectified Linear Unit (ReLU) activations. Under the non-parametric regression framework, it is assumed that the ground-truth function is from a reproducing kernel Hilbert space (RKHS) induced by a neural tangent kernel (NTK) of ReLU DNN, and a dataset is given with the noises. Without a delicate adoption of early stopping, we prove that the overparametrized DNN trained by vanilla gradient descent does not recover the ground-truth function. It turns out that the estimated DNN's $L_{2}$ prediction error is bounded away from $0$. As a complement of the above result, we show that the $\ell_{2}$-regularized gradient descent enables the overparametrized DNN achieve the minimax optimal convergence rate of the $L_{2}$ prediction error, without early stopping. Notably, the rate we obtained is faster than $\mathcal{O}(n^{-1/2})$ known in the literature.
One-sentence Summary: Study the generalization of overparametrized deep neural network with relu activation function with noisy dataset.
17 Replies
{"ft_lang_label":"__label__en","ft_lang_prob":0.77734643,"math_prob":0.94216406,"size":1340,"snap":"2022-40-2023-06","text_gpt3_token_len":303,"char_repetition_ratio":0.120508984,"word_repetition_ratio":0.0,"special_character_ratio":0.19328359,"punctuation_ratio":0.0913242,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9869067,"pos_list":[0,1,2],"im_url_duplicate_count":[null,null,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2023-02-09T05:06:26Z\",\"WARC-Record-ID\":\"<urn:uuid:03747d14-015a-4c58-a693-20ed652885cd>\",\"Content-Length\":\"29873\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:3485a82c-1325-4d18-8dc6-ec9607734e22>\",\"WARC-Concurrent-To\":\"<urn:uuid:b9f371b8-3a06-4584-8225-77510268e463>\",\"WARC-IP-Address\":\"35.184.86.251\",\"WARC-Target-URI\":\"https://openreview.net/forum?id=bZJbzaj_IlP\",\"WARC-Payload-Digest\":\"sha1:L3WSG5JUXUYCN6ZUFQQME7KAZ5WXBQIP\",\"WARC-Block-Digest\":\"sha1:N55NS5KYF6RAZJHFYIZFOUP7533EEBI3\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2023/CC-MAIN-2023-06/CC-MAIN-2023-06_segments_1674764501407.6_warc_CC-MAIN-20230209045525-20230209075525-00443.warc.gz\"}"}