## Neural Empirical Bayes

Saeed Saremi, Aapo Hyvärinen; 20(181):1−23, 2019.

### Abstract

We unify kernel density estimation and empirical Bayes and address a set of problems in unsupervised machine learning with a geometric interpretation of those methods, rooted in the concentration of measure phenomenon. Kernel density is viewed symbolically as $X\rightharpoonup Y$ where the random variable $X$ is smoothed to $Y= X+N(0,\sigma^2 I_d)$, and empirical Bayes is the machinery to denoise in a least-squares sense, which we express as $X \leftharpoondown Y$. A learning objective is derived by combining these two, symbolically captured by $X \rightleftharpoons Y$. Crucially, instead of using the original nonparametric estimators, we parametrize the energy function with a neural network denoted by $\phi$; at optimality, $\nabla \phi \approx -\nabla \log f$ where $f$ is the density of $Y$. The optimization problem is abstracted as interactions of high-dimensional spheres which emerge due to the concentration of isotropic Gaussians. We introduce two algorithmic frameworks based on this machinery: (i) a “walk-jump” sampling scheme that combines Langevin MCMC (walks) and empirical Bayes (jumps), and (ii) a probabilistic framework for associative memory, called NEBULA, defined a la Hopfield by the gradient flow of the learned energy to a set of attractors. We finish the paper by reporting the emergence of very rich “creative memories” as attractors of NEBULA for highly-overlapping spheres.

[abs][pdf][bib]
{"ft_lang_label":"__label__en","ft_lang_prob":0.8859616,"math_prob":0.982888,"size":1537,"snap":"2021-31-2021-39","text_gpt3_token_len":384,"char_repetition_ratio":0.086105675,"word_repetition_ratio":0.0,"special_character_ratio":0.21665582,"punctuation_ratio":0.101886794,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9958292,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2021-08-04T22:08:57Z\",\"WARC-Record-ID\":\"<urn:uuid:7e0873b7-6e95-4bcd-9fbd-6ccc826fd390>\",\"Content-Length\":\"5995\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:5b8a23d4-ead9-468b-a6c3-52ce41914aeb>\",\"WARC-Concurrent-To\":\"<urn:uuid:9b75aa59-073b-4193-9ad4-d19cbb455c74>\",\"WARC-IP-Address\":\"128.52.131.20\",\"WARC-Target-URI\":\"https://jmlr.org/papers/v20/19-216.html\",\"WARC-Payload-Digest\":\"sha1:FMTS4JMPPZ6BXCQKCOYEVDK32SQXNCVJ\",\"WARC-Block-Digest\":\"sha1:QGSBFLQ43FMGN4XONQEASG5MIUTVG3AP\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2021/CC-MAIN-2021-31/CC-MAIN-2021-31_segments_1627046155188.79_warc_CC-MAIN-20210804205700-20210804235700-00068.warc.gz\"}"}