## Deep Sparse Rectifier Neural Networks

Glorot, X., Bordes, A., & Bengio, Y. (2011). Deep sparse rectifier neural networks. AISTATS ’11: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, 15, 315–323. https://doi.org/10.1.1.208.6449

• The rectifier function $\max(0,x)$ is both a useful model for neuron activation in neuroscience and an efficient activation function in neural networks.
• Biological neurons can have activations that are asymmetric ($1 \rightarrow -1$), symmetric ($1 \rightarrow 1$), or one-sided ($1 \rightarrow 0$)
• Neurons encode information in a sparse manner: only 1-4% are active at once. This is a tradeoff between expressiveness and low energy use. Neural nets without $\ell_1$ regularisation do not have this property.
• Biological neurons use very different activations to NNs:
• their firing rate is zero for all current less than zero, then increases sub-linearly.
• artificial neurons use sigmoid or tanh activations, which are both asymmetrical about zero (not present in biology)
• tanh is preferred to sigmoid because its steady state is zero.

### Consequences of sparsity

• Information disentangling – the networks will be robust to small input changes
• Efficient variable-size representation – inputs can be in a variable-sized data structure.
• More likely to be linearly separable
• Excess sparsity may reduce predictive capability.

### Rectifier neurons

• Because real neurons rarely reach their saturation (where increases in current no longer affect firing rate), they can be well approximated with the rectifier function.
• A rectifier automatically produces sparsity, 50% of activations will be initialised to zero.
• Much easier to compute than tanh or sigmoid.
• If one is worried about differentiability, the softplus function $f(x) = \ln(1+e^x)$ can be used. However, experiments suggest that the hard zeroes are good for NN performance.
• Rectifier nets need more hidden units to represent any antisymmetry in data.

### Experimental results

• For image classification, sparsity does not hurt performance until 85% of neurons are zeroes.
• Rectifiers outperform softplus activations
• No improvement using pretraining autoencoders – easier to use.
• Rectifiers work very well with supervised and semi-supervised problems, but in the latter case pretraining is needed.
• Very strong performance on text sentiment analysis: lower RMSE than tanh at 50% sparsity.
{"ft_lang_label":"__label__en","ft_lang_prob":0.7971432,"math_prob":0.80709785,"size":2436,"snap":"2020-10-2020-16","text_gpt3_token_len":545,"char_repetition_ratio":0.10978618,"word_repetition_ratio":0.0,"special_character_ratio":0.2224959,"punctuation_ratio":0.1262136,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.95032996,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2020-02-26T00:59:36Z\",\"WARC-Record-ID\":\"<urn:uuid:2f3c26d4-ab88-4708-8bde-9b3542bc489d>\",\"Content-Length\":\"9408\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:52776b73-f456-4cd0-8564-6c201c59d4b9>\",\"WARC-Concurrent-To\":\"<urn:uuid:4f4c938f-1730-4669-9d33-031d731ca7ba>\",\"WARC-IP-Address\":\"185.199.109.153\",\"WARC-Target-URI\":\"https://reasonabledeviations.com/notes/papers/deep_sparse_nn/\",\"WARC-Payload-Digest\":\"sha1:I5INHMTGSXVIGRV3AK7PEKEILE7TQOJX\",\"WARC-Block-Digest\":\"sha1:ZKX5HAUTAQ2QLI74OJKVRDSYGKLIRR6G\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2020/CC-MAIN-2020-10/CC-MAIN-2020-10_segments_1581875146176.73_warc_CC-MAIN-20200225233214-20200226023214-00517.warc.gz\"}"}