Activation Functions

# Rectified Linear Units

Rectified Linear Units, or ReLUs, are a type of activation function that are linear in the positive dimension, but zero in the negative dimension. The kink in the function is the source of the non-linearity. Linearity in the positive dimension has the attractive property that it prevents non-saturation of gradients (contrast with sigmoid activations), although for half of the real line its gradient is zero.

$$f\left(x\right) = \max\left(0, x\right)$$

#### Papers

Paper Code Results Date Stars
{"ft_lang_label":"__label__en","ft_lang_prob":0.73849386,"math_prob":0.9822495,"size":898,"snap":"2023-40-2023-50","text_gpt3_token_len":280,"char_repetition_ratio":0.11744966,"word_repetition_ratio":0.0,"special_character_ratio":0.2973274,"punctuation_ratio":0.11904762,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.97714394,"pos_list":[0,1,2],"im_url_duplicate_count":[null,null,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2023-09-27T08:16:01Z\",\"WARC-Record-ID\":\"<urn:uuid:709e310d-23fb-4024-a8f2-b62250442861>\",\"Content-Length\":\"69239\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:d72dee40-ddf1-4ab6-8c3a-88105af1b1e5>\",\"WARC-Concurrent-To\":\"<urn:uuid:973186ce-38e7-43da-86ad-85734dd03d62>\",\"WARC-IP-Address\":\"104.26.12.155\",\"WARC-Target-URI\":\"https://paperswithcode.com/method/relu\",\"WARC-Payload-Digest\":\"sha1:TJ3M3YJVOTQFC4VN6QPJ2VANKOVTNYG3\",\"WARC-Block-Digest\":\"sha1:HSFXU34B43VRGYG347OU3JF5QSRQCOCR\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2023/CC-MAIN-2023-40/CC-MAIN-2023-40_segments_1695233510284.49_warc_CC-MAIN-20230927071345-20230927101345-00264.warc.gz\"}"}