# Momentum

A technique used when minimizing a function using gradient descent that helps avoid local minima.

One of the biggest problems when using gradient descent to minimize a function is that the optimization process may get stuck in a local minimum. Momentum is a technique that tries to mitigate that by allowing the optimization to "gather speed" and jump over small "hills" in the function.

## Detailed formula explanation

The formulat for momentum used in a gradient descent optimization can be written as: The formula is based on a simple iterative optimization algorithm, that can be expressed as: In a simple gradient descent optimization, we define the update of the weights as: If we want to add momentum, we add an additional term - the weight update in the previous step times a decay factor. The decay factor `α` is just a number between 0 and 1 defining how much of the previous update will be taken into account. If we set `α` = 0 then no momentum is applied, while `α` = 1 means a lot of momentum. A useful analogy is a ball rolling down a hill. If the hill is steep, the ball will accelerate (we update the weights more). This will help the ball jump over small local minima and continue down the hill (to a smaller value of the function). More momentum means a heavier ball with higher inertia. The final formula of gradient descent optimization with momentum becomes: Back to Math Terms
{"ft_lang_label":"__label__en","ft_lang_prob":0.8452345,"math_prob":0.94019395,"size":1341,"snap":"2023-14-2023-23","text_gpt3_token_len":275,"char_repetition_ratio":0.13388182,"word_repetition_ratio":0.0,"special_character_ratio":0.20283371,"punctuation_ratio":0.07450981,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9941766,"pos_list":[0,1,2,3,4,5,6,7,8,9,10,11,12],"im_url_duplicate_count":[null,1,null,2,null,2,null,1,null,1,null,1,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2023-03-27T19:21:08Z\",\"WARC-Record-ID\":\"<urn:uuid:9a2617ca-ae9f-416a-90e7-d0345f47959d>\",\"Content-Length\":\"10944\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:346980ca-fa01-48af-b12e-37cf619cf6bc>\",\"WARC-Concurrent-To\":\"<urn:uuid:1e7d5073-7119-495e-94ad-18d316abf273>\",\"WARC-IP-Address\":\"34.148.79.160\",\"WARC-Target-URI\":\"https://suboptimal.wiki/explanation/momentum/\",\"WARC-Payload-Digest\":\"sha1:QZJM6UZIPS6PB43B6KYTNER4WWFQ3J5B\",\"WARC-Block-Digest\":\"sha1:LUWQ4ZPROSYQWENM3VIRVN7AYJ2Q6OMQ\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2023/CC-MAIN-2023-14/CC-MAIN-2023-14_segments_1679296948684.19_warc_CC-MAIN-20230327185741-20230327215741-00062.warc.gz\"}"}