Scale learning rates by dividing with the square root of accumulated squared gradients. See  for further description.

Parameters
loss_or_grads: symbolic expression or list of expressions

A scalar loss expression, or a list of gradient expressions

params: list of shared variables

The variables to generate update expressions for

learning_rate: float or symbolic scalar

The learning rate controlling the size of update steps

epsilon: float or symbolic scalar

Small value added for numerical stability

Returns
OrderedDict

A dictionary mapping each parameter to its update expression

Notes

Using step size eta Adagrad calculates the learning rate for feature i at time step t as:

$\eta_{t,i} = \frac{\eta} {\sqrt{\sum^t_{t^\prime} g^2_{t^\prime,i}+\epsilon}} g_{t,i}$

as such the learning rate is monotonically decreasing.

Epsilon is not included in the typical formula, see .

Optimizer can be called without both loss_or_grads and params in that case partial function is returned

References

1

Duchi, J., Hazan, E., & Singer, Y. (2011): Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121-2159.

2

Examples

>>> a = aesara.shared(1.)
>>> b = a*2
{"ft_lang_label":"__label__en","ft_lang_prob":0.61599606,"math_prob":0.9801387,"size":1621,"snap":"2022-40-2023-06","text_gpt3_token_len":422,"char_repetition_ratio":0.12368584,"word_repetition_ratio":0.0,"special_character_ratio":0.26341763,"punctuation_ratio":0.17123288,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.99648494,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2022-09-27T02:10:42Z\",\"WARC-Record-ID\":\"<urn:uuid:db375583-134c-4e81-8cc8-01f590307417>\",\"Content-Length\":\"69883\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:17f0e1e7-c5bf-4645-999b-eaa0531ab803>\",\"WARC-Concurrent-To\":\"<urn:uuid:3e2359c7-af7a-443a-bb02-c0c1b069422b>\",\"WARC-IP-Address\":\"104.17.33.82\",\"WARC-Target-URI\":\"https://docs.pymc.io/en/latest/api/generated/pymc.adagrad.html\",\"WARC-Payload-Digest\":\"sha1:CGF4P2LMAXPMYPBY4VVPGCEHB6HLEWAO\",\"WARC-Block-Digest\":\"sha1:YBWAJGYRLK2UVJ5YXPY7JICQN3BUGQQI\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2022/CC-MAIN-2022-40/CC-MAIN-2022-40_segments_1664030334974.57_warc_CC-MAIN-20220927002241-20220927032241-00200.warc.gz\"}"}