So awesome, having the option to cache q.T@q is a great feature, sacrificing some memory for a solid speed-up. What Lucas is doing with heavyball is an absolute boon to open-source optimizer research.

I hope to match heavyball’s utility for JAX users when I release distributed PSGD Kron in a few days. It should be, to the best of my knowledge, the most performant open-source second-order optimizer for JAX!