How to Smooth Gradient Estimates for Steepest Descent Optimization
In the Probabilistic settings we have many methods applied to the Stochastic Gradient Descent in order to decrease the variance of the Gradient Estimation (ADAM / RMS Prop / AdaDelta, etc...).

The nice thing is to utilize them in deterministic settings.
So for instance you can use Momentum which to Signal Processing guy will look just like applying IIR / AR filter on the Gradient.  

There are also methods derived from the Convex Optimization world settings.
There is nothing prevents us using them in any setting besides there are no guarantees.
For instance you can use FISTA Method or Nesterov Accelerated Gradient Descent.

More References:


Deniz Yuret's Homepage  - Alec Radford's animations for optimization algorithms.
Sebastian Ruder - An overview of gradient descent optimization algorithms.
Shao-An Lu - SGD > Adam?? Which One Is The Best Optimizer: Dogs-VS-Cats Toy Experiment.
FAST AI (Sylvain Gugger and Jeremy Howard) - AdamW and Super-convergence is now the fastest way to train neural nets.