

Exponential Moving Average (EMA)
EMA is a technique that allows one to push the accuracy of a model without increasing its complexity or inference time. It performs an exponential moving average on the model weights and this leads to increased accuracy and more stable models. The averaging happens every few iterations and its decay parameter was tuned via grid search. 
Below you can see the optimal values for our recipe:
model_ema=True, 
model_ema_steps=32, 
model_ema_decay=0.99998,
