or skips optimizer.step()
scaler.step(optimizer)
# Updates the scale for next iteration
   scaler.update()
```

 Figure 4: AMP recipe

Picking The Right Approach
Out-of-the-box mixed precision training with either float16 or bfloat16 is effective at speeding up the convergence of many deep learning models, but some models may require more careful numerical accuracy management. Here are some options:

Full float32 precision. Floating point tensors and modules are created in float32 precision by default in PyTorch, but this is a historic artifact not representative of training most modern deep learning networks. It’s rare that networks need this much numerical accuracy.
