Spaces:
Runtime error
Runtime error
File size: 1,962 Bytes
8a42f8f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
.. role:: hidden
:class: hidden-section
apex.fp16_utils
===================================
This submodule contains utilities designed to streamline the mixed precision training recipe
presented by NVIDIA `on Parallel Forall`_ and in GTC 2018 Sessions
`Training Neural Networks with Mixed Precision: Theory and Practice`_ and
`Training Neural Networks with Mixed Precision: Real Examples`_.
For Pytorch users, Real Examples in particular is recommended.
Full runnable Python scripts demonstrating ``apex.fp16_utils``
can be found on the Github page:
| `Simple FP16_Optimizer demos`_
|
| `Distributed Mixed Precision Training with imagenet`_
|
| `Mixed Precision Training with word_language_model`_
|
|
.. _`on Parallel Forall`:
https://devblogs.nvidia.com/mixed-precision-training-deep-neural-networks/
.. _`Training Neural Networks with Mixed Precision: Theory and Practice`:
http://on-demand.gputechconf.com/gtc/2018/video/S8923/
.. _`Training Neural Networks with Mixed Precision: Real Examples`:
http://on-demand.gputechconf.com/gtc/2018/video/S81012/
.. _`Simple FP16_Optimizer demos`:
https://github.com/NVIDIA/apex/tree/master/examples/FP16_Optimizer_simple
.. _`Distributed Mixed Precision Training with imagenet`:
https://github.com/NVIDIA/apex/tree/master/examples/imagenet
.. _`Mixed Precision Training with word_language_model`:
https://github.com/NVIDIA/apex/tree/master/examples/word_language_model
.. automodule:: apex.fp16_utils
.. currentmodule:: apex.fp16_utils
Automatic management of master params + loss scaling
----------------------------------------------------
.. autoclass:: FP16_Optimizer
:members:
.. autoclass:: LossScaler
:members:
.. autoclass:: DynamicLossScaler
:members:
Manual master parameter management
----------------------------------
.. autofunction:: prep_param_lists
.. autofunction:: master_params_to_model_params
.. autofunction:: model_grads_to_master_grads
|