are unchanged.
When training using FP16 precision, some models may fail to converge
with FP16 denorms flushed to zero. Denormal values more frequently
occur in the backward pass of training during gradient calculation.
PyTorch by default will use the rocBLAS and MIOpen alternate
implementations during the backward pass. The default behavior can be
overridden using environment variables, ROCBLAS_INTERNAL_FP16_ALT_IMPL
and MIOPEN_DEBUG_CONVOLUTION_ATTRIB_FP16_ALT_IMPL. The behavior of
these environment variables is as follows:
+-----------------+-------------+-------------+
|                 | forward     | backward    |
|=================|=============|=============|
| Env unset       | original    | alternate   |
+-----------------+-------------+-------------+
| Env set to 1    | alternate   | alternate   |
+-----------------+-------------+-------------+
| Env set to 0    | original    | original    |
+-----------------+-------------+-------------+