are different from the input). By default, TF32 tensor cores are
disabled for matrix multiplications and enabled for convolutions,
although most neural network workloads have the same convergence
behavior when using TF32 as they have with fp32. We recommend enabling
TF32 tensor cores for matrix multiplications with
"torch.backends.cuda.matmul.allow_tf32 = True" if your network does
not need full float32 precision. If your network needs full float32
precision for both matrix multiplications and convolutions, then TF32
tensor cores can also be disabled for convolutions with
"torch.backends.cudnn.allow_tf32 = False".
For more information see TensorFloat32.
Reduced Precision Reduction for FP16  and BF16 GEMMs
Half-precision GEMM operations are typically done with intermediate
accumulations (reduction) in single-precision for numerical accuracy
and improved resilience to overflow. For performance, certain GPU