In PyTorch 1.12, Torchscript is updating its default fuser  (for Volta and later CUDA accelerators) to nvFuser, which supports a wider range of operations and is faster than NNC, the previous fuser for CUDA devices. A soon to be published blog post will elaborate on nvFuser and show how it speeds up training on a variety of networks. 
See the nvFuser documentation for more details on usage and debugging.
Changes to float32 matrix multiplication precision on Ampere and later CUDA hardware