nable cuDNN autotuner before launching the inference by setting:


	Copied
import torch

torch.backends.cudnn.benchmark = True

Use tf32 instead of fp32 (on Ampere and later CUDA devices)
	
On Ampere and later CUDA devices matrix multiplications and convolutions can use the TensorFloat32 (TF32) mode for faster but slightly less accurate computations. By default PyTorch enables TF32 mode for convolutions but not matrix multiplications, and unles