sing a reduced numerical precision (such as float16 or bfloat16) for inference doesn’t affect the generation quality but significantly improves latency. The benefits of using bfloat16 compared to float16 are hardware dependent, but modern GPUs tend to favor bfloat16. bfloat16 is much more resilient when used with quantization compared to float16, but more recent versions of the quantization library (torchao) we used don’t have numerical issues 