nds:  Why bfloat16? Using a reduced numerical precision (such as float16, bfloat16) to run inference doesn’t affect the generation quality but significantly improves latency. The benefits of using the bfloat16 numerical precision as compared to float16 are hardware-dependent. Modern generations of GPUs tend to favor bfloat16. Furthermore, in our experiments, we bfloat16 to be much more resilient when used with quantization in comparison to floa