
4x reduction in model size;
2-4x reduction in memory bandwidth;
2-4x faster inference due to savings in memory bandwidth and faster compute with int8 arithmetic (the exact speed up varies depending on the hardware, the runtime, and the model).

Quantization does not however come without additional cost. Fundamentally quantization means introducing approximations and the resulting networks have slightly less accuracy. These techniques attempt to minimize the gap between the full floating point accuracy and the quantized accuracy.
We designed quantization to fit into the PyTorch framework. The means that:
1. PyTorch has data types corresponding to quantized tensors, which share many of the features of tensors.