


BERT/Transformer
Dynamic Quantization
Throughput dominated by compute/memory bandwidth for weights


CNN
Static Quantization
Throughput limited by memory bandwidth for activations


CNN
Quantization Aware Training
In the case where accuracy can't be achieved with static quantization


Performance Results
Quantization provides a 4x reduction in the model size and a speedup of 2x to 3x compared to floating point implementations depending on the hardware platform and the model being benchmarked. Some sample results are:



Model
Float Latency (ms)