INT4 quantization is a technique used in AI inference that reduces both the memory cost and latency while preserving model accuracy. It is a type of quantization that reduces the number of bits used to represent the weights and activations of a neural network model to 4 bits.

This technique has been shown to be effective in reducing the memory cost and latency of transformer models, which are commonly used in natural language processing tasks.

To generate an INT4 model, the original model is fine-tuned and a quantized INT4 model can be generated using the range data from the "fake" quantization layers.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence