Update README.md
Browse files
README.md
CHANGED
|
@@ -19,6 +19,7 @@ pipeline_tag: text-generation
|
|
| 19 |
This model was obtained by quantizing the weights and activations of [Bielik-11B-v.2.2-Instruct](https://huggingface.co/speakleash/Bielik-11B-v2.2-Instruct) to FP8 data type, ready for inference with vLLM >= 0.5.0.
|
| 20 |
AutoFP8 is used for quantization. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
|
| 21 |
Only the weights and activations of the linear operators within transformers blocks are quantized. Symmetric per-tensor quantization is applied, in which a single linear scaling maps the FP8 representations of the quantized weights and activations.
|
|
|
|
| 22 |
FP8 compuation is supported on Nvidia GPUs with compute capability > 8.9 (Ada Lovelace, Hopper).
|
| 23 |
|
| 24 |
|
|
|
|
| 19 |
This model was obtained by quantizing the weights and activations of [Bielik-11B-v.2.2-Instruct](https://huggingface.co/speakleash/Bielik-11B-v2.2-Instruct) to FP8 data type, ready for inference with vLLM >= 0.5.0.
|
| 20 |
AutoFP8 is used for quantization. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
|
| 21 |
Only the weights and activations of the linear operators within transformers blocks are quantized. Symmetric per-tensor quantization is applied, in which a single linear scaling maps the FP8 representations of the quantized weights and activations.
|
| 22 |
+
|
| 23 |
FP8 compuation is supported on Nvidia GPUs with compute capability > 8.9 (Ada Lovelace, Hopper).
|
| 24 |
|
| 25 |
|