Lin-K76 commited on
Commit
4862bb4
·
verified ·
1 Parent(s): d3479a2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -2
README.md CHANGED
@@ -2,6 +2,9 @@
2
  tags:
3
  - fp8
4
  - vllm
 
 
 
5
  ---
6
 
7
  # DeepSeek-Coder-V2-Lite-Instruct-FP8
@@ -19,13 +22,13 @@ tags:
19
  - **Version:** 1.0
20
  - **Model Developers:** Neural Magic
21
 
22
- Quantized version of [DeepSeek-Coder-V2-Lite-Instruct](deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct).
23
  <!-- It achieves an average score of 73.19 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 73.48. -->
24
  It achieves an average score of 79.60 on the [HumanEval+](https://github.com/openai/human-eval?tab=readme-ov-file) benchmark, whereas the unquantized model achieves 79.33.
25
 
26
  ### Model Optimizations
27
 
28
- This model was obtained by quantizing the weights and activations of [DeepSeek-Coder-V2-Lite-Instruct](deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct) to FP8 data type, ready for inference with vLLM >= 0.5.2.
29
  This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
30
 
31
  Only the weights and activations of the linear operators within transformers blocks are quantized. Symmetric per-tensor quantization is applied, in which a linear scaling per output dimension maps the FP8 representations of the quantized weights and activations.
 
2
  tags:
3
  - fp8
4
  - vllm
5
+ license: other
6
+ license_name: deepseek-license
7
+ license_link: LICENSE
8
  ---
9
 
10
  # DeepSeek-Coder-V2-Lite-Instruct-FP8
 
22
  - **Version:** 1.0
23
  - **Model Developers:** Neural Magic
24
 
25
+ Quantized version of [DeepSeek-Coder-V2-Lite-Instruct](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct).
26
  <!-- It achieves an average score of 73.19 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 73.48. -->
27
  It achieves an average score of 79.60 on the [HumanEval+](https://github.com/openai/human-eval?tab=readme-ov-file) benchmark, whereas the unquantized model achieves 79.33.
28
 
29
  ### Model Optimizations
30
 
31
+ This model was obtained by quantizing the weights and activations of [DeepSeek-Coder-V2-Lite-Instruct](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct) to FP8 data type, ready for inference with vLLM >= 0.5.2.
32
  This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
33
 
34
  Only the weights and activations of the linear operators within transformers blocks are quantized. Symmetric per-tensor quantization is applied, in which a linear scaling per output dimension maps the FP8 representations of the quantized weights and activations.