neuralmagic
/

DeepSeek-R1-Distill-Llama-70B-quantized.w4a16

Text Generation

text-generation-inference

compressed-tensors

Model card Files Files and versions Community

Update README.md

#1

by nm-research - opened Feb 20

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -289,7 +289,7 @@ lm_eval \
 ## Inference Performance
-This model achieves up to 2.7x speedup in single-stream deployment and up to 1.5x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario.
 The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.6.7.2, and [GuideLLM](https://github.com/neuralmagic/guidellm).
 <details>

 ## Inference Performance
+This model achieves up to 3.0x speedup in single-stream deployment and up to 2.6x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario.
 The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.6.7.2, and [GuideLLM](https://github.com/neuralmagic/guidellm).
 <details>