RedHatAI
/

Qwen2-0.5B-Instruct-quantized.w4a16

Text Generation

text-generation-inference

4-bit precision

Model card Files Files and versions

abhinavnmagic commited on Jul 16, 2024

Commit

8191cf6

·

verified ·

1 Parent(s): 4f7c556

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -144,7 +144,7 @@ quantize_config = BaseQuantizeConfig(
   group_size=128,
   desc_act=True,
   model_file_base_name="model",
-  damp_percent=0.1,
 )
 model = AutoGPTQForCausalLM.from_pretrained(

   group_size=128,
   desc_act=True,
   model_file_base_name="model",
+  damp_percent=0.01,
 )
 model = AutoGPTQForCausalLM.from_pretrained(