RedHatAI
/

Qwen2.5-VL-3B-Instruct-quantized.w8a8

Image-Text-to-Text

text-generation-inference

8-bit precision

compressed-tensors

Model card Files Files and versions Community

shubhrapandit commited on 9 days ago

Commit

c21b685

·

verified ·

1 Parent(s): 8c3aab4

Update README.md

Files changed (1) hide show

README.md +1 -0

README.md CHANGED Viewed

@@ -226,6 +226,7 @@ lm_eval \
   --model vllm \
   --model_args pretrained="<model_name>",dtype=auto,max_model_len=4096,max_gen_toks=2048,max_num_seqs=128,tensor_parallel_size=<n>,gpu_memory_utilization=0.9 \
   --tasks mgsm_cot_native \
   --num_fewshot 0 \
   --batch_size auto \
   --output_path output_dir

   --model vllm \
   --model_args pretrained="<model_name>",dtype=auto,max_model_len=4096,max_gen_toks=2048,max_num_seqs=128,tensor_parallel_size=<n>,gpu_memory_utilization=0.9 \
   --tasks mgsm_cot_native \
+  --apply_chat_template \
   --num_fewshot 0 \
   --batch_size auto \
   --output_path output_dir