yintongl commited on
Commit
7e8a069
·
verified ·
1 Parent(s): 4dcf92b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -13
README.md CHANGED
@@ -56,20 +56,19 @@ Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.gi
56
  lm_eval --model hf --model_args pretrained="Intel/gemma-7b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
57
  ```
58
 
59
- | Metric | FP16 | int4 |
60
  | -------------- | ------ | ------ |
61
- | Avg. | 0.6239 | 0.6307 |
62
- | mmlu | 0.6162 | 0.6147 |
63
- | lambada_openai | 0.6751 | 0.7204 |
64
- | hellaswag | 0.6047 | 0.5903 |
65
- | winogrande | 0.7324 | 0.7514 |
66
- | piqa | 0.7943 | 0.7949 |
67
- | truthfulqa_mc1 | 0.3097 | 0.3011 |
68
- | openbookqa | 0.3320 | 0.3400 |
69
- | boolq | 0.8278 | 0.8269 |
70
- | rte | 0.6534 | 0.7076 |
71
- | arc_easy | 0.8178 | 0.7959 |
72
- | arc_challenge | 0.4991 | 0.4940 |
73
 
74
 
75
 
 
56
  lm_eval --model hf --model_args pretrained="Intel/gemma-7b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
57
  ```
58
 
59
+ | Metric | BF16 | int4 |
60
  | -------------- | ------ | ------ |
61
+ | Avg. | 0.6302 | 0.6262 |
62
+ | mmlu | 0.6199 | 0.6111 |
63
+ | lambada_openai | 0.7316 | 0.7252 |
64
+ | hellaswag | 0.6059 | 0.6033 |
65
+ | winogrande | 0.7506 | 0.7451 |
66
+ | piqa | 0.8014 | 0.8058 |
67
+ | truthfulqa_mc1 | 0.3121 | 0.2889 |
68
+ | openbookqa | 0.3220 | 0.3380 |
69
+ | boolq | 0.8339 | 0.8281 |
70
+ | arc_easy | 0.8253 | 0.8152 |
71
+ | arc_challenge | 0.4991 | 0.5017 |
 
72
 
73
 
74