Cannot reproduce accuracy of mncai/Llama2-7B-guanaco-dolphin-500 gsm8k
#2
by
zhentaocc
- opened
with batch size = 1, the result I got was 13.12, while the reported is 5.99
I was using python main.py --model=hf-causal-experimental --model_args="pretrained=mncai/Llama2-7B-guanaco-dolphin-500 gsm8k" --tasks=gsm8k --num_fewshot=5 --batch_size=1 --no_cache
And I found different settings for batch size result in different accuracy.