Usage of --apply_chat_template in lm_eval benchmarks
#1
by
VlSav
- opened
Hello!
Could you please clary why in several lm_eval benchmark " --apply_chat_template" parameter was used, but in others it was not utilized?
(according to command lines of lm_eval execution in readme.md).
We used --apply_chat_template to replicate the setup used by Meta in the official release of Llama-3.1 models. For benchmarks not reported by Meta we use the standard setup as the OpenLLM leaderboard.
alexmarques
changed discussion status to
closed