•World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15]
•Reading Comprehension (0-shot): BoolQ [8], QuAC [7]
•Math: GSM8K [10] (8-shot) with maj@8 and MATH [13] (4-shot) with maj@4
•Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot)
•Popular aggregated results: MMLU [ 12] (5-shot), BBH [ 23] (3-shot), and AGI Eval [ 29]
(3-5-shot, English multiple-choice questions only)
Detailed results for Mistral 7B, Llama 2 7B/13B, and Code-Llama 7B are reported in Table 2. Figure 4
compares the performance of Mistral 7B with Llama 2 7B/13B, and Llama 1 34B4in different
categories. Mistral 7B surpasses Llama 2 13B across all metrics, and outperforms Llama 1 34B on
most benchmarks. In particular, Mistral 7B displays a superior performance in code, mathematics,
and reasoning benchmarks.
4Since Llama 2 34B was not open-sourced, we report results for Llama 1 34B.
3