Code-Llama 7B Finetuned 36.9% 62.9% 62.3% 72.8% 59.4% 34.5% 11.0% 34.9% 31.1% 52.5% 5.2% 20.8%
Mistral 7B Pretrained 60.1% 81.3% 75.3% 83.0% 80.0% 55.5% 28.8% 69.9% 30.5% 47.5% 13.1% 52.2%
Table 2: Comparison of Mistral 7B with Llama. Mistral 7B outperforms Llama 2 13B on all metrics, and
approaches the code performance of Code-Llama 7B without sacrificing performance on non-code benchmarks.
Size and Efficiency. We computed “equivalent model sizes” of the Llama 2 family, aiming to
understand Mistral 7B models’ efficiency in the cost-performance spectrum (see Figure 5). When
evaluated on reasoning, comprehension, and STEM reasoning (specifically MMLU), Mistral 7B
mirrored performance that one might expect from a Llama 2 model with more than 3x its size. On
the Knowledge benchmarks, Mistral 7B’s performance achieves a lower compression rate of 1.9x,
which is likely due to its limited parameter count that restricts the amount of knowledge it can store.