which is likely due to its limited parameter count that restricts the amount of knowledge it can store.
Evaluation Differences. On some benchmarks, there are some differences between our evaluation
protocol and the one reported in the Llama 2 paper: 1) on MBPP, we use the hand-verified subset 2)
on TriviaQA, we do not provide Wikipedia contexts.
4 Instruction Finetuning
ModelChatbot Arena
ELO RatingMT Bench
WizardLM 13B v1.2 1047 7.2
Mistral 7B Instruct 1031 6.84 +/- 0.07
Llama 2 13B Chat 1012 6.65
Vicuna 13B 1041 6.57
Llama 2 7B Chat 985 6.27
Vicuna 7B 997 6.17
Alpaca 13B 914 4.53
Table 3: Comparison of Chat models. Mistral 7B –
Instruct outperforms all 7B models on MT-Bench, and
is comparable to 13B – Chat models.To evaluate the generalization capabilities of
Mistral 7B, we fine-tuned it on instruction datasets
publicly available on the Hugging Face repository.
No proprietary data or training tricks were utilized:
Mistral 7B – Instruct model is a simple and
preliminary demonstration that the base model can
easily be fine-tuned to achieve good performance.
In Table 3, we observe that the resulting model,