automated benchmarks. Our models are released under the Apache 2.0 license.
Code: https://github.com/mistralai/mistral-src
Webpage: https://mistral.ai/news/announcing-mistral-7b/
1 Introduction
In the rapidly evolving domain of Natural Language Processing (NLP), the race towards higher model
performance often necessitates an escalation in model size. However, this scaling tends to increase
computational costs and inference latency, thereby raising barriers to deployment in practical,
real-world scenarios. In this context, the search for balanced models delivering both high-level
performance and efficiency becomes critically essential. Our model, Mistral 7B, demonstrates that
a carefully designed language model can deliver high performance while maintaining an efficient
inference. Mistral 7B outperforms the previous best 13B model (Llama 2, [ 26]) across all tested
benchmarks, and surpasses the best 34B model (LLaMa 34B, [ 25]) in mathematics and code
generation. Furthermore, Mistral 7B approaches the coding performance of Code-Llama 7B [ 20],
without sacrificing performance on non-code related benchmarks.
Mistral 7B leverages grouped-query attention (GQA) [ 1], and sliding window attention (SWA) [ 6,3].
GQA significantly accelerates the inference speed, and also reduces the memory requirement during