Mistral 7B takes a significant step in balancing the goals of getting high performance while keeping
large language models efficient. Through our work, our aim is to help the community create more
affordable, efficient, and high-performing language models that can be used in a wide range of
real-world applications.
2 Architectural details
Figure 1: Sliding Window Attention. The number of operations in vanilla attention is quadratic in the sequence
length, and the memory increases linearly with the number of tokens. At inference time, this incurs higher
latency and smaller throughput due to reduced cache availability. To alleviate this issue, we use sliding window
attention: each token can attend to at most Wtokens from the previous layer (here, W= 3). Note that tokens
outside the sliding window still influence next word prediction. At each attention layer, information can move
forward by Wtokens. Hence, after kattention layers, information can move forward by up to k×Wtokens.
Parameter Value
dim 4096
n_layers 32
head_dim 128
hidden_dim 14336
n_heads 32
n_kv_heads 8
window_size 4096
context_len 8192
vocab_size 32000
Table 1: Model architecture.Mistral 7B is based on a transformer architecture [ 27]. The main
parameters of the architecture are summarized in Table 1. Compared
to Llama, it introduces a few changes that we summarize below.