in Figure 2 for W= 3. On a sequence length of 32k tokens, this reduces the cache memory usage
by 8x, without impacting the model quality.
1https://github.com/mistralai/mistral-src
2https://github.com/skypilot-org/skypilot
3https://huggingface.co/mistralai
2
Figure 2: Rolling buffer cache. The cache has a fixed size of W= 4. Keys and values for position iare stored
in position imod Wof the cache. When the position iis larger than W, past values in the cache are overwritten.
The hidden state corresponding to the latest generated tokens are colored in orange.
Pre-fill and Chunking. When generating a sequence, we need to predict tokens one-by-one, as
each token is conditioned on the previous ones. However, the prompt is known in advance, and we
can pre-fill the ( k,v) cache with the prompt. If the prompt is very large, we can chunk it into smaller
pieces, and pre-fill the cache with each chunk. For this purpose, we can select the window size as
our chunk size. For each chunk, we thus need to compute the attention over the cache and over the
chunk. Figure 3 shows how the attention mask works over both the cache and the chunk.
godog0000100000thetoThecatsatonthe