chunk. Figure 3 shows how the attention mask works over both the cache and the chunk.
godog0000100000thetoThecatsatonthe
1matand111sawthe1000doggoto
100000110000000011100000011110PastCacheCurrent
Figure 3: Pre-fill and chunking. During pre-fill of the cache, long sequences are chunked to limit memory
usage. We process a sequence in three chunks, “The cat sat on”, “the mat and saw”, “the dog go to”. The figure
shows what happens for the third chunk (“the dog go to”): it attends itself using a causal mask (rightmost block),
attends the cache using a sliding window (center block), and does not attend to past tokens as they are outside of
the sliding window (left block).
3 Results
We compare Mistral 7B to Llama, and re-run all benchmarks with our own evaluation pipeline for
fair comparison. We measure performance on a wide variety of tasks categorized as follow:
•Commonsense Reasoning (0-shot): Hellaswag [ 28], Winogrande [ 21], PIQA [ 4], SIQA [ 22],
OpenbookQA [19], ARC-Easy, ARC-Challenge [9], CommonsenseQA [24]
•World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15]