parameters of the architecture are summarized in Table 1. Compared
to Llama, it introduces a few changes that we summarize below.
Sliding Window Attention. SWA exploits the stacked layers of a transformer to attend information beyond the window size W. The hidden
state in position iof the layer k,hi, attends to all hidden states from
the previous layer with positions between i−Wandi. Recursively,
hican access tokens from the input layer at a distance of up to W×k
tokens, as illustrated in Figure 1. At the last layer, using a window size
ofW= 4096 , we have a theoretical attention span of approximately
131Ktokens. In practice, for a sequence length of 16K and W= 4096 ,
changes made to FlashAttention [ 11] and xFormers [ 18] yield a 2x
speed improvement over a vanilla attention baseline.
Rolling Buffer Cache. A fixed attention span means that we can limit our cache size using a rolling
buffer cache. The cache has a fixed size of W, and the keys and values for the timestep iare stored
in position imodWof the cache. As a result, when the position iis larger than W, past values
in the cache are overwritten, and the size of the cache stops increasing. We provide an illustration
in Figure 2 for W= 3. On a sequence length of 32k tokens, this reduces the cache memory usage