DAPE V2: Process Attention Score as Feature Map for Length Extrapolation
Transformers;data-adaptive positional encoding;long context;length generalization
**Q9:Could the authors elaborate on "Proposition 1: Transformers incorporating convolution operations can perform associative recall tasks without the need for positional encoding"? The rationale behind this proposition is unclear and requires further explanation.**

A9 Associative recall is one of the key capabilities of transformer models in handling language tasks. Previous research demonstrates that transformers achieve associative recall through positional embeddings. Intuitively, transformers can perform copy tasks because information about previous tokens is passed to subsequent tokens. For example, in the sequence {a, b, c, d, e, f, a}, the transformer can output "b" as the next token after the second "a" because the initial occurrence of "a" has been "copied" to "b." This allows the second "a" to attend to "b" and predict it as the next token. The mechanism enabling this copying is positional embedding. Most positional embeddings decay with respect to relative position, meaning attention is concentrated on neighboring tokens. This implicit associative recall mechanism has been theoretically verified and is learned during training [2]. However, a similar mechanism can be realized more directly and efficiently using convolution, eliminating the need for implicit learning through positional encodings (Proposition 1). The proof sketch is provided in lines 265-275.

Reference:

[1] Fang, L., Wang, Y., Liu, Z., Zhang, C., Jegelka, S., Gao, J., ... & Wang, Y. (2024). What is Wrong with Perplexity for Long-context Language Modeling?. arXiv preprint arXiv:2410.23771.

[2] Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36, 2024.

If there are any questions, please let us know. And if you think that we have addressed your concerns, could you please consider raising the score? Thank you very much for your support.
Response to Reviewer UyHR (Part 5/5)
**Q8: The benchmarks employed in the study are limited, reducing the generalizability of the findings.**

A8: **Our experiment setting follows the DAPE (NeurIPS 2024). Besides the experiment restuls on Arxiv and Books dataset, we also have the experiments on 14 downstream tasks which is evaluated via accuracy matrics.**

| **Level** | **Task**                              | **Baseline**|  **Baseline**|  **Baseline**| **Baseline** | **Baseline** | **DAPE (Kernel Size 1)**| **DAPE (Kernel Size 1)** | **DAPE (Kernel Size 1)** | **DAPE (Kernel Size 3)** | **DAPE (Kernel Size 3)** | **DAPE (Kernel Size 3)**|
|-----------|---------------------------------------|----------|--------------|-----------|------------|----------|-------------------------------|------------|----------|-------------------------------|------------|----------|
|  |                               | **RoPE** | **Relative** | **ALiBi** | **Kerple** | **FIRE** | **ALiBi** | **Kerple** | **FIRE** | **ALiBi** | **Kerple** | **FIRE** |
| **R**     | **Even Pairs**                        | 99.98    | 96.60        | 73.52     | 57.50      | 73.86    | 99.99                          | 99.58      | **100**      | 99.99                          | **100**       | **100**      |
|           | **Modular Arithmetic Simple**         | 21.35    | 20.84        | 20.02     | 21.79      | 21.09    | 23.58                          | **24.47**      | 24.46    | 21.48                          | 23.90      | 23.43    |
|           | **Parity Check** †††                 | 50.05    | 50.09        | 50.09     | 50.07      | 50.97    | 50.30                          | 50.07      | 50.04    | 50.13                          | **52.51**      | 50.11    |
|           | **Cycle Navigation** †††             | 27.63    | 26.95        | 24.64     | 29.47      | 28.41    | 22.99                          | **34.53**      | 27.54    | 24.43                          | 24.32      | 24.34    |
| **DCF**   | **Stack Manipulation**                | 61.49    | 64.73        | 66.42     | 66.93      | 69.33    | 68.18                          | **72.04**      | 70.90    | 58.90                          | 68.18      | 60.90    |
|           | **Reverse String**                    | 65.23    | 65.59        | 71.09     | 71.54      | 65.89    | 73.37                          | 70.74      | 76.40    | 56.61                          | **81.84**      | 70.11    |
|           | **Modular Arithmetic Brackets**      | 31.25    | 31.74        | 30.56     | 24.79      | 30.92    | 31.34                          | **32.37**      | 31.50    | 29.46                          | 26.13      | 27.00    |
|           | **Solve Equation**                    | 21.85    | 22.93        | 19.92     | 21.15      | 22.06    | 20.03                          | 22.49      | 22.42    | 20.26                          | **23.95**      | 23.62    |
| **CS**    | **Duplicate String**                  | 64.97    | 67.66        | 65.13     | 66.72      | 69.03    | 70.84                          | **72.95**      | 72.71    | 52.96                          | 57.03      | 66.01    |
|           | **Missing Duplicate**                 | 63.37    | 72.34        | 74.21     | 79.06      | 79.27    | 83.41                          | 87.57      | 89.17    | 59.33                          | **99.65**      | 74.83    |
|           | **Odds First**                        | 61.00    | 61.57        | 59.88     | 62.59      | 63.28    | 63.78                          | **67.08**      | 66.34    | 57.35                          | 56.87      | 56.57    |
|           | **Binary Addition**                   | 55.59    | 56.96        | 54.72     | 56.35      | 55.70    | 59.71                          | **60.88**      | 56.62    | 57.49                          | 55.32      | 57.86    |
|           | **Compute Sqrt**                      | 51.88    | 51.63        | 50.63     | 51.11      | 50.80    | 51.64                          | 51.33      | **52.46**    | 52.08                          | 51.76      | 51.93    |
|           | **Bucket Sort** †††                   | 98.12    | 99.31        | 98.45     | 99.38      | **99.57**    | 99.38                          | 98.81      | 99.37    | 96.61                          | 99.06      | 98.56    |


**Different tasks have different optimal kernel sizes** For example, on Missing Duplicate task, the  $DAPE_{1x3}$-Kerple improves the 87.57 of DAPE-Kerple to 99.65. However, on the Stack Manipulation task, the  $DAPE_{1x3}$-Kerple decreases the 72.04 of DAPE-Kerple to 68.18. the larger kernel size does not always lead to better performance. Overall, larger kernel size provides a potential way to improve the Transformer length extrapolation performance, and we usually could find a suitable kernel size (ranging from 1×1 to larger
kernel sizes) to achieve better performance than without further processing attention score.
Response to Reviewer UyHR (Part 4/5)
* When $ \Delta P = 0 $: The model’s information gain from the full sequence is negligible, indicating an entropy level comparable to local attention (e.g., models like ALiBi when the evaluation length is 1024). This suggests the model does not leverage context beyond a limited range.

* When $\Delta P < 0 $: Processing the entire sequence increases entropy, resulting in worse performance than focusing only on the last $ T_{\text{train}} $ tokens. This implies negative information gain and limited extrapolation capability (e.g. such as RoPE), as the model may overfit to recent tokens without capturing broader context effectively.
    
* When $\Delta P > 0$: The model benefits from the information within $ x[:T_{\text{train}}] $, achieving a reduction in entropy that reflects positive information gain. This suggests the model leverages contextual information beyond the training sequence, indicating extrapolation capability.

By examining $ \Delta P$, we can evaluate the model’s ability to reduce entropy and gain information from extended sequences, providing a measure of its extrapolative power.


**Q7: There is no discussion of computational efficiency metrics such as FLOPS, which would be valuable for assessing the proposed method's practicality.**

A7: We have discussed the computation cost in Section 4.2 and real time cost in Section 4.9.

**Computation Cost**
When T becomes larger, the FLOPS cost of $DAPE_{1x3}$ is $\mathcal{O}(B \cdot (h \cdot d \cdot T^2 + 3 \cdot h \cdot D_{\text{DAPE}} \cdot T^2))$, where $B$, $h$, $d$, $T$ and $D_{\text{DAPE}}$ are the batch size, attention head number, attention hidden dimension, sequence length and DAPE hidden dimension.

**Time Cost in milliseconds**
| **Method**                        | **350M Total** | **Ratio** | **2.7B Total** | **Ratio** | **6.7B Total** | **Ratio** |
|------------------------------------|----------------|-----------|----------------|-----------|----------------|-----------|
| RoPE            | 210.01         | 0.8306    | 472.63         | 1.0472    | 635.57         | 0.8564    |
| T5's bias]   | 355.16         | 1.4046    | 537.62         | 1.1912    | 808.85         | 1.0899    |
| ALiBi         | 172.60         | 0.6826    | 325.95         | 0.7222    | 596.77         | 0.8041    |
| Kerple           | 189.91         | 0.7511    | 370.32         | 0.8205    | 661.82         | 0.8918    |
| FIRE           | 248.13         | 0.9813    | 432.63         | 0.9586    | 797.68         | 1.0748    |
| $\text{DAPE}$-Kerple | 224.22         | 0.8868    | 422.48         | 0.9361    | 717.46         | 0.9667    |
| $\text{DAPE}_{1\times3}$-Kerple   | 252.84         | 1.0000    | 451.29         | 1.0000    | 742.10         | 1.0000    |


**As the model size increases, the additional computational cost ratio gradually decreases**. As
shown in the above Table, when the model size is 350M, the time cost for Kerple is 189.91 ms, while
DAPE-Kerple takes 224.22 ms, and DAPE1×3-Kerple requires 252.84 ms. Compared to DAPE1×3-
Kerple, the time cost ratios for Kerple and DAPE-Kerple are 0.7511 and 0.8868, respectively. As
the model size increases from 350M to 2.7B and 6.7B, the time cost ratio for Kerple rises from
0.7511 to 0.8205 and 0.8918, respectively. Similarly, the time cost ratio for DAPE-Kerple increases
from 0.8868 to 0.9361 and 0.9677. Therefore, as the model size increases, the time cost ratio also
increases, indicating that the additional computational cost decreases progressively.
Response to Reviewer UyHR (Part 3/5)
**A6: The current popular solution for long-context extrapolation is to fine-tune RoPE-based LLMs on long-context data, which is not addressed in the baseline comparisons.**

A6: We answer the question in two questions:  1) the definition of length extrapolation; 2) RoPE Extrapolation Ability; 3) Our long-context evaluation.

**The Definition of Length Extrapolation**

According to ALiBi paper, the length extrapolation is that the model is trained on sequence length $T_{train}$ and validate on $T_{valid}$ and $T_{valid}$  is larger than $T_{train}$. Therefore, for the length extrapolation setting, usually, we do not further train the model beyond the training length or longer context data.


**RoPE Extrapolation Ability**

The RoPE extrapolation is relatively poor, suggesting that the perplexity will quickly increase when  $T_{valid}$ is larger than  $T_{train}.$ And this is proved in previous works, such as  YaRN.


**Our Long-Context Evaluation**

We understand that the reviewer would like to know more about the performance on long-context so we additionally employ $\Delta P$ to further evaluate the model. According to LongPPL [1], such a method has a strong correlation with long-context performance.

In this discussion, we explore how to effectively use perplexity as a metric, incorporating concepts of information gain and entropy. Let $L(\cdot)$ represent the process for calculating loss, and $ M(x) $ denote the logit output generated by the model after processing an input sequence $ x $. For evaluating model performance, we define $ P(M(x), K) $ as follows:
* Process the entire sequence $ x $ using $ M(x)$.
*  Compute the perplexity on the last $ K $ tokens of the sequence.

To interpret information gain, we consider the training sequence length $ T_{\text{train}}$. Given an input $ x $, we calculate the change in loss/perplexity, $ \Delta P$, as: $\Delta P = P(M(x[-T_{\text{test}}:]), T_{\text{train}}) - P(M(x), T_{\text{test}})$

The term $\Delta P$ provides insights into the model's information gain relative to local and global context, allowing us to quantify entropy in terms of model uncertainty reduction. We interpret $ \Delta L $ as follows:

    


| **Method**                               | **RoPE** | **ALiBi** | **Kerple** | **DAPE-Kerple** | **$DAPE_{1x3}$-Kerple** |
|------------------------------------------|----------|-----------|------------|-----------------|-----------------------------|
| $ P(M(x_{512}), T_{test}=256)$              | 19.74    | 20.04     | 19.83      | 19.25           | 18.95                        |
| $ P(M(x_{1024}), T_{test}=256)$             | 261.39   | 19.74     | 19.19      | 18.28           | 17.92                        |
| $ P(M(x_{1024}[-T_{train}]:), T_{test}=256)$| 19.51    | 19.79     | 19.58      | 19.03           | 18.74                        |
| $\Delta P_{1024}$                        | -241.88  | 0.05      | 0.39       | 0.75            | **0.82**                         |
| $ P(M(x_{2048}), T_{test}=256)$             | 411.23   | 20.17     | 20.48      | 17.20           | 16.79                        |
| $ P(M(x_{2048}[-T_{train}]:), T_{test}=256)$| 18.74    | 19.03     | 19.84      | 18.28           | 18.01                        |
| $\Delta P_{2048}$                        | -392.49  | -1.14     | -0.64      | 1.08            |  **1.22**                         |
| $ P(M(x_{4096}), T_{test}=256)$             | 635.80   | 20.50     | 28.33      | 17.58           | 17.05                        |
| $ P(M(x_{4096}[-T_{train}]:), T_{test}=256)$| 19.11    | 19.35     | 19.07      | 18.59           | 18.19                        |
| $\Delta P_{4096}$                        | -616.69  | -1.15     | -9.26      | 1.01            |  **1.14**                         |
| $ P(M(x_{8192}), T_{test}=256)$             | 762.86   | 21.30     | 40.94      | 17.85           | 17.20                        |
| $ P(M(x_{8192}[-T_{train}]:), T_{test}=256)$| 19.78    | 20.02     | 19.85      | 19.38           | 18.98                        |
| $\Delta P_{8192}$                        | -743.08  | -1.28     | -21.09     | 1.53            |  **1.78**                         |
Response to Reviewer UyHR (Part 2/5)
Dear Reviewer 7aAt,

Thank you very much for your suggestion, we will address your concerns below.

**Q1: Basic variables such as X, W_Q, and W_K are not adequately explained in the context, despite the fact that Transformers are quite popular.**

A1: Thank you very much for your notice. We have further explained the notations in our paper.

**Q2: "The result of DAPE-NoPE (the Zheng et al. (2024) only combines DAPE with ALiBi, Kerple and FIRE but not with NoPE or RoPE)." This sentence is confusing and seems disconnected from the preceding context.**

A2: The DAPE is designed for Additive RPE (such as ALiBi, Kerple or FIRE), but it does not try NoPE or RoPE. We have revised the presentation to the following.

**Original**: The result of DAPE-NoPE (the Zheng et al. (2024) only combine DAPE with ALiBi, Kerple
and FIRE but not with NoPE or RoPE). 

**Currently**: The DAPE~\cite{zheng2024dape} is designed for additive RPE but not trying NoPE or RoPE, and we present the results of DAPE-NoPE and DAPE-RoPE in the following.

**Q3: Line 191-192 mentions: "potentially hindering the evolution of next-generation Transformer models," which lacks clarity and context.**

A3: While the efficiency of transformers, particularly the quadratic cost of attention computation, is a significant concern in practical applications, we argue that the current transformer architecture may still lack sufficient expressiveness, as performance limitations are evident in certain cases. Sacrificing the expressiveness of transformers in favor of efficiency could hinder the development of architectures that meet the growing demands for large language models (LLMs). Therefore, in our work, we do not prioritize efficiency variants of transformers. Instead, we introduce additional convolution to the attention mechanism, which, though computationally more intensive, is able to enhance the model's capabilities.


**Q4: Line 198 states: "RoPE first computes the classic attention scores of key-query multiplication with RoPE." This description is unclear and requires further elaboration.**

A4: Thank you very much for your suggestion. We have revised the presentations.

**Original:** RoPE first computes the classic attention scores of key-query multiplication with RoPE." This description is unclear and requires further elaboration.

**Current**: In the DAPE-RoPE configuration, DAPE-RoPE first computes the classic attention scores of key-query multiplication with RoPE, which are then refined using the MLPs.

**Q5: The authors fail to adequately explain the rationale and motivation for applying convolution to embed position information, abruptly transitioning to technical details without sufficient context.**

A5: We have discussed the motivation in Section 3.2. The following is our thinking process.

* DAPE is designed for additive RPE (such as ALiBI, Kerple, or FIRE), but not for NoPE or RoPE. **Then, what if there is no position encoding (NoPE)?**
* Then, we conduct experiments to combine DAPE with NoPE and RoPE.
   * For NoPE, we find that **DAPE-NoPE could directly improve the performance but DAPE-RoPE cannot**.
   * For DAPE-NoPE, there is no bias matrix and DAPE could still improve the performance, suggesting that there have a more general underlying cause. The formulation of DAPE-NoPE is $QK^T + f(QK^T)$, while Q is query embedding and K is key embedding. 
   * We then realize that $QK^T + f(QK^T)$ is very similar to the ResNet's skip connection, and the MLP could be regarded as a 1x1 convolution operation. Then, we try 1x3 convolution and find that $DAPE_{1x3}$-NoPE achieves better performance than $DAPE$-NoPE.
   * Also, though DAPE-RoPE cannot improve performance (compared to RoPE), the $DAPE_{1x3}$-RoPE could improve the performance compared to RoPE
* Therefore, **until here, we finally realize that the we could extend the DAPE to ALL Transformer Attention (whatever the position encoding is) as long as we regards the Attention Score as Feature Maps**.
Response to Reviewer UyHR (Part 1/5)
**Q3: Line 126: It is hard to buy the insight: Transformer’s length extrapolation ability is limited by the expressiveness of the naive query-key dot product. This conclusion is drawn by showing DAPE without position encoding still achieves improvement. But there exists another explanation as follows. Transformer’s length extrapolation ability is limited due to the lack of accurate position encoding. MLP in DAPE implicitly learns the spatial information from the dot product of query and key, thus improving the performance. And extending MLP to 1x3 convolution can further improve encoding the spatial information.**

A3: It seems that our explanation and the above explanation do not conflict.


**The two explanation do not conflict**
We suggest that actually the length extrapolation is limited by the query and key dot product expressiveness. And the above explanation suggests that the MLP actually learns the spatial information from the dot product of query and key. Let us explain step by step.

* First, for the decoder-only transformer, the query and key dot product (without MLP) already could contain position information (spatial information), and this is already proved in NoPE paper [1]. 
* Therefore, the MLP can enhance the existing spatial information but not generate spatial information from scratch because the original query and key dot product already contain such spatial information.
* Then, why previous DAPE-NoPE could improve the performance? The reason is a query and key dot products (without MLP) have limited expressiveness so that the learned spatial information is limited, and MLP enhances such expressiveness so that spatial information is enhanced.
* **Therefore, it seems that our explanation and the above explanation do not conflict.**


**A4: Discussion about an important reference is missing. “On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location” (in CVPR 2020), by Osman Semih Kayhan and Jan C. van Gemert. It found that the boundary effects operate even far from the image boundary, allowing the network to exploit absolute spatial location all over the image. This may help explain why convolution introduces more gains.**

A4: Thank you very much for your notice. We have added it to the reference and discussed it in our paper.

If there are any questions, please let us know. And if you think that we have addressed your concerns, could you please consider raising the score? Thank you very much for your support.


Reference:

[1] Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., & Reddy, S. (2024). The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36.
Response to Reviewer UyHR (Part 2/2)
Dear Reviewer UyHR,

Thank you very much for your comment, we will address your concerns below.

**Q1: The major concern is this paper only introduces an incremental change over DAPE, I.e. extending the MLP in attention model to 1x3 convolution. In addition, compared to the gap between DAPE and other baselines, the gap between this paper and DAPE is relatively small.**

A1: **Our target is eliminating the effect of training length as much as possible**, and ***Transformer performance has an upper bound (with the same training tokens) which is almost reached by our work**.*

| **Length & Batch Size**                           | **128** | **256** | **512** | **1024** | **2048** | **4096** | **8192** |
|---------------------------------------------------|---------|---------|---------|----------|----------|----------|----------|
| **RoPE (Length 4096 & Batch 1)**                  | 38.37   | 33.21   | 27.34   | 25.50    | 23.55    | 24.58    | 152.54   |
| **AliBi (Length 128 & Batch 32)**                 | 32.27   | 29.22   | 27.06   | 27.41    | 26.29    | 26.81    | 27.70    |
| **DAPE$_{1x3}$-Kerple (Length 128 & Batch 32)**   | **31.07**   | 27.81   | 24.38   | 23.58    | 22.41    | 23.20    | 23.52    |
| **DAPE$_{1x3}$-Kerple (Length 256 & Batch 16)**   | 31.18   | **27.61**   | 23.66   | 22.59    | 21.25    | 22.25    | 22.48    |
| **DAPE$_{1x3}$-Kerple (Length 512 & Batch 8)**    | 31.73   | 27.85   | **23.48**   | **22.17**    | 20.68    | 21.15    | 20.98    |
| **DAPE$_{1x3}$-Kerple (Length 1024 & Batch 4)**   | 32.74   | 28.53   | 23.73   | 22.18    | **20.61**    | **20.97**    | **20.60**    |
| **DAPE$_{1x3}$-Kerple (Length 2048 & Batch 2)**   | 34.15   | 29.55   | 24.33   | 22.66    | 20.88    | 21.08    | 20.79    |


**With the same training tokens, compared to $DAPE_{1x3}$ with longer training lengths, $DAPE_{1x3}$ with shorter training lengths can achieve comparable performance, indicating that $DAPE_{1x3}$ enhances the model's understanding of text structure**. On the arXiv dataset, $DAPE_{1x3}$-Kerple with training lengths of 512 demonstrates performance close to that of training with a length of 4096 when the evaluation length is 4096. Moreover, the performance curves for training lengths of 1024, and 2048 are almost identical. This trend is also observed with the Books3 dataset. These results indicate that $DAPE_{1x3}$-Kerple effectively helps the model comprehend text structure, enabling it to extend to longer lengths.


| Method   |512       | 1024       | 2048       | 4096   | 8192 | 
|-----|-----------|-----------|-----------|------------|------------|
| NoPE   | 4.682893  | 31.79675  | 1867.460  | 4666.604   | 5334.850   |
| DAPE-NoPE   | 4.632995  | 12.72003  | 751.7807  | 2033.337   | 2618.137   | 
| $DAPE_{1x3}-NoPE$   | 4.479389  | 6.316404  | 56.93893  | 196.8072   | 259.8106   | 
| RoPE  | 4.579118  | 43.62613  | 144.0501  | 278.8791   | 297.0656   | 
| DAPE-RoPE   | 4.531534  | 73.31698  | 174.4878  | 316.8401   | 306.7887   | 
| $DAPE_{1x3}-RoPE$   | 4.487977  | 13.10098  | 29.41894  | 53.97933   | 68.32326   |

* For NoPE: reduce the perplexity **from 2618.13 to 259.81.**
* For RoPE: reduce the perplexity **from 306.78 to 68.32.** 


**Q2: This paper could be written in a more straightforward way, by directly showing the difference between it and DAPE, and highlighting why it is crucial. Readers may be confusion about the contribution of this paper and DAPE.**

A2: We have highlighted the difference between DAPE and this work at the beginning of Section 3 Method. We directly copy it below.

**The two key differences between DAPE and this work are**: 
* **1) Insight:** DAPE attributes length extrapolation performance gains to adaptive position encoding and *DAPE believes that the Bias Matrix is Necessary*, while this work finds DAPE could still improve performance without position encoding so that **we take a broader view, explaining that the Transformer's length extrapolation ability is limited by the expressiveness of the naive query-key dot product, which can be enhanced using image processing techniques;**
* **2) Performance:** DAPE is designed for additive RPE and may underperform with non-additive RPE (e.g., RoPE), whereas this work suggests that increasing kernel size (e.g., with  $DAPE_{1x3}$) may improve RoPE's performance**.
Response to Reviewer UyHR (Part 1/2)
* When $ \Delta P = 0 $: The model’s information gain from the full sequence is negligible, indicating an entropy level comparable to local attention (e.g., models like ALiBi when the evaluation length is 1024). This suggests the model does not leverage context beyond a limited range.
    
* When $\Delta P < 0 $: Processing the entire sequence increases entropy, resulting in worse performance than focusing only on the last $ T_{\text{train}} $ tokens. This implies negative information gain and limited extrapolation capability (e.g. such as RoPE), as the model may overfit to recent tokens without capturing broader context effectively.
    
* When $\Delta P > 0$: The model benefits from the information within $ x[:T_{\text{train}}] $, achieving a reduction in entropy that reflects positive information gain. This suggests the model leverages contextual information beyond the training sequence, indicating extrapolation capability.

By examining $ \Delta P$, we can evaluate the model’s ability to reduce entropy and gain information from extended sequences, providing a measure of its extrapolative power.



**Q7: I’d suggest the authors replace some of the references through the seminal works in their introduction in terms of how Transformers have made an impact (e.g. noting CV but not citing ViT/DeiT isn’t good research practice, as these authors should be acknowledged)**

A7: Thank you very much for your comments. We have added the suggested works, including CV-related, ViT/DeiT, swin transformer and so on.

**Q8: I’d like to suggest the authors to check and potentially slightly rework the manuscript in terms of preciseness of their wording; While I am aware this might be due to language barrier, there are multiple instances where statements are misleading/confusing/too general, e.g.**

A8: Thank you very much for your suggestions, we have revised the corresponding sentences and we are carefully checking other presentations.

* Abstract (FFN): 
   * Original: The attention mechanism is a fundamental component of the Transformer model,
contributing to interactions among distinct tokens, in contrast to earlier feedforward neural networks
   * Current: The attention mechanism is a fundamental component of the Transformer model,
contributing to interactions among distinct tokens

* L 49: ‘rendering the outputs non-sensical’ 
   * Original: Without these encodings,
token generation would lack the necessary contextual order, rendering the outputs nonsensical.
   * Current: Without these encodings,
token generation would lack the necessary contextual order.

If there are any questions, please let us know. And if you think that we have addressed your concerns, could you please consider raising the score? Thank you very much for your support.
Response to Reviewer i54z (Part 4/4)
**Q5: I’d like the authors to include actual insights based on their experiences and the background knowledge of working with these different approaches (FIRE, ALiBi, Kerple, etc.) – e.g. is one generally preferable? If not, what are the situations you would recommend combining DAPEv2 with any particular one of these**

A5: The Kerple is a good choice for almost all setting, the FIRE may need longer training length to present its ability, and do not use ALiBi unless necessary .

* It is easy to train Kerple, as Kerple usually has few trainable parameters compared to FIRE. If you do not know which one to use, directly use Kerple.
* FIRE may have better performance, but may needs longer training length (diverges at 128 but works well at 512).  FIRE $b(i,j) = f_{\theta}\left(\frac{\psi(i-j)}{\psi(\max\{L, i\})}\right)$ so that we may needs longer training length or more trianing tokens to well-train the $f_{\theta}$.
* Do not use ALiBi unless necessary. The ALiBi will quickly becomes local attetnion with the sequence length increases.


**Q6: In Figure 6, although the model can cheat, I’d be curious why the authors think that the DAPEv2-ALiBi becomes significantly less stable (than both non-cheating and original-non-cheating)**

A6: **The ALiBi actually almost become local attention to keep the low perplexity so that it actually could not get information from previous sequence, while other position encoings (such as RoPE or Kerple) does not abandon long-distance information and they cannot handle them well so that the perplexity is higher.**
In this discussion, we explore how to effectively use perplexity as a metric, incorporating concepts of information gain and entropy. Let $L(\cdot)$ represent the process for calculating loss, and $ M(x) $ denote the logit output generated by the model after processing an input sequence $ x $. For evaluating model performance, we define $ P(M(x), K) $ as follows:
* Process the entire sequence $ x $ using $ M(x)$.
*  Compute the perplexity on the last $ K $ tokens of the sequence.

To interpret information gain, we consider the training sequence length $ T_{\text{train}}$. Given an input $ x $, we calculate the change in loss/perplexity, $ \Delta P$, as: $\Delta P = P(M(x[-T_{\text{train}}:]), T_{\text{test}}) - P(M(x), T_{\text{test}})$



The term $\Delta P$ provides insights into the model's information gain relative to local and global context, allowing us to quantify entropy in terms of model uncertainty reduction. We interpret $ \Delta L $ as follows:


| **Method**                               | **RoPE** | **ALiBi** | **Kerple** | **DAPE-Kerple** | **$DAPE_{1x3}$-Kerple** |
|------------------------------------------|----------|-----------|------------|-----------------|-----------------------------|
| $ P(M(x_{512}), T_{test}=256)$              | 19.74    | 20.04     | 19.83      | 19.25           | 18.95                        |
| $ P(M(x_{1024}), T_{test}=256)$             | 261.39   | 19.74     | 19.19      | 18.28           | 17.92                        |
| $ P(M(x_{1024}[-T_{train}]:), T_{test}=256)$| 19.51    | 19.79     | 19.58      | 19.03           | 18.74                        |
| $\Delta P_{1024}$                        | -241.88  | 0.05      | 0.39       | 0.75            | **0.82**                         |
| $ P(M(x_{2048}), T_{test}=256)$             | 411.23   | 20.17     | 20.48      | 17.20           | 16.79                        |
| $ P(M(x_{2048}[-T_{train}]:), T_{test}=256)$| 18.74    | 19.03     | 19.84      | 18.28           | 18.01                        |
| $\Delta P_{2048}$                        | -392.49  | -1.14     | -0.64      | 1.08            |  **1.22**                         |
| $ P(M(x_{4096}), T_{test}=256)$             | 635.80   | 20.50     | 28.33      | 17.58           | 17.05                        |
| $ P(M(x_{4096}[-T_{train}]:), T_{test}=256)$| 19.11    | 19.35     | 19.07      | 18.59           | 18.19                        |
| $\Delta P_{4096}$                        | -616.69  | -1.15     | -9.26      | 1.01            |  **1.14**                         |
| $ P(M(x_{8192}), T_{test}=256)$             | 762.86   | 21.30     | 40.94      | 17.85           | 17.20                        |
| $ P(M(x_{8192}[-T_{train}]:), T_{test}=256)$| 19.78    | 20.02     | 19.85      | 19.38           | 18.98                        |
| $\Delta P_{8192}$                        | -743.08  | -1.28     | -21.09     | 1.53            |  **1.78**                         |
Response to Reviewer i54z (Part 3/4)
**A2: Most results (in fact, almost all) are reported with for ‘Kerple’, which seems to work well (e.g. Figure 2, Figure 3, and Figure 3) in combination with DAPEv2; However, when looking at the ‘broader’ applicability in Figure 5, it quickly becomes clear that results across the board are much more inconsistent**

Q2: Thank may be misunderstanding here. **We will explain it step by step:** 1) We have discussed in Section 4.7: Different experiment settings may have different optimal kernel sizes and larger kernel size does nto always bring better performance but provide the potential; 2) The Explanation of $DAPE_{1x3}$-ALiBi performance; 3) The revised presentation of Section 4.4.

**We have already highlighted several times in our paper (Section 4.7 and Section 4.8) that Different experiment settings may have different optimal kernel sizes**.

As the Table shown in Q1 (Paper Section 4.7), larger kernel size doest not always brings better performance. Also, we also hightlight in Section 4.8 that Different tasks have different optimal kernel sizes and The large kernel size performance improvement is related to the baseline bias matrix. Therefore, the choice of kernel size is related to bias matrix and different experiment setting, and different setting has different optimal kernel size.


**The Explanation of $DAPE_{1x3}$-ALiBi performance**
Different experiment setting has different optimal kernel size. **The $DAPE_{1x3}-ALiBi$ needs a larger training sequence length so that $DAPE_{1x3}-ALiBi$ achieves better performance than $DAPE_{1x1}-ALiBi$ when the training length is 512** but worse performance when the training length is 128.

**The revised presentation of Section 4.4.**
We add the following sequence to Section 4.4: The $DAPE_{1x3}-ALiBi$ may needs longer training length so that the performance is better than $DAPE-ALiBi$

**Q3: Results vary a lot in terms of which one is better for which task and combined with which ‘pe-method’, and I don’t see this discussed in the manuscript appropriately.**

A3: We have discussed the impact of **bias matrix (which is pe-method)** in Section 4.8. We directly copy the discussion here.

**Different tasks have different optimal kernel sizes, as shown in Appendix E and Appendix D.**
For example, on MISSING DUPLICATE task, the DAPE1×3-Kerple improves the 87.57 of DAPEKerple to 99.65. However, on the STACK MANIPULATIONtask, the DAPE1×3-Kerple decreases
the 72.04 of DAPE-Kerple to 68.18. Also, as shown in Appendix D, the larger kernel size does
not always lead to better performance. Overall, larger kernel size provides a potential way to improve the Transformer length extrapolation performance, and we usually could find a suitable kernel
size (ranging from 1×1 to larger kernel sizes) to achieve better performance than without further
processing attention score.

**The large kernel size performance improvement is related to the baseline bias matrix(which is pe-method)**. As
shown in Appendix E, the best performance is usually achieved by further processing attention
scores via kernel size 1 or 3. Moreover, on 11 permutation-variant tasks, the DAPE1×3-Kerple
achieves better performance on 8 of 11 tasks compared to Kerple. And the DAPE1×3-FIRE achieves
better performance on 6 of 11 tasks compared to FIRE. This suggests that the large kernel size
performance improvement is related to the baseline bias matrix.

**For language modeling tasks, with enough longer training length (such as at least 512), the larger kernel size usualy brings the lower perplexity/loss.**


**Q4: General: A wider discussion of the limitations would significantly help any reader/user, and I’d suggest the authors consider being upfront about these and provide the reader with helpful guidance (Similarly when using DAPEv2 with FIRE, while there is some improvement, it still ‘diverges’ quickly)**

A4: The improvement is related to the baseline pe-method (such as ALiBi, Kerple, or FIRE), and longer training length could partialy solve the diverges problem.

* **Longer Training Length**. The performane diverges quickly when the training length is small (such as 128) but works well when the training length is larger (such as 512). 
* **Baseline PE Method is important for the performance**. Choose the suitable bias matrix (such as Kerple or FIRE) for training.
* The effect of $D_{DAPE}$. The hidden dimension of DAPE could be small, and 10 is enough (as shown in Figure 7).
Response to Reviewer i54z (Part 2/4)