Gated Delta Networks: Improving Mamba2 with Delta Rule
linear RNN;state-space model;linear transformer;subquadractic model;linear attention;delta rule;mamba
Yes
No ethics review needed.
- There are some typos in Section 3.1 (e.g., L.239 or L.256).
- This paper is well-organized and clearly written.
- Their presentation, especially highlighting contribution points, was good to understand and follow.
- The proposed method is simple, but seems effective in various tasks.
To mitigate quadratic scaling with sequence length of standard Transformers, Linear Transformers with kernelized dot-product linear attention are proposed. In this paper, the authors propose Gated DeltaNet, which leverages the gating mechanism from Mamba2 and the delta update rule from DeltaNet. The hybrid architecture of Gated DeltaNet, SWA, Mamba2 layers outperforms baselines in language modeling, reasoning, and recall-intensive tasks.
- How are the baselines trained? Those few-shot performance look a little lower than other papers (like Mamba). And also, could you compare with some other well-pretrained Transformer models to show the effectiveness of the proposed architecture?
- State sizes of Gated DeltaNet H1 and H2 seem much larger than all the other baselines---models that use SWA have the larger sizes. In case of Gated DeltaNet (with 256 state size) performance looks similar to Mamba2 and DeltaNet, where the gated delta rule can be seen as having minimal impact on performance. Also, as Samba already shows a very good performance, I guess the performance gain would come from hybrid structure, not from gated delta rule itself.
- I suggested the authors to compare efficiency of models, which is an important factor for a new architecture.
- In length extrapolation experiments, why does Gated DeltaNet show robust performance while Mamba2 and DeltaNet show higher perplexity as the length grows. Gated DeltaNet is the combination of both methods.
Yes
No ethics review needed.
1. Could you provide quantitative metrics on the computational efficiency of Gated DeltaNet compared to other models, such as training/inference speed and memory consumption?

I will read the rebuttal in the discussion period and adjust my evaluation.
## Strengths

1. This paper propose a new linear Transformer that implements the gated delta rule, allowing for more effective memory management over long sequences.
2. The authors extend the delta rule's parallel algorithm to incorporate gating, ensuring hardware-efficient training using chunkwise parallelism and tensor core acceleration.
3. Experiments demonstrate that Gated DeltaNet can outperforms existing models like Mamba2 (which uses gating) and DeltaNet (which uses the delta rule) on various tasks.
This paper addresses the limitations of linear Transformers in handling long sequences and recall-intensive tasks. While linear Transformers reduce computational complexity compared to traditional Transformers, they often struggle with tasks that require remembering and recalling information over extended contexts. Authors focus on (1) Gating Mechanism: Allows the model to adaptively forget irrelevant information by applying a dynamic decay to the hidden state. This enables fast and efficient memory erasure but may not precisely target specific memories. (2) Delta Update Rule: Enables precise and targeted updates to memory by selectively replacing old key-value pairs with new ones, to propose a new approach called Gated Delta Rule, which combines the strengths of both mechanisms.
## Weakness

1. The paper lacks an in-depth theoretical analysis of why the combination of gating and the delta rule improves memory capacity and how it affects the model's ability to handle long sequences, providing theoretical insights regarding the memory retention and forgetting properties of GatedDeltaNet would strengthen the paper's contributions.
2. Although the authors claims that Gated DeltaNet generalizes better to longer sequences than DeltaNet / Mamba2, the experimental results on length extrapolation are limited (only one analysis on PG19). The evaluation focuses on sequences up to a certain length without exploring the model's performance on substantially longer sequences. Expanding the extrapolation experiments to include much longer sequences would provide stronger evidence.
Yes
No ethics review needed.
1. Is there any visualized results or case studies about why the introduction of scalar value decay term resulted in following improvements in downstream task like AR?

2. How about the training / inference speed e.g. throughput comparing with DeltaNet, Mamba2, and hybrid model e.g. Samba?

3. Does the UT transform really helps the acceleration of training? How about some ablation studies on w/ and w/o the UT transform?
This paper gives detailed and structured introduction to a series of d^2 linear attention model e.g. GLA, DeltaNet and Mamba2, following a newly proposed Gated DeltaRule as its main contribution.  The improvements of delta rule can be considered as one strength in this paper, with a scalar-values decay term for easy training. Also, the UT transform for using Tensor Core with high-efficiency serves as a mitigation to the extended WY representation is unique, yet could be an individual interest to the community.

 The motivation for combining gating with the delta rule is compelling, showing how these two mechanisms address specific weaknesses in memory management. The writing in this paper is generally clear, structured, and detailed.
The paper introduces Gated DeltaNet as an enhancement to linear transformers, combining two memory management mechanisms: the gating mechanism from Mamba2 and the delta update rule from DeltaNet. Linear transformers, known for efficiency, often struggle with recall-intensive tasks. The gating mechanism enables flexible, rapid memory clearance, while the delta rule allows for targeted updates. By integrating these, Gated DeltaNet achieves both efficient memory erasure and selective memory retention, addressing limitations faced by linear transformers in handling extended sequences and complex memory recall tasks.

The proposed model demonstrates superior performance compared to Mamba2 and DeltaNet across various tasks, including language modeling, commonsense reasoning, and associative-recall. Additionally, the study explores hybrid models that combine Gated DeltaNet with sliding window attention or Mamba2 layers to further improve retrieval capacity. Extensive experiments validate the effectiveness of the gated delta rule, highlighting its advantage in managing memory over long sequences and maintaining hardware efficiency through chunkwise parallelism for training.
One weakness of the paper lies in the use of a scalar-value decay term, which, while intended to enhance memory management, does not consistently outperform Mamba2 in associative recall tasks. As seen in Table 2, although Gated DeltaNet shows a slight improvement in average accuracy across six associative recall tasks, Mamba2 still performs better in three of them. This raises questions about the efficacy of the decay term in improving recall performance. Intuitively, adding a decay term implies a rigid truncation of information over time, which may not necessarily benefit associative recall tasks that rely on longer memory retention. Providing case studies that track changes in hidden states during tasks could clarify the impact of the decay term on associative recall. Such insights would help to understand how the decay term influences memory retention and retrieval processes within the model, making the improvement more transparent and justifiable.

Another issue concerns the length extrapolation experiments involving DeltaNet. The paper would benefit from clearly labeling the four model types and specifying the experimental setup in detail. For example, it appears that DeltaNet performs even worse than Mamba in extrapolation, which contradicts findings from other experiments within the paper. This inconsistency suggests that additional clarification on the model configurations, including whether any of the models employed a hybrid architecture, is necessary. Given that attention mechanisms offer significant advantages in associative recall capabilities, it is essential to specify if the models were using sliding window attention or other hybrid elements, as this could substantially influence the results and better contextualize the model comparisons.

Typo: 
- line 256, "As we can see in ." need a reference?
- line 313, 319, 334, "Training" -> "Training."
- line 374, "State size is strongly correlated with final performance." need a \paragraph{}.
Yes
No ethics review needed.
How do you think the application scenario of linear-complexity models? Since it is going to be common sense that linear complexity makes a sacrifice on long-context capability.
1. The paper is well written, showing the connection and improvement with previous works.

2. Gated DeltaNet makes kernel optimization to achieve better training and inference throughput, which is essential for downstream application and community reproduction.

3. Gated DeltaNet shows strong performance than other linear-complexity models, which is a good academic contribution in model architecture area.
This paper proposes Gated DeltaNet, a novel model architecture combing gated recurrent mechanism and Delta update rule. Gated DeltaNet support chunk-parallel computation as efficiency optimization. Besides simple Gated DeltaNet, the paper also discuses the hybrid design with Sliding-Window Attention or Mamba layers.

In experiments, the paper shows strong modeling capability of Gated DeltaNet. First, Gated DeltaNet outperforms Transformer and other linear-complexity models in language modeling and zero-shot downstream tasks. Second, Gated DeltaNet shows a strong length-extrapolation ability in language modeling. In ablation studies, the paper makes comparisons with other design choices.
1. This paper does not discuss the performance on long-context tasks. Since the quadratic complexity only matters when the sequence is long, long-context performance is the most important metric for linear-complexity models.

2. This paper does not compare Gated DeltaNet with other linear-complexity models on training and inference efficiency. Since Delta-rule is harder for parallel computation, there may be a trade-off between performance and computation cost.
We introduce Gated DeltaNet, which combines the gating mechanism from Mamba2 with the delta rule from DeltaNet, achieving superior performance compared to both models individually.
@inproceedings{
anonymous2024gated,
title={Gated Delta Networks: Improving Mamba2 with Delta Rule},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=r8H7xhYPwz},
note={under review}
}
Linear Transformers have emerged as efficient alternatives to standard Transformers due to their inference efficiency, achieving competitive performance across various tasks, though they often struggle with recall-intensive tasks. Recently, two mechanisms—the gating mechanism and the delta update rule—have been used to enhance linear Transformers. We found these two mechanisms to be complementary: the gating mechanism enables fast, adaptive memory erasure, while the delta rule allows for more precise and targeted memory updates. In this work, we introduce the gated delta rule, which combines both mechanisms, and extend the delta rule's parallel algorithm to incorporate gating. Our experiments demonstrate that linear Transformers with the gated delta rule, dubbed Gated DeltaNet, consistently outperform Mamba2 (a gated linear transformer) and DeltaNet in language modeling, common sense reasoning, and real-world in-context recall-intensive tasks. Additionally, we explore hybrid models that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, further enhancing retrieval capabilities.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
linear RNN state-space model linear transformer subquadractic model linear attention delta rule mamba
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b4364be17e738609185d8d77f9c2ae800f22c28c.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Gated Delta Networks: Improving Mamba2 with Delta Rule
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission