Reading + AI
LibrarySign in
Open the original paper on arXiv (new tab)

Gated Delta Networks: Improving Mamba2 with Delta Rule

Songlin Yang, Jan Kautz, Ali Hatamizadeh

arXiv:2412.06464v3 · This workspace stays on this version. Check arXiv for newer versions

Affiliations and author notes
  • MIT CSAIL
  • yangsl66@mit.edu
  • NVIDIA
  • jkautz@nvidia.com
  • ahatamizadeh@nvidia.com
  • Equation contribution. Work done during SY’s internship at NVIDIA.
Some source details need review 4 to check

Original formulas, table values and figures are preserved. Comparison can miss differences; use the original PDF to check details.

Reading view checked: 153 math expressions, 31 rendering failures; 10 of 10 images available.

Compared 115 of 115 source passages against 22 of 22 PDF pages.

  • Some graphics are laid out differently in the HTML source than in the paper (text or formulas outside their frame); check the original PDF. (2)
  • Some formulas could not be rendered; their original TeX is shown. Check the original PDF. (31)
  • S3_F1: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (too-small); the image shows the HTML layout, which differs from the PDF.
  • S4_F3: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (too-small); the image shows the HTML layout, which differs from the PDF.

10 of 10 figure, table and graphic images captured. Open original PDF

Abstract

Linear Transformers have gained attention as efficient alternatives to standard Transformers, but their performance in retrieval and long-context tasks has been limited. To address these limitations, recent work has explored two distinct mechanisms: gating for adaptive memory control and the delta update rule for precise memory modifications. We observe that these mechanisms are complementary—gating enables rapid memory erasure while the delta rule facilitates targeted updates. Building on this insight, we introduce the gated delta rule and develop a parallel training algorithm optimized for modern hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks, including language modeling, common-sense reasoning, in-context retrieval, length extrapolation, and long-context understanding. We further enhance performance by developing hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance.
Code: https://github.com/NVlabs/GatedDeltaNet

The Transformer architecture has significantly advanced the capabilities of Large Language Models (LLMs), showcasing exceptional performance across a wide range of tasks due to its effective attention mechanism. This mechanism excels in precise sequence modeling and leverages the parallel processing capabilities of modern GPUs during training. However, the self-attention component scales quadratically with sequence length, leading to substantial computational demands that pose challenges for both training and inference.

To mitigate these issues, researchers have explored alternatives such as linear Transformers (Katharopoulos et al., 2020a), which replace traditional softmax-based attention with kernelized dot-product-based linear attention, substantially reducing memory requirements during inference by reframing as a linear RNN with matrix-valued states. While early versions of linear Transformers underperformed in language modeling tasks compared to standard Transformers, recent enhancements—such as incorporating data-dependent gating mechanisms akin to those in LSTMs, exemplified by models like GLA (Yang et al., 2024a) and Mamba2 (Dao & Gu, 2024a)—have shown promising improvements. However, challenges persist in managing information over long sequences, particularly for in-context retrieval tasks where traditional Transformers maintain their advantage (Arora et al., 2023a; Arora et al., 2024a; Jelassi et al., 2024; Wen et al., 2024; Akyürek et al., 2024).

This phenomenon is not surprising: linear Transformers can be interpreted as implementing an outer-product-based key-value association memory, reminiscent of tensor product representation (Smolensky, 1990). However, the number of orthogonal key-value pairs they can store is bounded by the model’s dimensionality. When the sequence length exceeds this dimension, “memory collisions“ become inevitable, hindering exact retrieval (Schlag et al., 2021a).

Mamba2 addresses this limitation by introducing a simple gated update rule, St=αtSt−1+vtkt⊺{\mathbf{S}}_{t}=\alpha_{t}{\mathbf{S}}_{t-1}+{\bm{v}}_{t}{\bm{k}}_{t}^{\intercal}, which uniformly decays all key-value associations at each time step by a dynamic ratio, αt∈(0,1)\alpha_{t}\in(0,1). However, this approach does not account for the varying importance of different key-value associations, potentially leading to inefficient memory utilization. If the model needs to forget a specific key-value association, all key-value associations are equally forgotten, making the process less targeted and efficient.

In contrast, the linear Transformer with the delta rule (Widrow et al., 1960), known as DeltaNet (Schlag et al., 2021a; Yang et al., 2024b), selectively updates memory by (softly) replacing an old key-value pair with the incoming one in a sequential manner. This method has demonstrated impressive performance in synthetic benchmarks for in-context retrieval. However, since this process only modifies a single key-value pair at a time, the model lacks the ability to rapidly clear outdated or irrelevant information, especially during context switches where previous data needs to be erased. Consequently, DeltaNet has been found to perform moderately on real-world tasks (Yang et al., 2024b), likely due to the absence of a robust memory-clearing mechanism.

Recognizing the complementary advantages of the gated update rule and the delta rule in memory management, we propose the gated delta rule, a simple and intuitive mechanism that combines both approaches. This unified rule enables flexible memory control: it can promptly clear memory by setting αt→0\alpha_{t}\rightarrow 0, while selectively updating specific content without affecting other information by setting αt→1\alpha_{t}\rightarrow 1 (effectively switching to the pure delta rule).

The remaining challenge lies in implementing the gated delta rule in a hardware-efficient manner. Building upon Yang et al. (2024b)’s efficient algorithm that parallelizes the delta rule computation using the WY representation (Bischof & Loan, 1985), we carefully extend their approach to incorporate the gating terms. Our extension preserves the benefits of chunkwise parallelism (Hua et al., 2022b; Sun et al., 2023a; Yang et al., 2024a; Yang et al., 2024b), enabling hardware-efficient training.

Our resulting architecture, Gated DeltaNet, consistently outperforms both Mamba2 and DeltaNet across a comprehensive suite of benchmarks, including language modeling, commonsense reasoning, in-context retrieval, length extrapolation, and long-context understanding. Building on these results, we also develop hybrid architectures that strategically combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, further enhancing both training efficiency and model performance.

It is known that the linear transformer (Katharopoulos et al., 2020b) can be formulated as the following linear recurrence when excluding normalization and query/key activations:

St=St−1+vtkt⊺∈Rdv×dk,ot=Stqt∈Rdv\displaystyle{\mathbf{S}}_{t}={\mathbf{S}}_{t-1}+{\bm{v}}_{t}{\bm{k}}_{t}^{\intercal}\in\mathbb{R}^{d_{v}\times d_{k}},\qquad\qquad{\bm{o}}_{t}={\mathbf{S}}_{t}{\bm{q}}_{t}\in\mathbb{R}^{d_{v}}

where dkd_{k} and dvd_{v} represent the (head) dimensions for query/key and value, respectively. By expanding the recurrence, we can express it in both vector form (left) and matrix form (right) as follows:

ot=∑i=1t(viki⊺)qt=∑i=1tvi(ki⊺qt)∈Rdv,O=(QK⊺⊙M)V∈RL×dv\displaystyle{\bm{o}}_{t}=\sum_{i=1}^{t}({\bm{v}}_{i}{\bm{k}}_{i}^{\intercal}){\bm{q}}_{t}=\sum_{i=1}^{t}{\bm{v}}_{i}({\bm{k}}_{i}^{\intercal}{\bm{q}}_{t})\in\mathbb{R}^{d_{v}},\qquad{\mathbf{O}}=({\mathbf{Q}}{\mathbf{K}}^{\intercal}\odot{\mathbf{M}}){\mathbf{V}}\in\mathbb{R}^{L\times d_{v}}

where LL is the sequence length, and M∈RL×L{\mathbf{M}}\in\mathbb{R}^{L\times L} is the causal mask defined by Mij=0{\mathbf{M}}_{ij}=0 when i<ji<j, and 11 otherwise.

However, this vanilla linear attention underperforms Transformers in language modeling by a large margin. To address this, it is common to add a decay term to forget historical information. Here we take Mamba2 (Dao & Gu, 2024a) as an example, which can be represented by the following linear recurrence (up to specific parameterization):

{\mathbf{S}}_{t}={\color[rgb]{0,0,1}\alpha_{t}}{\mathbf{S}}_{t-1}+{\bm{v}}_{t}{\bm{k}}_{t}^{\intercal},\qquad{\bm{o}}_{t}={\mathbf{S}}_{t}{\bm{q}}_{t}

where {\color[rgb]{0,0,1}\alpha_{t}\in(0,1)} is a data-dependent scalar-valued decay term that varies with tt. Define the cumulative decay product \color[rgb]{0,0,1}{\gamma_{j}=\prod_{i=1}^{j}\alpha_{i}}, and by expanding the recurrence, we can express the result in both a vector form (left) and a matrix parallel form (right):

{\bm{o}}_{t}=\sum_{i=1}^{t}\left({{\color[rgb]{0,0,1}\frac{\gamma_{t}}{\gamma_{i}}}}{\bm{v}}_{i}{\bm{k}}_{i}^{\intercal}\right){\bm{q}}_{t}=\sum_{i=1}^{t}{\bm{v}}_{i}\left({\color[rgb]{0,0,1}\frac{\gamma_{t}}{\gamma_{i}}}{\bm{k}}_{i}^{\intercal}{\bm{q}}_{t}\right),\qquad{\mathbf{O}}=\left(\left({\mathbf{Q}}{\mathbf{K}}^{\intercal}\right)\odot{{\color[rgb]{0,0,1}\Gamma}}\right){\mathbf{V}}

Here, {\color[rgb]{0,0,1}\Gamma\in\mathbb{R}^{L\times L}} is a decay-aware causal mask where \color[rgb]{0,0,1}{\Gamma_{ij}=\frac{\gamma_{i}}{\gamma_{j}}} if i≥ji\geq j and {\color[rgb]{0,0,1}\Gamma_{ij}=0} otherwise. The equivalence between these parallel and recurrent forms is also referred to as the state space duality (SSD) described in Dao & Gu (2024a). This recurrence structure appears in several other architectures including Gated RFA (Peng et al., 2021), xLSTM (Beck et al., 2024), and Gated RetNet (Sun et al., 2024b). When γt\gamma_{t} is data-independent, the formulation reduces to RetNet (Sun et al., 2023a) and Lightning-Attention (Qin et al., 2024a). Furthermore, if γt\gamma_{t} is extended to be matrix-valued rather than scalar-valued, efficient training algorithms remain possible when parameterized with an outer-product structure, as demonstrated by Yang et al. (2024a) and used by Yang et al. (2024a); Peng et al. (2024); Qin et al. (2024b); Zhang et al. (2024); Chou et al. (2024); He et al. (2025); Lu et al. (2025).

However, both the recurrent and parallel forms are not ideal for efficient training (Hua et al., 2022b; Yang et al., 2024a), which motivates the use of the chunkwise parallel form (Hua et al., 2022b; Sun et al., 2023a) for hardware-efficient, linear-time training, as introduced below. To summarize, the chunkwise parallel form splits inputs and outputs into several chunks of size CC, and computes outputs for each chunk based on the final state of the previous chunk and the query/key/value blocks of the current chunk. Following the notation of Sun et al. (2023b); Yang et al. (2024a); Yang et al. (2024b), we take the query block, q{\bm{q}}, as an example. We denote Q[t]:=qtC+1:(t+1)C+1{\mathbf{Q}}_{[t]}:={\bm{q}}_{tC+1:(t+1)C+1} as the query block for chunk tt, and q[t]r:=qtC+r{\bm{q}}_{[t]}^{r}:={\bm{q}}_{tC+r} as the rr-th query within chunk tt. The initial state of chunk tt is defined as S[t]:=S[t]0=S[t−1]C{\mathbf{S}}_{[t]}:={\mathbf{S}}_{[t]}^{0}={\mathbf{S}}_{[t-1]}^{C}. By partially expanding the recurrence, we have

S[t]r=S[t]+∑i=1rv[t]ik[t]i⊺∈Rdv×dk,o[t]r=S[t]rq[t]r=S[t]q[t]r+∑i=1rv[t]i(k[t]i⊺q[t]r)∈Rdv\displaystyle{\mathbf{S}}_{[t]}^{r}={\mathbf{S}}_{[t]}+\sum_{i=1}^{r}{\bm{v}}_{[t]}^{i}{\bm{k}}_{[t]}^{i\intercal}\in\mathbb{R}^{d_{v}\times d_{k}},\qquad{\bm{o}}_{[t]}^{r}={\mathbf{S}}_{[t]}^{r}{\bm{q}}_{[t]}^{r}={\mathbf{S}}_{[t]}{\bm{q}}_{[t]}^{r}+\sum_{i=1}^{r}{\bm{v}}_{[t]}^{i}\left({\bm{k}}_{[t]}^{i\intercal}{\bm{q}}_{[t]}^{r}\right)\in\mathbb{R}^{d_{v}}

Equivalently, in matrix form:

S[t+1]=S[t]+V[t]K[t]⊺∈Rdv×dk,O[t]=Q[t]S[t]⊺+(Q[t]K[t]⊺⊙M)V[t]∈RC×dv\displaystyle{\mathbf{S}}_{[t+1]}={\mathbf{S}}_{[t]}+{\mathbf{V}}_{[t]}{\mathbf{K}}_{[t]}^{\intercal}\in\mathbb{R}^{d_{v}\times d_{k}},\qquad{\mathbf{O}}_{[t]}={\mathbf{Q}}_{[t]}{\mathbf{S}}_{[t]}^{\intercal}+\left({\mathbf{Q}}_{[t]}{\mathbf{K}}_{[t]}^{\intercal}\odot{\mathbf{M}}\right){\mathbf{V}}_{[t]}\in\mathbb{R}^{C\times d_{v}}

where M∈RC×C{\mathbf{M}}\in\mathbb{R}^{C\times C} is the causal mask. The above equations are rich in matrix multiplications (matmuls), allowing for tensor-core-based hardware optimization. This chunkwise algorithm could be easily extended to linear attention with decay:

\begin{aligned}\displaystyle{\mathbf{S}}_{[t+1]}={\color[rgb]{0,0,1}\overrightarrow{{\mathbf{S}}_{[t]}}}+{\mathbf{V}}_{[t]}^{\intercal}{\color[rgb]{0,0,1}\overrightarrow{{\mathbf{K}}_{[t]}}}\in\mathbb{R}^{d_{v}\times d_{k}}, & & \displaystyle{\mathbf{O}}_{[t]}={\color[rgb]{0,0,1}{\overleftarrow{{\mathbf{Q}}_{[t]}}}}{\mathbf{S}}_{[t]}^{\intercal}+\left({\mathbf{Q}}_{[t]}{\mathbf{K}}_{[t]}^{\intercal}\odot{\color[rgb]{0,0,1}\Gamma_{[t]}}\right){\mathbf{V}}_{[t]}\in\mathbb{R}^{C\times d_{v}}\end{aligned}\tag{(1)}

where {\color[rgb]{0,0,1}(\Gamma_{[t]})_{ij}=\frac{\gamma_{[t]}^{i}}{\gamma_{[t]}^{j}},\gamma_{[t]}^{j}=\prod_{j=tC+1}^{tC+j}\alpha_{j}}. 1^{1} Here we use the left arrow (⋅←\overleftarrow{\cdot}) or the right arrow (⋅→\overrightarrow{\cdot}) to denote a variable decaying to the first position and the last position of each chunk, respectively,

\begin{aligned}\displaystyle{\color[rgb]{0,0,1}\overleftarrow{{\bm{q}}_{[t]}^{r}}} & \displaystyle={\color[rgb]{0,0,1}\gamma_{[t]}^{r}}{\bm{q}}_{[t]}^{r} & & t\end{aligned}
\begin{aligned}\displaystyle{\color[rgb]{0,0,1}\overrightarrow{{\bm{k}}_{[t]}^{r}}} & \displaystyle={\color[rgb]{0,0,1}\frac{\gamma_{[t]}^{C}}{\gamma_{[t]}^{r}}}{\bm{k}}_{[t]}^{r} & & t\end{aligned}
\begin{aligned}\displaystyle{\color[rgb]{0,0,1}\overrightarrow{{\mathbf{S}}_{[t]}}} & \displaystyle={\color[rgb]{0,0,1}\gamma_{[t]}^{C}}{\mathbf{S}}_{[t]} & & t\end{aligned}\tag{(2)}

and likewise for other variables (e.g., {\color[rgb]{0,0,1}\overrightarrow{{\bm{v}}}}). The SSD decomposition algorithm introduced in Mamba2 is largely equivalent to this chunkwise algorithm. For a more generalized approach, Yang et al. (2024a) proposed an extended chunkwise algorithm for linear attention that incorporates fine-grained decay mechanisms.

The delta update rule (Widrow et al., 1960; Schlag et al., 2021b) dynamically erases the value (vtold{\bm{v}}_{t}^{\text{old}}) associated with the current input key (kt{\bm{k}}_{t}) and writes a new value (vtnew{\bm{v}}_{t}^{\text{new}}), which is a linear combination of the current input value and the old value based on the “writing strength” βt∈(0,1)\beta_{t}\in(0,1).2^{2}

St=St−1−(St−1kt)⏟vtoldkt⊺+(βtvt+(1−βt)St−1kt))⏟vtnewkt⊺=St−1(I−βtktkt⊺)+βtvtkt⊺\begin{aligned}\displaystyle{\mathbf{S}}_{t} & \displaystyle={\mathbf{S}}_{t-1}-\underbrace{\left({\mathbf{S}}_{t-1}{\bm{k}}_{t}\right)}_{{\bm{v}}_{t}^{\text{old}}}{\bm{k}}_{t}^{\intercal}+\underbrace{\left(\beta_{t}{\bm{v}}_{t}+(1-\beta_{t}){\mathbf{S}}_{t-1}{\bm{k}}_{t})\right)}_{{\bm{v}}_{t}^{\text{new}}}{\bm{k}}_{t}^{\intercal}={\mathbf{S}}_{t-1}\left({\mathbf{I}}-\beta_{t}{\bm{k}}_{t}{\bm{k}}_{t}^{\intercal}\right)+\beta_{t}{\bm{v}}_{t}{\bm{k}}_{t}^{\intercal}\end{aligned}

As shown above, DeltaNet implements a first-order linear recurrence with generalized Householder transition matrices (I−βtktkt⊺)\left({\mathbf{I}}-\beta_{t}{\bm{k}}_{t}{\bm{k}}_{t}^{\intercal}\right). Despite demonstrating superior associative recall and language modeling performance (Schlag et al., 2021a), DeltaNet received limited attention due to computational inefficiency until Yang et al. (2024b) introduced a hardware-efficient chunkwise training algorithm, as detailed below.

By partially expanding the recurrence, we have

S[t]r=S[t](∏i=1rI−β[t]ik[t]ik[t]i⊺)⏟:=P[t]r+∑i=1r(β[t]iv[t]ik[t]i⊺∏j=i+1r(I−β[t]jk[t]jk[t]j⊺))⏟:=H[t]r\displaystyle{\mathbf{S}}_{[t]}^{r}={\mathbf{S}}_{[t]}\underbrace{\left(\prod_{i=1}^{r}{\mathbf{I}}-\beta_{[t]}^{i}{\bm{k}}_{[t]}^{i}{\bm{k}}_{[t]}^{i\intercal}\right)}_{:={\mathbf{P}}_{[t]}^{r}}+\underbrace{\sum_{i=1}^{r}\left(\beta^{i}_{[t]}{\bm{v}}^{i}_{[t]}{\bm{k}}_{[t]}^{i\intercal}\prod_{j=i+1}^{r}\left({\mathbf{I}}-\beta_{[t]}^{j}{\bm{k}}^{j}_{[t]}{\bm{k}}_{[t]}^{j\intercal}\right)\right)}_{:={\mathbf{H}}_{[t]}^{r}}
(3)

where P[t]j{\mathbf{P}}_{[t]}^{j} involves cumulative products of generalized Householder matrices, which could be optimized by the classical WY representation (Bischof & Loan, 1985):

P[t]r=I−∑i=1rw[t]ik[t]i⊺∈Rdk×dkw[t]r=β[t]r(k[t]r−∑i=1r−1(w[t]i(k[t]i⊺k[t]r)))∈Rdk\begin{aligned}\displaystyle{\mathbf{P}}_{[t]}^{r} & \displaystyle={\mathbf{I}}-\sum_{i=1}^{r}\mathbf{w}_{[t]}^{i}{\bm{k}}_{[t]}^{i\intercal}\in\mathbb{R}^{d_{k}\times d_{k}} & & \displaystyle\mathbf{w}_{[t]}^{r}=\beta_{[t]}^{r}\left({\bm{k}}_{[t]}^{r}-\sum_{i=1}^{r-1}\left(\mathbf{w}_{[t]}^{i}({\bm{k}}_{[t]}^{i\intercal}{\bm{k}}_{[t]}^{r})\right)\right)\in\mathbb{R}^{d_{k}}\end{aligned}
(4)

Likewise, H[t]r{\mathbf{H}}_{[t]}^{r} could be represented as:

H[t]r=∑i=1ru[t]ik[t]i⊺∈Rdv×dku[t]r=β[t]r(v[t]r−∑i=1r−1(u[t]i(k[t]i⊺k[t]r)))∈Rdv\begin{aligned}\displaystyle{\mathbf{H}}_{[t]}^{r} & \displaystyle=\sum_{i=1}^{r}\mathbf{u}_{[t]}^{i}{\bm{k}}_{[t]}^{i\intercal}\in\mathbb{R}^{d_{v}\times d_{k}} & & \displaystyle\mathbf{u}_{[t]}^{r}=\beta_{[t]}^{r}\left({\bm{v}}_{[t]}^{r}-\sum_{i=1}^{r-1}\left(\mathbf{u}_{[t]}^{i}({\bm{k}}_{[t]}^{i\intercal}{\bm{k}}_{[t]}^{r})\right)\right)\in\mathbb{R}^{d_{v}}\end{aligned}
(5)

and in matrix form: P[t]=I−W[t]⊤K[t]∈Rdk×dk{\mathbf{P}}_{[t]}={\mathbf{I}}-{\mathbf{W}}_{[t]}^{\top}{\mathbf{K}}_{[t]}\in\mathbb{R}^{d_{k}\times d_{k}}, H[t]=U[t]⊤K[t]∈Rdv×dk{\mathbf{H}}_{[t]}={\mathbf{U}}_{[t]}^{\top}{\mathbf{K}}_{[t]}\in\mathbb{R}^{d_{v}\times d_{k}}. By using the UT transform (Joffrain et al., 2006), we can further write W{\mathbf{W}} and U{\mathbf{U}} in matrix form:

T[t]=[I+strictLower⁡(diag⁡(β[t])K[t]K[t]⊺)]−1diag⁡(β[t])∈RC×C\displaystyle{\mathbf{T}}_{[t]}=\left[{\mathbf{I}}+\operatorname{strictLower}\left(\operatorname{diag}(\beta_{[t]}){\mathbf{K}}_{[t]}{\mathbf{K}}_{[t]}^{\intercal}\right)\right]^{-1}\operatorname{diag}\left(\beta_{[t]}\right)\in\mathbb{R}^{C\times C}
(6)
W[t]=T[t]K[t]∈RC×dk,U[t]=T[t]V[t]∈RC×dv\displaystyle{\mathbf{W}}_{[t]}={\mathbf{T}}_{[t]}{\mathbf{K}}_{[t]}\in\mathbb{R}^{C\times d_{k}},\qquad{\mathbf{U}}_{[t]}={\mathbf{T}}_{[t]}{\mathbf{V}}_{[t]}\in\mathbb{R}^{C\times d_{v}}
(7)

Substituting these back into Eq. 3 yields a hardware-efficient chunkwise algorithm for DeltaNet that leverages matmuls, enabling tensor core based GPU optimization:

S[t+1]=S[t]P[t]+H[t]=S[t]+(U[t]−W[t]S[t]⊺)⊺K[t]∈Rdv×dk\begin{aligned}\displaystyle{\mathbf{S}}_{[t+1]} & \displaystyle={\mathbf{S}}_{[t]}{\mathbf{P}}_{[t]}+{\mathbf{H}}_{[t]}={\mathbf{S}}_{[t]}+\left({\mathbf{U}}_{[t]}-{\mathbf{W}}_{[t]}{\mathbf{S}}_{[t]}^{\intercal}\right)^{\intercal}{\mathbf{K}}_{[t]} & \displaystyle\in\mathbb{R}^{d_{v}\times d_{k}}\end{aligned}
(8)
O[t]=Q[t]S[t]⊺+(Q[t]K[t]⊺⊙M)(U[t]−W[t]S[t]⊺)∈RC×dv\begin{aligned}\displaystyle{\mathbf{O}}_{[t]} & \displaystyle={\mathbf{Q}}_{[t]}{\mathbf{S}}_{[t]}^{\intercal}+({\mathbf{Q}}_{[t]}{\mathbf{K}}_{[t]}^{\intercal}\odot{\mathbf{M}})\left({\mathbf{U}}_{[t]}-{\mathbf{W}}_{[t]}{\mathbf{S}}_{[t]}^{\intercal}\right) & \displaystyle\in\mathbb{R}^{C\times d_{v}}\end{aligned}
(9)

The proposed gated delta rule is simple yet effective:

\displaystyle{\mathbf{S}}_{t}={\mathbf{S}}_{t-1}\left({\color[rgb]{0,0,1}{\alpha_{t}}}({\mathbf{I}}-\beta_{t}{\bm{k}}_{t}{\bm{k}}_{t}^{\intercal})\right)+\beta_{t}{\bm{v}}_{t}{\bm{k}}_{t}^{\intercal}\tag{(10)}

where the data-dependent gating term \color[rgb]{0,0,1}{\alpha_{t}}\in(0,1) controls state decay. This formulation unifies the advantages of both gating mechanisms and the delta rule: the gating term enables adaptive memory management, while the delta update structure facilitates effective key-value association learning.

We present a formal analysis of the gated delta rule through the lens of the online learning framework introduced by Liu et al. (2024). In this framework, recurrent state updates emerge as closed-form solutions to an online learning problem, as shown in Table 1. Recent linear RNN architectures typically incorporate a regularization term in their online learning objective to prevent state divergence from previous values, thereby enabling memory retention. However, this retention mechanism becomes problematic when the state becomes saturated with information. In such cases, each state would encode a superposition of multiple information pieces, making precise retrieval challenging. To address this limitation, Mamba2 and Gated DeltaNet introduce an adaptive scaling factor αt\alpha_{t} that relaxes the regularization term, allowing controlled deviations between St{\mathbf{S}}_{t} and St−1{\mathbf{S}}_{t-1}. This modification enables dynamic memory management through selective forgetting, which could be useful in filtering out irrelevant information (see §3.2).

On the other hand, Linear Attention (LA) and Mamba2 use a simple negative inner-product loss -⟨Stkt,vt⟩\langle{\mathbf{S}}_{t}{\bm{k}}_{t},{\bm{v}}_{t}\rangle, while Longhorn (Liu et al., 2024) uses a more expressive online regression objective ∥Stkt−vt∥2\|{\mathbf{S}}_{t}{\bm{k}}_{t}-{\bm{v}}_{t}\|^{2} for better modeling of key-value associations. The resulting Longhorn’s update rule closely resembles the delta update rule, 3^{3} suggesting the superiority of the (gated) delta rule over Mamba2 in in-context associative recall.

From the perspective of fast weight programming (Irie et al., 2022a) and test-time training (Sun et al., 2024a) and regression (Wang et al., 2025), the hidden state S{\mathbf{S}} can be interpreted as a (fast) weight matrix, with the delta rule optimizing the online regression objective L(St)=12∥Stkt−vt∥2\mathcal{L}({\mathbf{S}}_{t})=\frac{1}{2}\|{\mathbf{S}}_{t}{\bm{k}}_{t}-{\bm{v}}_{t}\|^{2} via test-time stochastic gradient descent (SGD):

St+1=St−βt∇L(St)=St−βt(Stkt−vt)kt⊺=St(I−βtktkt⊺)+βtvtkt⊺\begin{aligned}\displaystyle{\mathbf{S}}_{t+1} & \displaystyle={\mathbf{S}}_{t}-\beta_{t}\nabla\mathcal{L}({\mathbf{S}}_{t})={\mathbf{S}}_{t}-\beta_{t}({\mathbf{S}}_{t}{\bm{k}}_{t}-{\bm{v}}_{t}){\bm{k}}_{t}^{\intercal}={\mathbf{S}}_{t}\left({\mathbf{I}}-\beta_{t}{\bm{k}}_{t}{\bm{k}}_{t}^{\intercal}\right)+\beta_{t}{\bm{v}}_{t}{\bm{k}}_{t}^{\intercal}\end{aligned}

where βt\beta_{t} represents the (adaptive) learning rate. From this perspective, the gated delta rule can be viewed as incorporating an adaptive weight decay term αt\alpha_{t} into the SGD update, a technique widely used in deep learning (Krogh & Hertz, 1991; Andriushchenko et al., 2023). Concurrently, Titans (Behrouz et al., 2024) demonstrated the effectiveness of incorporating weight decay mechanisms in RNN test-time SGD updates.

Table 1
MethodOnline Learning ObjectiveOnline Update
LA∥St−St−1∥F2−2⟨Stkt,vt⟩\displaystyle\|{\mathbf{S}}_{t}-{\mathbf{S}}_{t-1}\|_{F}^{2}-2\langle{\mathbf{S}}_{t}{\bm{k}}_{t},{\bm{v}}_{t}\rangleSt=St−1+vtktT\displaystyle{\mathbf{S}}_{t}={\mathbf{S}}_{t-1}+{\bm{v}}_{t}{\bm{k}}_{t}^{T}
Mamba2∥St−αtSt−1∥F2−2⟨Stkt,vt⟩\displaystyle\|{\mathbf{S}}_{t}-\alpha_{t}{\mathbf{S}}_{t-1}\|_{F}^{2}-2\langle{\mathbf{S}}_{t}{\bm{k}}_{t},{\bm{v}}_{t}\rangleSt=αtSt−1+vtktT\displaystyle{\mathbf{S}}_{t}=\alpha_{t}{\mathbf{S}}_{t-1}+{\bm{v}}_{t}{\bm{k}}_{t}^{T}
Longhorn∥St−St−1∥F2−βt∥Stkt−vt∥2\displaystyle\|{\mathbf{S}}_{t}-{\mathbf{S}}_{t-1}\|_{F}^{2}-\beta_{t}\|{\mathbf{S}}_{t}{\bm{k}}_{t}-{\bm{v}}_{t}\|^{2}St=St−1(I−ϵktktT)+ϵtvtktT,ϵt=βt1+βtkt⊤kt\displaystyle{\mathbf{S}}_{t}={\mathbf{S}}_{t-1}({\mathbf{I}}-\epsilon{\bm{k}}_{t}{\bm{k}}_{t}^{T})+\epsilon_{t}{\bm{v}}_{t}{\bm{k}}_{t}^{T},\epsilon_{t}=\frac{\beta_{t}}{1+\beta_{t}{\bm{k}}_{t}^{\top}{\bm{k}}_{t}}
DeltaNet∥St−St−1∥F2−2⟨Stkt,βt(vt−St−1kt)⟩\displaystyle\|{\mathbf{S}}_{t}-{\mathbf{S}}_{t-1}\|_{F}^{2}-2\langle{\mathbf{S}}_{t}{\bm{k}}_{t},\beta_{t}\left({\bm{v}}_{t}-{\mathbf{S}}_{t-1}{\bm{k}}_{t}\right)\rangleSt=St−1(I−βtktktT)+βtvtktT\displaystyle{\mathbf{S}}_{t}={\mathbf{S}}_{t-1}({\mathbf{I}}-\beta_{t}{\bm{k}}_{t}{\bm{k}}_{t}^{T})+\beta_{t}{\bm{v}}_{t}{\bm{k}}_{t}^{T}
Gated DeltaNet∥St−αtSt−1∥F2−2⟨Stkt,βt(vt−αtSt−1kt)⟩\displaystyle\|{\mathbf{S}}_{t}-\alpha_{t}{\mathbf{S}}_{t-1}\|_{F}^{2}-2\langle{\mathbf{S}}_{t}{\bm{k}}_{t},\beta_{t}\left({\bm{v}}_{t}-\alpha_{t}{\mathbf{S}}_{t-1}{\bm{k}}_{t}\right)\rangleSt=St−1(αt(I−βtktktT))+βtvtktT\displaystyle{\mathbf{S}}_{t}={\mathbf{S}}_{t-1}\left(\alpha_{t}({\mathbf{I}}-\beta_{t}{\bm{k}}_{t}{\bm{k}}_{t}^{T})\right)+\beta_{t}{\bm{v}}_{t}{\bm{k}}_{t}^{T}
Table 1: Comparison of different linear RNN models and their corresponding online learning objectives using the framework from Liu et al. (2024). For convenience, we simplify Longhorn’s vector-valued β{\bm{\beta}} to scalar β\beta.
View original tableOriginal Table 1
Table 2
S-NIAH-1S-NIAH-2S-NIAH-3
(pass-key retrieval)(number in haystack)(uuid in haystack)
Model1K2K4K8K1K2K4K8K1K2K4K
DeltaNet97.496.899.098.898.445.618.614.485.247.022.4
Mamba299.298.865.430.499.498.856.217.064.447.64.6
Gated DeltaNet98.488.491.491.8100.099.892.229.686.684.227.6
Table 2: Zero-shot performance comparison on S-NIAH benchmark suite for 1.3B models (see §4 for setups)
View original tableOriginal Table 2

To better understand the complementary strength between the delta rule and the gated rule, we present a case study on the Single Needle-In-A-Haystack (S-NIAH) benchmark suite from RULER (Hsieh et al., 2024), where a key-value pair acts as a needle in the haystack (context) and the model must recall the value when given the key. Table 2 presents the results and we draw three main observations:

In the simplest S-NIAH-1 setting with repeated synthetic context, models memorize minimal information, testing long-term retention. DeltaNet achieves near-perfect performance across all sequence lengths. Mamba2 degrades significantly beyond 2K sequences since it decays historical information too quickly, while Gated DeltaNet’s degradation is less severe thanks to the use of delta rule.

In S-NIAH-2/3 with real-world-essay context, models store all potentially relevant information, testing efficient memory management. With fixed state size, lack of clearance causes memory collision—information becomes superimposed and indistinguishable. DeltaNet’s performance drops significantly at longer sequences due to poor memory clearance. Mamba2 and Gated DeltaNet maintain better performance through gating mechanisms that filter irrelevant information.

In S-NIAH-3, values change from numbers to UUIDs, testing complex pattern memorization. Mamba2’s performance drops quickly, while Gated DeltaNet performs better, verifying that the delta rule indeed has better memorization ability.

In this subsection, we derive a hardware-efficient chunkwise algorithm for training Gated DeltaNet. By partially expanding the recurrence in Eq. 10, we have

\displaystyle{\mathbf{S}}_{[t]}^{r}={\mathbf{S}}_{[t]}\underbrace{\left(\prod_{i=1}^{r}{\color[rgb]{0,0,1}{\alpha_{[t]}^{i}}}\left({\mathbf{I}}-\beta_{[t]}^{i}{\bm{k}}_{[t]}^{i}{\bm{k}}_{[t]}^{i\intercal}\right)\right)}_{:=\mathbf{F}_{[t]}^{r}}+\underbrace{\sum_{i=1}^{r}\left(\beta^{i}_{[t]}{\bm{v}}^{i}_{[t]}{\bm{k}}_{[t]}^{i\intercal}\prod_{j=i+1}^{r}{\color[rgb]{0,0,1}{\alpha_{[t]}^{j}}}\left({\mathbf{I}}-\beta_{[t]}^{j}{\bm{k}}^{j}_{[t]}{\bm{k}}_{[t]}^{j\intercal}\right)\right)}_{:={\mathbf{G}}_{[t]}^{r}}

It is easy to see that \mathbf{F}_{[t]}^{r}={\color[rgb]{0,0,1}\gamma_{[t]}^{r}}\rm{P}_{[t]}^{r}={\color[rgb]{0,0,1}\overleftarrow{\rm{P}_{[t]}^{r}}}. As for G[t]r{\mathbf{G}}_{[t]}^{r}, we adapt Eq. 5 as follows,

\begin{aligned}\displaystyle{\mathbf{G}}_{[t]}^{r}=\sum_{i=1}^{r}{\color[rgb]{0,0,1}\frac{\gamma_{[t]}^{r}}{\gamma_{[t]}^{i}}}\tilde{\mathbf{u}}_{[t]}^{i}{\bm{k}}_{[t]}^{i\intercal}\in\mathbb{R}^{d_{v}\times d_{k}} & & \displaystyle\tilde{\mathbf{u}}_{[t]}^{r}=\beta_{[t]}^{r}\left({\bm{v}}_{[t]}^{r}-\sum_{i=1}^{r-1}\left(\tilde{\mathbf{u}}_{[t]}^{i}({\color[rgb]{0,0,1}\frac{\gamma_{[t]}^{r}}{\gamma_{[t]}^{i}}}{\bm{k}}_{[t]}^{i\intercal}{\bm{k}}_{[t]}^{r})\right)\right)\in\mathbb{R}^{d_{v}}\end{aligned}

(see §A for a proof). By UT transform, we have the matrix form:

\begin{aligned}\displaystyle\widetilde{{\mathbf{U}}_{[t]}}=\left[{\mathbf{I}}+\operatorname{strictLower}\left(\operatorname{diag}\left(\beta_{[t]}\right)({\color[rgb]{0,0,1}\Gamma_{[t]}}\odot{\mathbf{K}}_{[t]}{\mathbf{K}}_{[t]}^{\intercal})\right)\right]^{-1}\operatorname{diag}\left(\beta_{[t]}\right){\mathbf{V}}_{[t]} & & \displaystyle\in\mathbb{R}^{C\times d_{v}}\end{aligned}

Similar to how Mamba2 extends linear attention (Eq. 1), we can adapt DeltaNet’s chunkwise algorithm (Eq. 8-9) for Gated DeltaNet to enable hardware-efficient training as follows:

\begin{aligned}\displaystyle{\mathbf{S}}_{[t+1]} & \displaystyle={\color[rgb]{0,0,1}\overrightarrow{{\mathbf{S}}_{[t]}}}+\left({\widetilde{{\mathbf{U}}_{[t]}}}-{\color[rgb]{0,0,1}\overleftarrow{{\mathbf{W}}_{[t]}}}{\mathbf{S}}_{[t]}^{\intercal}\right)^{\intercal}{\color[rgb]{0,0,1}\overrightarrow{{\mathbf{K}}_{[t]}}} & & \displaystyle\in\mathbb{R}^{d_{v}\times d_{k}}\end{aligned}
\begin{aligned}\displaystyle{\mathbf{O}}_{[t]} & \displaystyle={\color[rgb]{0,0,1}\overleftarrow{{\mathbf{Q}}_{[t]}}}{\mathbf{S}}_{[t]}^{\intercal}+({\mathbf{Q}}_{[t]}{\mathbf{K}}_{[t]}^{\intercal}\odot\mathbf{M})\left({{\widetilde{{\mathbf{U}}_{[t]}}}}-{\color[rgb]{0,0,1}\overleftarrow{{\mathbf{W}}_{[t]}}}{\mathbf{S}}_{[t]}^{\intercal}\right) & & \displaystyle\in\mathbb{R}^{C\times d_{v}}\end{aligned}

where \color[rgb]{0,0,1}\overleftarrow{{\bm{q}}_{[t]}^{r}}=\gamma_{[t]}^{r}\color[rgb]{0,0,0}{\bm{q}}_{[t]}^{r}, \color[rgb]{0,0,1}\overleftarrow{\mathbf{w}_{[t]}^{r}}=\gamma_{[t]}^{r}\color[rgb]{0,0,0}\mathbf{w}_{[t]}^{r}, \color[rgb]{0,0,1}\overrightarrow{{\bm{k}}_{[t]}^{r}}=\frac{\gamma_{[t]}^{C}}{\gamma_{[t]}^{r}}\color[rgb]{0,0,0}{\bm{k}}_{[t]}^{r}, and \color[rgb]{0,0,1}\overrightarrow{{\mathbf{S}}_{[t]}}=\gamma_{[t]}^{C}\color[rgb]{0,0,0}{\mathbf{S}}_{[t]} like we defined in Eq. 2.

The basic Gated DeltaNet follows Llama’s macro architecture, stacking token mixer layers with SwiGLU MLP layers, but replaces self-attention with gated delta rule token mixing. Fig. 1 (right) shows its block design. For the gated delta rule (Eq. 10), queries, keys and values {q,k,v}\{{\bm{q}},{\bm{k}},{\bm{v}}\} are generated through linear projection, short convolution and SiLU, with L2 normalization applied to q,k{\bm{q}},{\bm{k}} for training stability. α,β\alpha,\beta use linear projection only.4^{4} Following Sun et al. (2023a), the output is processed through normalization and gating before applying output projection.

Figure 1
Figure 1: Visualization of the (hybrid) architecture and block design of Gated DeltaNet models. Gated DeltaNet-H1 and H2 use Gated DeltaNet + SWA and Mamba2 + Gated DeltaNet + SWA patterns, respectively. In the block design, query/key paths consist of linear proj., shortconv., SiLU and L2 norm; value path includes linear proj., shortconv. and SiLU; alpha/beta use linear proj.; and output gate applies linear proj. with SiLU.

Linear transformers have limitations in modeling local shifts and comparisons, and their fixed state size makes it hard for retrieval tasks (Arora et al., 2024a). Following recent hybrid architectures like Griffin (De et al., 2024) and Samba (Ren et al., 2024), we combine linear recurrent layers with sliding window attention (SWA), resulting in GatedDeltaNet-H1. We also stack Mamba2, GatedDeltaNet and SWA, resulting in GatedDeltaNet-H2.

Our experiments encompass a comprehensive comparison of recent state-of-the-art architectures, including pure Transformer models, RNN-based approaches, and hybrid architectures. We evaluate against the following baselines: RetNet (Sun et al., 2023a), HGRN2 (Qin et al., 2024b), Mamba (Gu & Dao, 2023), Mamba2 (Dao & Gu, 2024b), Samba (Ren et al., 2024), and DeltaNet (Yang et al., 2024b). For fair comparison, all models are trained under identical conditions with 1.3B parameters on 100B tokens sampled from the FineWeb-Edu dataset (Penedo et al., 2024). We use the AdamW optimizer with a peak learning rate of 4e-4, weight decay of 0.1, and gradient clipping of 1.0. The learning rate follows a cosine annealing schedule with a 1B token warm-up period and batch size of 0.5M tokens. All models employ the Llama2 tokenizer with a vocabulary size of 32,000. For sequence modeling, we set the training length to 4K tokens, with Samba and our hybrid models using a sliding window size of 2K. See § B.1 for evaluation settings and § B.2 for ablation studies.

Table 3
ModelWiki.LMB.LMB.PIQAHella.Wino.ARC-eARC-cSIQABoolQAvg.
ppl ↓\downarrowppl ↓\downarrowacc ↑\uparrowacc ↑\uparrowacc_n ↑\uparrowacc ↑\uparrowacc ↑\uparrowacc_n ↑\uparrowacc ↑\uparrowacc ↑\uparrow
Recurrent models
RetNet19.0817.2740.5270.0749.1654.1467.3433.7840.7860.3952.02
HGRN219.1017.6939.5470.4549.5352.8069.4035.3240.6356.6651.79
Mamba17.9215.0643.9871.3252.9152.9569.5235.4037.7661.1353.12
Mamba216.5612.5645.6671.8755.6755.2472.4737.8840.2060.1354.89
DeltaNet17.7116.8842.4670.7250.9353.3568.4735.6640.2255.2952.14
Gated DeltaNet16.4212.1746.6572.2555.7657.4571.2138.3940.6360.2455.32
Attention or hybrid models
Transformer++18.5318.3242.6070.0250.2353.5168.8335.1040.6657.0952.25
Samba16.1313.2944.9470.9453.4255.5668.8136.1739.9662.1154.00
Gated DeltaNet-H116.0712.1247.7372.5756.5358.4071.7540.1041.4063.2156.40
Gated DeltaNet-H215.9112.5548.7672.1956.8857.7771.3339.0741.9161.5556.18
Table 3: Performance comparison on language modeling and zero-shot common-sense reasoning.
View original tableOriginal Table 3
Table 4
ModelsSWDESQDFDATQANQDropAvg
Recurrent models
RetNet14.028.57.054.416.217.322.9
HGRN28.325.34.851.214.216.920.1
Mamba9.825.83.754.314.917.421.0
Mamba219.133.625.361.020.819.229.8
DeltaNet17.930.918.453.917.318.626.2
Gated DeltaNet25.434.823.760.020.019.830.6
Attention or hybrid models
Transformer++29.538.052.258.322.521.637.0
Samba33.039.250.557.723.520.237.3
Gated DeltaNet-H135.639.752.060.124.622.239.0
Gated DeltaNet-H238.240.450.763.324.823.340.1
Table 4: Accuracy on recall-world retrieval tasks with input truncated to 2K tokens. SQD: SQUADE. TQA: Trivial QA.
View original tableOriginal Table 4

In Table 3, we present the language modeling perplexity and zero-shot accuracy on commonsense reasoning benchmarks for models with 400M and 1.3B parameters. Gated DeltaNet consistently outperforms other linear models, including RetNet, HGRN2, Mamba, Mamba2, and DeltaNet, at both scales. As expected, the hybrid variant further enhances performance.

Table 4 presents results on real-world recall-intensive tasks used by Arora et al. (2024b). As expected, linear recurrent models show a significant performance gap compared to Transformers, while hybrid models combining linear recurrence and attention outperform pure attention models in retrieval tasks.

For pure recurrent models, despite DeltaNet’s superior performance on synthetic in-context retrieval tasks (Yang et al., 2024b), its real-world retrieval performance lags behind Mamba2, consistent with our observations in S-NIAH-2 and S-NIAH-3 (Table 2). Gated DeltaNet outperforms both DeltaNet and Mamba2 thanks to its gated delta rule, though the improvement margin is smaller than in Table 2. We attribute this reduced performance gap to instruction-unaligned small language models being prone to repetition errors, which are the primary source of errors in these tasks (cf. Arora et al. (2024b, Appendix E)). Since this issue is largely independent of the update rule choice, the performance differences between models are less pronounced compared to Table 2.

Figure 2
Figure 2: Length extrapolation on six long benchmarks.

As shown in Fig.2, we evaluate the models’ capacity to extrapolate to sequences of up to 20K tokens across six long-context benchmarks. Gated DeltaNet achieves the lowest overall perplexity across tasks among RNN models. While we observe mixed results in length extrapolation, Gated DeltaNet exhibits relatively more robust performance, suggesting better memory management. The hybrid models further improve upon this by leveraging attention for local context modeling, which reduces the memory management burden on their recurrent components. Future work will explore these models’ capabilities on even longer sequences.

As demonstrated in Table 5, we evaluated the models’ performance on LongBench (Bai et al., 2023). In recurrent models, Gated DeltaNet shows consistent advantages, especially in single-doc QA, few-shot in-context learning, and Code tasks, demonstrating its superior capabilities in retrieval, in-context learning, and state tracking, respectively.

Table 5
Single-Doc QAMulti-Doc QASummarizationFew-shotCodeAvg
ModelNQAQQAMFQHQA2WMMusGvRQMSMNsTRCTQASSMLCCRBP
Recurrent models
RetNet12.110.719.110.718.05.84.815.87.919.018.012.814.117.913.2
HGRN210.712.119.111.315.76.05.215.19.216.015.810.318.620.813.5
Mamba13.010.120.410.116.76.07.215.98.423.121.911.217.919.014.6
DeltaNet12.910.821.510.913.25.16.513.57.215.523.311.617.620.313.6
Mamba211.111.318.611.815.16.76.714.57.413.023.68.417.920.613.5
Gated DeltaNet14.114.023.313.714.45.87.516.47.930.022.423.018.722.116.6
Attention or hyrbid models
Transformer++11.89.310.010.94.26.17.415.86.616.913.53.917.218.711.0
Samba12.512.925.411.219.76.89.115.711.020.022.722.818.121.115.9
Gated DeltaNet-H114.512.326.612.623.66.19.116.112.833.523.926.815.519.217.8
Gated DeltaNet-H212.713.027.112.720.67.510.416.213.040.522.727.919.922.118.4
Table 5: Accuracy on 14 tasks from LongBench (Bai et al., 2023): Narrative QA, QasperQA, MultiField QA, HotpotQA, 2WikiMulti QA, Musique, GovReport, QMSum, MultiNews, TRec, Trivia QA, SamSum, LCC, and RepoBench-P by order.
View original tableOriginal Table 5
Figure 3
Figure 3: Training throughput comparison of 1.3B models on a single H100 GPU.

The training throughput comparison across different models is presented in Fig. 3. As our analysis shows, the proposed gated delta rule introduces only marginal overhead compared to the original delta rule, with Gated DeltaNet achieving essentially the same throughput as DeltaNet. Both are slightly slower than Mamba2 (2-3K tokens/sec) due to their more expressive transition matrices.

The Transformer++ achieves the best performance in the 2K context window domain, thanks to the highly optimized Flash-Attention-2 kernel (Dao, 2023). Consequently, hybrid approaches combining 2K window-size SWA attention with other token mixers demonstrate higher throughput than standalone mixers: Samba outperforms Mamba, while Gated DeltaNet-H1 and -H2 outperform Gated DeltaNet. Notably, Gated DeltaNet-H1 maintains compelling training throughput across all sequence lengths, even on short sequences.

Large linear recurrent language models have attracted significant attention due to their training and inference efficiency. The field of linear RNNs has rapidly evolved from using data-independent decay mechanisms, as exemplified by models like S4 (Gu et al., 2022), S5 (Smith et al., 2023), LRU (Orvieto et al., 2023), RWKV4/5 (Peng et al., 2023), and RetNet (Sun et al., 2023a), to incorporating data-dependent decay mechanisms in more recent architectures such as HGRN1/2 (Qin et al., 2024b; Qin et al., 2023b), Mamba1/2 (Gu & Dao, 2023; Dao & Gu, 2024a), RWKV6 (Peng et al., 2024), GSA (Zhang et al., 2024). This transition stems from the proven advantages of gating/forgetting mechanisms (termed selective mechanisms in Mamba)—a classical concept originating in the gated RNN literature (Gers et al., 2000) whose significance has been consistently reaffirmed (Greff et al., 2015; van der Westhuizen & Lasenby, 2018; Qin et al., 2024b; Qin et al., 2023b; Gu & Dao, 2023).

Modern forget gates differ from traditional designs like those in LSTM by removing the dependency on the previous hidden state, relying solely on input data. This modification enables efficient parallelism across sequence lengths (Martin & Cundy, 2018; Qin et al., 2023b). The absence of a forget gate has been a notable limitation in DeltaNet, and our gated extension addresses this gap in a natural, effective, and hardware-efficient way. We also note a recent concurrent work RWKV-7 5^{5} using a similar idea, but with a more relaxable formalism using diagonal-plus-low-rank transitions: St=St−1(diag⁡(dt)−atbt⊤)+vtkt⊤{\mathbf{S}}_{t}={\mathbf{S}}_{t-1}(\operatorname{diag}(\mathbf{d}_{t})-\mathbf{a}_{t}\mathbf{b}_{t}^{\top})+{\bm{v}}_{t}{\bm{k}}_{t}^{\top} where dt,at,bt∈Rdk\mathbf{d}_{t},\mathbf{a}_{t},\mathbf{b}_{t}\in\mathbb{R}^{d_{k}}. The chunkwise algorithm could be similarly adapted to this case, as implemented in Flash Linear Attention (Yang & Zhang, 2024). 6^{6}

The delta learning rule demonstrates superior memory capacity compared to Hebbian learning (Gardner, 1988; Prados & Kak, 1989), an advantage DeltaNet leverages while linear transformers rely on Hebbian-like rules. This memory capacity advantage is evident in synthetic in-context learning tasks and extends to language modeling (Irie et al., 2021; Yang et al., 2024b), reinforcement learning (Irie et al., 2022b), and image generation (Irie & Schmidhuber, 2023). Yang et al. (2024b) parallelized delta rule computation and demonstrated how DeltaNet’s data-dependent identity-plus-low-rank structure (I−βtktkt⊺{\mathbf{I}}-\beta_{t}{\bm{k}}_{t}{\bm{k}}_{t}^{\intercal}) offers greater flexibility than Mamba2’s data-dependent diagonal matrices (αtI\alpha_{t}{\mathbf{I}}). This structural advantage could enable complex reasoning, including regular language recognition (Fan et al., 2024; Grazzi et al., 2024) and state-tracking beyond TC0^{0} complexity (Merrill et al., 2024)—crucial for coding and reasoning applications.

Despite these significant advantages, the delta rule faces theoretical limitations (Irie et al., 2023) and shows only moderate performance on real-world datasets (Yang et al., 2024b), suggesting room for improvement. Previous attempts to enhance expressiveness through nonlinear recurrence (Irie et al., 2021; Irie et al., 2022b) addressed some limitations but sacrificed training parallelism, creating a performance-efficiency tradeoff. Recent work proposes some enhancements without compromising parallelism for better state tracking performance, including using negative eigenvalues (Grazzi et al., 2024) and multiple products of householder transition matrices (Siems et al., 2025) which enable high-rank transformations. These methods could be applied to Gated DeltaNet seamlessly.

From a (online) learning objective perspective, alternative formulations could further extend expressiveness: nonlinear regression (L(St)=12∣∣fSt(kt)−vt∣∣2\mathcal{L}({\mathbf{S}}_{t})=\frac{1}{2}||f_{{\mathbf{S}}_{t}}({\bm{k}}_{t})-{\bm{v}}_{t}||^{2}) as in TTT (Sun et al., 2024a) and Titans (Behrouz et al., 2024), where fSf_{\mathbf{S}} is a nonlinear function parameterized by S{\mathbf{S}}; or regression considering the entire history (L(St)=12∑i=1t∣∣Stki−vi∣∣2\mathcal{L}({\mathbf{S}}_{t})=\frac{1}{2}\sum_{i=1}^{t}||{\mathbf{S}}_{t}{\bm{k}}_{i}-{\bm{v}}_{i}||^{2}) as in Mesa layer (von Oswald et al., 2024)—analogous to the difference between Least Mean Square and Recursive Least Square algorithms. However, these more expressive variants introduce nonlinear recurrence and require workarounds, such as performing nonlinear updates only after processing entire chunks (as in TTT and Titans); or approximating nonlinear recurrence methods like Lim et al. (2024); Gonzalez et al. (2024); Schöne et al. (2025).

In this work, we explore interleaving hybrid attention layers across layers, which is commonly used such as in MiniMax-01 (MiniMax et al., 2025) and Hybrid Mamba2-Attention (Waleffe et al., 2024). It is also interesting to investigate hybrid linear/softmax attention within a single layer (Hua et al., 2022a; Zancato et al., 2024; Munkhdalai et al., 2024; Nunez et al., 2024; Dong et al., 2025; Zhang et al., 2025).

In this work, we introduced Gated DeltaNet, which enables better key-value association learning compared to Mamba2 and more adaptive memory clearance than DeltaNet, leading to consistently better empirical results across various tasks. We extended the parallel algorithm from  Yang et al. (2024b) to enable hardware-efficient training of Gated DeltaNet. Our hybrid Gated DeltaNet model achieves even higher training throughput and overall performance, making it well-suited for practical deployment.

We thank Yu Zhang for assistance with figure creation and model evaluation; Kazuki Irie for providing valuable feedback on the draft; Simeng Sun and Zhixuan Lin for insightful discussions on long-sequence task evaluation settings; and Eric Alcaide and Volodymyr Kyrylov for their helpful discussions on the online learning perspective of DeltaNet.

References

  1. Akyürek et al. (2024) Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-Context Language Learning: Architectures and Algorithms, 2024. URL https://arxiv.org/abs/2401.12973.Source
  2. Andriushchenko et al. (2023) Maksym Andriushchenko, Francesco D’Angelo, Aditya Vardhan Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning? ArXiv, abs/2310.04415, 2023. URL https://api.semanticscholar.org/CorpusID:263829417.Source
  3. Arora et al. (2023a) Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. ArXiv preprint, abs/2312.04927, 2023a. URL https://arxiv.org/abs/2312.04927.Source
  4. Arora et al. (2023b) Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes, 2023b. URL https://arxiv.org/abs/2304.09433.Source
  5. Arora et al. (2024a) Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff. ArXiv preprint, abs/2402.18668, 2024a. URL https://arxiv.org/abs/2402.18668.Source
  6. Arora et al. (2024b) Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher Ré. Just read twice: closing the recall gap for recurrent language models, 2024b. URL https://arxiv.org/abs/2407.05483.Source
  7. Bai et al. (2023) Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. ArXiv preprint, abs/2308.14508, 2023. URL https://arxiv.org/abs/2308.14508.Source
  8. Beck et al. (2024) Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. ArXiv preprint, abs/2405.04517, 2024. URL https://arxiv.org/abs/2405.04517.Source
  9. Behrouz et al. (2024) Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time, 2024. URL https://arxiv.org/abs/2501.00663.Source
  10. Bischof & Loan (1985) Christian H. Bischof and Charles Van Loan. The WY representation for products of householder matrices. In SIAM Conference on Parallel Processing for Scientific Computing, 1985. URL https://api.semanticscholar.org/CorpusID:36094006.Source
  11. Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432–7439. AAAI Press, 2020. URL https://aaai.org/ojs/index.php/AAAI/article/view/6239.Source
  12. Chou et al. (2024) Yuhong Chou, Man Yao, Kexin Wang, Yuqi Pan, Rui-Jie Zhu, Jibin Wu, Yiran Zhong, Yu Qiao, Bo XU, and Guoqi Li. MetaLA: Unified optimal linear approximation to softmax attention map. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=Y8YVCOMEpz.Source
  13. Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2924–2936, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1300. URL https://aclanthology.org/N19-1300.Source
  14. Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv preprint, abs/1803.05457, 2018. URL https://arxiv.org/abs/1803.05457.Source
  15. Dao (2023) Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. ArXiv preprint, abs/2307.08691, 2023. URL https://arxiv.org/abs/2307.08691.Source
  16. Dao & Gu (2024a) Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv: 2405.21060, 2024a.
  17. Dao & Gu (2024b) Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 10041–10071. PMLR, 2024b. URL https://proceedings.mlr.press/v235/dao24a.html.Source
  18. Dasigi et al. (2021) Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4599–4610, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.365. URL https://aclanthology.org/2021.naacl-main.365.Source
  19. De et al. (2024) Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models, 2024. URL https://arxiv.org/abs/2402.19427.Source
  20. Dong et al. (2025) Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, ZIJIA CHEN, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Celine Lin, Jan Kautz, and Pavlo Molchanov. Hymba: A hybrid-head architecture for small language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=A1ztozypga.Source
  21. Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2368–2378, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1246. URL https://aclanthology.org/N19-1246.Source
  22. Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074–1084, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1102. URL https://aclanthology.org/P19-1102.Source
  23. Fan et al. (2024) Ting-Han Fan, Ta-Chung Chi, and Alexander Rudnicky. Advancing regular language reasoning in linear recurrent neural networks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pp. 45–53, Mexico City, Mexico, 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.naacl-short.4.Source
  24. Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2021.
  25. Gardner (1988) E. Gardner. The space of interactions in neural network models. Journal of Physics A, 21:257–270, 1988. URL https://api.semanticscholar.org/CorpusID:15378089.Source
  26. Gers et al. (2000) Felix A. Gers, Jürgen Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with LSTM. Neural Comput., 12(10):2451–2471, 2000.
  27. Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 70–79, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-5409. URL https://aclanthology.org/D19-5409.Source
  28. Gonzalez et al. (2024) Xavier Gonzalez, Andrew Warrington, Jimmy T.H. Smith, and Scott Linderman. Towards scalable and stable parallelization of nonlinear RNNs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=hBCxxVQDBw.Source
  29. Grazzi et al. (2024) Riccardo Grazzi, Julien N. Siems, Jorg K. H. Franke, Arber Zela, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear rnns through negative eigenvalues. 2024. URL https://api.semanticscholar.org/CorpusID:274141450.Source
  30. Greff et al. (2015) Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems, 28:2222–2232, 2015. URL https://api.semanticscholar.org/CorpusID:3356463.Source
  31. Gu & Dao (2023) Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. 2023.
  32. Gu et al. (2022) Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openreview.net/forum?id=uYLFoz1vlAC.Source
  33. Guo et al. (2023) Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian J. McAuley. Longcoder: A long-range pre-trained language model for code completion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 12098–12107. PMLR, 2023. URL https://proceedings.mlr.press/v202/guo23j.html.Source
  34. He et al. (2025) Zhihao He, Hang Yu, Zi Gong, Shizhan Liu, Jianguo Li, and Weiyao Lin. Rodimus*: Breaking the accuracy-efficiency trade-off with efficient attentions. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=IIVYiJ1ggK.Source
  35. Ho et al. (2020) Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609–6625, Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.580. URL https://aclanthology.org/2020.coling-main.580.Source
  36. Hsieh et al. (2024) Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? ArXiv preprint, abs/2404.06654, 2024. URL https://arxiv.org/abs/2404.06654.Source
  37. Hua et al. (2022a) Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V. Le. Transformer quality in linear time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9099–9117. PMLR, 2022a. URL https://proceedings.mlr.press/v162/hua22a.html.Source
  38. Hua et al. (2022b) Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V. Le. Transformer quality in linear time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9099–9117. PMLR, 2022b. URL https://proceedings.mlr.press/v162/hua22a.html.Source
  39. Huang et al. (2021) Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1419–1436, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.112. URL https://aclanthology.org/2021.naacl-main.112.Source
  40. Irie & Schmidhuber (2023) Kazuki Irie and Jürgen Schmidhuber. Images as weight matrices: Sequential image generation through synaptic learning rules. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=ddad0PNUvV.Source
  41. Irie et al. (2021) Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear transformers with recurrent fast weight programmers. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 7703–7717, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/3f9e3767ef3b10a0de4c256d7ef9805d-Abstract.html.Source
  42. Irie et al. (2022a) Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9639–9659. PMLR, 2022a. URL https://proceedings.mlr.press/v162/irie22a.html.Source
  43. Irie et al. (2022b) Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. A modern self-referential weight matrix that learns to modify itself. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9660–9677. PMLR, 2022b. URL https://proceedings.mlr.press/v162/irie22b.html.Source
  44. Irie et al. (2023) Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. Practical computational power of linear transformers and their recurrent and self-referential extensions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9455–9465, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.588. URL https://aclanthology.org/2023.emnlp-main.588.Source
  45. Jelassi et al. (2024) Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat After Me: Transformers are Better than State Space Models at Copying. ArXiv preprint, abs/2402.01032, 2024. URL https://arxiv.org/abs/2402.01032.Source
  46. Joffrain et al. (2006) Thierry Joffrain, Tze Meng Low, Enrique S. Quintana-Ortí, Robert A. van de Geijn, and Field G. Van Zee. Accumulating householder transformations, revisited. ACM Trans. Math. Softw., 32:169–179, 2006. URL https://api.semanticscholar.org/CorpusID:15723171.Source
  47. Joshi et al. (2017a) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611, Vancouver, Canada, 2017a. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/P17-1147.Source
  48. Joshi et al. (2017b) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611, Vancouver, Canada, 2017b. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/P17-1147.Source
  49. Katharopoulos et al. (2020a) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5156–5165. PMLR, 2020a. URL http://proceedings.mlr.press/v119/katharopoulos20a.html.Source
  50. Katharopoulos et al. (2020b) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5156–5165. PMLR, 2020b. URL http://proceedings.mlr.press/v119/katharopoulos20a.html.Source
  51. Kočiský et al. (2018) Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018. doi: 10.1162/tacl_a_00023. URL https://aclanthology.org/Q18-1023.Source
  52. Krogh & Hertz (1991) Anders Krogh and John A. Hertz. A simple weight decay can improve generalization. In Neural Information Processing Systems, 1991. URL https://api.semanticscholar.org/CorpusID:10137788.Source
  53. Kulis & Bartlett (2010) Brian Kulis and Peter L. Bartlett. Implicit online learning. In Johannes Fürnkranz and Thorsten Joachims (eds.), Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pp. 575–582. Omnipress, 2010. URL https://icml.cc/Conferences/2010/papers/429.pdf.Source
  54. Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026.Source
  55. Li & Roth (2002) Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002. URL https://aclanthology.org/C02-1150.Source
  56. Lim et al. (2024) Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim. Parallelizing non-linear sequential models over the sequence length, 2024. URL https://arxiv.org/abs/2309.12252.Source
  57. Liu et al. (2024) Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. @articleDBLP:journals/corr/abs-2407-14207, author = Bo Liu and Rui Wang and Lemeng Wu and Yihao Feng and Peter Stone and Qiang Liu, title = Longhorn: State Space Models are Amortized Online Learners, journal = CoRR, volume = abs/2407.14207, year = 2024, url = https://doi.org/10.48550/arXiv.2407.14207, doi = 10.48550/ARXIV.2407.14207, eprinttype = arXiv, eprint = 2407.14207, timestamp = Fri, 23 Aug 2024 08:12:16 +0200, biburl = https://dblp.org/rec/journals/corr/abs-2407-14207.bib, bibsource = dblp computer science bibliography, https://dblp.org : State space models are amortized online learners. ArXiv preprint, abs/2407.14207, 2024. URL https://arxiv.org/abs/2407.14207.Source
  58. Liu et al. (2023) Tianyang Liu, Canwen Xu, and Julian McAuley. RepoBench: Benchmarking repository-level code auto-completion systems. ArXiv preprint, abs/2306.03091, 2023. URL https://arxiv.org/abs/2306.03091.Source
  59. Lockard et al. (2019) Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. OpenCeres: When open information extraction meets the semi-structured web. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3047–3056, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1309. URL https://aclanthology.org/N19-1309.Source
  60. Lu et al. (2025) Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh, Boxing Chen, and Philippe Langlais. Regla: Refining gated linear attention, 2025. URL https://arxiv.org/abs/2502.01578.Source
  61. Martin & Cundy (2018) Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=HyUNwulC-.Source
  62. Merity et al. (2017) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=Byj72udxe.Source
  63. Merrill et al. (2024) William Merrill, Jackson Petty, and Ashish Sabharwal. The Illusion of State in State-Space Models, 2024. URL https://arxiv.org/abs/2404.08819.Source
  64. MiniMax et al. (2025) MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, and Zijia Wu. Minimax-01: Scaling foundation models with lightning attention, 2025. URL https://arxiv.org/abs/2501.08313.Source
  65. Munkhdalai et al. (2024) Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. ArXiv preprint, abs/2404.07143, 2024. URL https://arxiv.org/abs/2404.07143.Source
  66. Nunez et al. (2024) Elvis Nunez, Luca Zancato, Benjamin Bowman, Aditya Golatkar, Wei Xia, and Stefano Soatto. Expansion span: Combining fading memory and retrieval in hybrid state space models, 2024. URL https://arxiv.org/abs/2412.13328.Source
  67. Orvieto et al. (2023) Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Çaglar Gülçehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 26670–26698. PMLR, 2023. URL https://proceedings.mlr.press/v202/orvieto23a.html.Source
  68. Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525–1534, Berlin, Germany, 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1144. URL https://aclanthology.org/P16-1144.Source
  69. Penedo et al. (2024) Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. ArXiv preprint, abs/2406.17557, 2024. URL https://arxiv.org/abs/2406.17557.Source
  70. Peng et al. (2023) Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14048–14077, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.936. URL https://aclanthology.org/2023.findings-emnlp.936.Source
  71. Peng et al. (2024) Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Stanisław Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu. Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence, 2024. URL https://arxiv.org/abs/2404.05892.Source
  72. Peng et al. (2021) Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. Random feature attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=QtTKTdVrFBB.Source
  73. Prados & Kak (1989) DL Prados and SC Kak. Neural network capacity using delta rule. Electronics Letters, 3(25):197–199, 1989.
  74. Qin et al. (2023a) Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Fei Yuan, Xiao Luo, Y. Qiao, and Yiran Zhong. Transnormerllm: A faster and better large language model with improved transnormer. 2023a. URL https://api.semanticscholar.org/CorpusID:260203124.Source
  75. Qin et al. (2023b) Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023b. URL http://papers.nips.cc/paper_files/paper/2023/hash/694be3548697e9cc8999d45e8d16fe1e-Abstract-Conference.html.Source
  76. Qin et al. (2024a) Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models. 2024a.
  77. Qin et al. (2024b) Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion. ArXiv preprint, abs/2404.07904, 2024b. URL https://arxiv.org/abs/2404.07904.Source
  78. Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2124. URL https://aclanthology.org/P18-2124.Source
  79. Ren et al. (2024) Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. ArXiv preprint, abs/2406.07522, 2024. URL https://arxiv.org/abs/2406.07522.Source
  80. Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8732–8740. AAAI Press, 2020. URL https://aaai.org/ojs/index.php/AAAI/article/view/6399.Source
  81. Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4463–4473, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1454. URL https://aclanthology.org/D19-1454.Source
  82. Schlag et al. (2021a) Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9355–9366. PMLR, 2021a. URL http://proceedings.mlr.press/v139/schlag21a.html.Source
  83. Schlag et al. (2021b) Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9355–9366. PMLR, 2021b. URL http://proceedings.mlr.press/v139/schlag21a.html.Source
  84. Schöne et al. (2025) Mark Schöne, Babak Rahmani, Heiner Kremer, Fabian Falck, Hitesh Ballani, and Jannes Gladrow. Implicit language models are rnns: Balancing parallelization and expressivity, 2025. URL https://arxiv.org/abs/2502.07827.Source
  85. Siems et al. (2025) Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. Deltaproduct: Increasing the expressivity of deltanet through products of householders, 2025. URL https://arxiv.org/abs/2502.10297.Source
  86. Smith et al. (2023) Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=Ai8Hw3AXqks.Source
  87. Smolensky (1990) Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artif. Intell., 46(1-2):159–216, 1990. doi: 10.1016/0004-3702(90)90007-M. URL [https://doi.org/10.1016/0004-3702(90)90007-M](https://doi.org/10.1016/0004-3702(90)90007-M).Source
  88. Sun et al. (2024a) Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): Rnns with expressive hidden states. ArXiv preprint, abs/2407.04620, 2024a. URL https://arxiv.org/abs/2407.04620.Source
  89. Sun et al. (2023a) Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. ArXiv preprint, abs/2307.08621, 2023a. URL https://arxiv.org/abs/2307.08621.Source
  90. Sun et al. (2023b) Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. ArXiv preprint, abs/2307.08621, 2023b. URL https://arxiv.org/abs/2307.08621.Source
  91. Sun et al. (2024b) Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. ArXiv preprint, abs/2405.05254, 2024b. URL https://arxiv.org/abs/2405.05254.Source
  92. Trivedi et al. (2022) Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. doi: 10.1162/tacl_a_00475. URL https://aclanthology.org/2022.tacl-1.31.Source
  93. van der Westhuizen & Lasenby (2018) Jos van der Westhuizen and Joan Lasenby. The unreasonable effectiveness of the forget gate. ArXiv preprint, abs/1804.04849, 2018. URL https://arxiv.org/abs/1804.04849.Source
  94. von Oswald et al. (2024) Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklasson, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymyrov, Razvan Pascanu, and João Sacramento. Uncovering mesa-optimization algorithms in transformers, 2024. URL https://arxiv.org/abs/2309.05858.Source
  95. Waleffe et al. (2024) Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro. An empirical study of mamba-based language models, 2024. URL https://arxiv.org/abs/2406.07887.Source
  96. Wang et al. (2025) Ke Alexander Wang, Jiaxin Shi, and Emily B. Fox. Test-time regression: a unifying framework for designing sequence models with associative memory, 2025. URL https://arxiv.org/abs/2501.12352.Source
  97. Wen et al. (2024) Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval. ArXiv preprint, abs/2402.18510, 2024. URL https://arxiv.org/abs/2402.18510.Source
  98. Widrow et al. (1960) Bernard Widrow, Marcian E Hoff, et al. Adaptive switching circuits. In IRE WESCON convention record, volume 4, pp. 96–104. New York, 1960.
  99. Yang & Zhang (2024) Songlin Yang and Yu Zhang. FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism, 2024. URL https://github.com/sustcsonglin/flash-linear-attention. original-date: 2023-12-20T06:50:18Z.Source
  100. Yang et al. (2024a) Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 56501–56523. PMLR, 2024a. URL https://proceedings.mlr.press/v235/yang24ab.html.Source
  101. Yang et al. (2024b) Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. NeurIPS, 2024b.
  102. Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259.Source
  103. Zancato et al. (2024) Luca Zancato, Arjun Seshadri, Yonatan Dukler, Aditya Golatkar, Yantao Shen, Benjamin Bowman, Matthew Trager, Alessandro Achille, and Stefano Soatto. B’MOJO: Hybrid state space realizations of foundation models with eidetic and fading memory. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=RnQdRY1h5v.Source
  104. Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472. URL https://aclanthology.org/P19-1472.Source
  105. Zhang et al. (2025) Michael Zhang, Simran Arora, Rahul Chalamala, Benjamin Frederick Spector, Alan Wu, Krithik Ramesh, Aaryan Singhal, and Christopher Re. LoLCATs: On low-rank linearizing of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=8VtGeyJyx9.Source
  106. Zhang et al. (2024) Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, Peng Zhou, and Guohong Fu. Gated slot attention for efficient linear-time sequence modeling. 2024. URL https://api.semanticscholar.org/CorpusID:272593079.Source
  107. Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5905–5921, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.472. URL https://aclanthology.org/2021.naacl-main.472.Source

To reduce notation clutter, we only consider the first chunk here.

For St{\mathbf{S}}_{t}, the extended WY representation is

\displaystyle{\mathbf{S}}_{t}=\sum_{i=1}^{t}{\color[rgb]{0,0,1}\frac{\gamma_{t}}{\gamma_{i}}}\mathbf{u}_{i}{\bm{k}}_{i}^{\intercal},\qquad\mathbf{u}_{t}=\beta_{t}\left({\bm{v}}_{t}-\sum_{i=1}^{t-1}{\color[rgb]{0,0,1}\frac{\gamma_{t}}{\gamma_{i}}}\mathbf{u}_{i}{\bm{k}}_{i}^{T}{\bm{k}}_{t}\right)

We proof this by mathmetical induction.

Proof.
\begin{aligned}\displaystyle\centering{\mathbf{S}}_{t+1}\@add@centering & \displaystyle={\mathbf{S}}_{t}\left({\color[rgb]{0,0,1}\alpha_{t+1}}({\mathbf{I}}-\beta_{t+1}{\bm{k}}_{t+1}{\bm{k}}_{t+1}^{\intercal})\right)+\beta_{t+1}{\bm{v}}_{t+1}{\bm{k}}_{t+1}^{\intercal}\end{aligned}
\begin{aligned} & \displaystyle={\color[rgb]{0,0,1}\alpha_{t+1}}(\sum_{i=1}^{t}{\color[rgb]{0,0,1}\frac{\gamma_{t}}{\gamma_{i}}}\mathbf{u}_{i}{\bm{k}}_{i}^{\intercal})-{\color[rgb]{0,0,1}\alpha_{t+1}}\beta_{t+1}(\sum_{i=1}^{t}{\color[rgb]{0,0,1}\frac{\gamma_{t}}{\gamma_{i}}}\mathbf{u}_{i}{\bm{k}}_{i}^{\intercal}{\bm{k}}_{i}{\bm{k}}_{t+1}^{\intercal})+\beta_{t+1}{\bm{v}}_{t+1}{\bm{k}}_{t+1}^{\intercal}\end{aligned}
\begin{aligned} & \displaystyle=\sum_{i=1}^{t}{\color[rgb]{0,0,1}\frac{\gamma_{t+1}}{\gamma_{i}}}\mathbf{u}_{i}{\bm{k}}_{i}^{\intercal}+\underbrace{\beta_{t+1}\left({\bm{v}}_{t+1}-\sum_{i=1}^{t}{\color[rgb]{0,0,1}\frac{\gamma_{t+1}}{\gamma_{i}}}\mathbf{u}_{i}{\bm{k}}_{i}^{T}{\bm{k}}_{t+1}\right)}_{\mathbf{u}_{t+1}}{\bm{k}}_{t+1}^{\intercal}\end{aligned}
\begin{aligned} & \displaystyle=\sum_{i=1}^{t}{\color[rgb]{0,0,1}\frac{\gamma_{t+1}}{\gamma_{i}}}\mathbf{u}_{i}{\bm{k}}_{i}^{\intercal}+\underbrace{{\color[rgb]{0,0,1}\frac{\gamma_{t+1}}{\gamma_{t+1}}}}_{1}\mathbf{u}_{t+1}{\bm{k}}_{t+1}^{\intercal}\end{aligned}
\begin{aligned} & \displaystyle=\sum_{i=1}^{t+1}{\color[rgb]{0,0,1}\frac{\gamma_{t+1}}{\gamma_{i}}}\mathbf{u}_{i}{\bm{k}}_{i}^{\intercal}\end{aligned}

∎

Following Gu & Dao (2023), we evaluate our model on multiple commonsense reasoning benchmarks: PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019, Hella.;), WinoGrande (Sakaguchi et al., 2020, Wino.;), ARC-easy (ARC-e) and ARC-challenge (ARC-c) (Clark et al., 2018), SIQA (Sap et al., 2019), BoolQ (Clark et al., 2019), Wikitext (Merity et al., 2017, Wiki.;), and LAMBADA (Paperno et al., 2016, LMB.;).

Our evaluation comprises both synthetic and real-world tasks. For synthetic tasks, we utilize the Needle-In-A-Haystack Single (NIAH-S) benchmark suite from RULER (Hsieh et al., 2024), which includes three increasingly complex tasks: S-NIAH-1 (passkey retrieval), S-NIAH-2 (numerical needle in haystack), and S-NIAH-3 (word-based needle in haystack). For real-world tasks, following Arora et al. (2024b), we evaluate on diverse datasets: SWDE (Lockard et al., 2019) for structured HTML relation extraction, FDA (Arora et al., 2023b) for PDF key-value retrieval, and several question-answering datasets including SQuAD (Rajpurkar et al., 2018), TriviaQA (Joshi et al., 2017a), Drop (Dua et al., 2019), and NQ (Kwiatkowski et al., 2019). Since our pretrained models lack instruction tuning, we employ the Cloze Completion Formatting prompts provided by Arora et al. (2024b), which better align with our models’ next-word-prediction training objective.

We evaluate on 14 tasks from Longbench (Bai et al., 2023), encompassing: narrative comprehension (Narrative QA (Kočiský et al., 2018)), scientific understanding (QasperQA (Dasigi et al., 2021)), multi-hop reasoning (MultiField QA, HotpotQA (Yang et al., 2018), 2WikiMulti QA (Ho et al., 2020), Musique (Trivedi et al., 2022)), document summarization (GovReport (Huang et al., 2021), QMSum (Zhong et al., 2021), MultiNews (Fabbri et al., 2019)), and various specialized tasks (TRec (Li & Roth, 2002), Trivia QA (Joshi et al., 2017b), SamSum (Gliwa et al., 2019), LCC (Guo et al., 2023), and RepoBench-P (Liu et al., 2023)).

Table S.1
Gated DeltaNet Ablations (400M)Avg-PPL (↓{\downarrow})Avg-Acc (↑{\uparrow})
Gated DeltaNet w Head Dim 128,27.3547.26
Macro Design
w. naive Delta Rule30.8745.12
w/o. Short Conv28.9546.16
w/o. Output Gate29.1245.46
w/o. Output Norm27.5547.07
Normalization & Feature Map
w. L1L_{1}-norm & ReLU30.7945.92
w. L1L_{1}-norm & 1+ELU30.3446.05
w. L1L_{1}-norm & SiLU30.1846.09
w. L2L_{2}-norm & ReLU27.6746.94
w. L2L_{2}-norm & 1+ELU27.5847.17
Model Dimensions
w. Head Dim 6428.3146.35
w. Head Dim 25627.1347.38
Table S.1: Ablation study on the Gated DeltaNet block. Avg-PPL and Avg-Acc denote average perplexity and zero-shot commonsense reasoning accuracy (as in Table 3), respectively. All models have 400M parameters and are trained for 15B tokens on the same subset of FineWeb-Edu dataset (Penedo et al., 2024).
View original tableOriginal Table S.1
Table S.2
ModelWiki.LMB.LMB.PIQAHella.Wino.ARC-eARC-cSIQABoolQAvg.
ppl ↓\downarrowppl ↓\downarrowacc ↑\uparrowacc ↑\uparrowacc_n ↑\uparrowacc ↑\uparrowacc ↑\uparrowacc_n ↑\uparrowacc ↑\uparrowacc ↑\uparrow
Hybrid Ablations (500M/15B)
Gated DeltaNet + SWA + Mamba224.0228.2034.7767.0840.8450.7460.3528.8338.9461.4947.88
Gated DeltaNet + Mamba2 + SWA23.6926.8336.1767.5141.5151.8561.1929.7738.5853.7347.54
Mamba2 + SWA + Gated DeltaNet24.1425.2136.7964.9641.1852.0160.9030.0338.0759.4447.92
Mamba2 + Gated DeltaNet + SWA23.5424.1136.9266.4841.7052.7261.0630.5439.9160.5148.73
Table S.2: Ablation studies of Gated DeltaNet models. All evaluations are performed by using lm-evaluation-harness (Gao et al., 2021). All models use the Llama tokenizer and are trained on the same subset of the FineWeb-Edu dataset (Penedo et al., 2024).
View original tableOriginal Table S.2

Table S.1 presents ablation studies on the Gated DeltaNet block’s components. Our experiments demonstrate that both the short convolution and output gate are crucial for model performance, while output normalization yields marginal improvements. Consistent with Yang et al. (2024b), we found L2 normalization to be essential for optimal performance, though the choice of feature map was less influential. Nevertheless, SiLU consistently outperformed other activation functions, aligning with observations from Qin et al. (2023a). Through empirical analysis, we determined that a head dimension of 128 provides an optimal trade-off between performance and computational efficiency. Additionally, Table S.2 demonstrates that among various hybrid architectures, the combination of Mamba2, Gated DeltaNet, and SWA in this specific order produces superior results.