Hybrid models: what Gated DeltaNet replaces in the KV cache

The model on my inference box isn’t a pure transformer. Most of its layers are Gated DeltaNet, a form of linear attention with a fixed-size state, and only some layers are full attention with a KV cache. That’s why 262K context fits at all. It’s also why several fixes in my fork exist: a recurrent state behaves very differently from a cache.

The cost being replaced

Softmax attention keeps every past key and value. The cache grows linearly with context, and every decode step reads all of it. One head with a head dimension of 128 adds 2 × 128 = 256 cached values per token, forever.

Linear attention: a fixed-size state

Drop the softmax and use a feature map φ on queries and keys (Katharopoulos et al., 2020). Attention becomes associative, so the sum over the past can be kept as a running matrix:

S_t = S_{t−1} + v_t · φ(k_t)ᵀ          state: d_v × d_k per head
o_t = S_t · φ(q_t)

The state is a fixed d_v × d_k matrix per head: 128 × 128 = 16,384 values, whatever the context length. One softmax head’s cache passes that after 16,384 / 256 = 64 tokens. At 262K tokens it’s 4,096 times larger.

The catch is that a purely additive state can’t forget. Every key–value association is summed in, old ones interfere with new ones, and recall degrades as the sequence grows.

The delta rule: write the correction, not the sum

DeltaNet (Schlag et al., 2021; made parallel by Yang et al., 2024) updates the state like an online regression. It reads what the state currently predicts for k_t, then writes only the error:

S_t = S_{t−1} + β_t · (v_t − S_{t−1} k_t) · k_tᵀ

If the state already maps k_t to v_t, nothing changes. If it maps it to something stale, the old association is overwritten. β_t ∈ (0, 1) is a learned, per-token write strength.

Gated DeltaNet: add decay

Gated DeltaNet (Yang, Kautz and Hatamizadeh, 2025) adds a learned decay gate α_t ∈ (0, 1), so the state can also forget wholesale when the context moves on:

S_t = α_t · S_{t−1} · (I − β_t k_t k_tᵀ) + β_t · v_t k_tᵀ

Decay handles “this section is over”; the delta update handles “this specific fact changed”. Together they make the fixed state usable over long sequences.

Why hybrids keep some attention

A fixed-size state is a lossy summary. For exact retrieval, such as copying a variable name defined 100K tokens ago, softmax attention over the full cache is still better. So hybrids interleave the two: mostly Gated DeltaNet layers, plus a minority of full-attention layers that keep a KV cache. Qwen3-Next published a 3:1 ratio of Gated DeltaNet to attention layers.

That’s why my 27B model’s cache works out to 32,768 values per token. Only its attention layers contribute; the DeltaNet layers add nothing that grows.

What a recurrent state changes for serving

The cache is easy to manipulate: drop entries and you’ve rewound time. A recurrent state is a single matrix that has already folded everything in. Three consequences:

1. Prefix reuse needs snapshots. With a pure cache, reusing a shared prefix means keeping the first N entries. With a recurrent state, the state at position N can’t be recovered from a later one. You need a snapshot taken at N. My fork keeps up to 32 prompt checkpoints of about 149 MiB each. I made capturing them asynchronous so they stop stalling prefill; the fork note covers it.
2. Speculative decoding needs real rollback. When draft tokens are rejected, a cache just discards their entries. A recurrent state has already absorbed them, so the engine needs the pre-draft state back. Several of my MTP fixes are exactly this: rollback for rejected drafts, and re-syncing pending state when memory changes.
3. Long runs expose numerics. A state updated for hundreds of thousands of steps accumulates error in a way that a cache of independent entries doesn’t. One of my fixes stops recurrent-state drift at high context, which otherwise ended in a dead loop.

Prefill takes a chunked path. Within a chunk of tokens the updates are computed as dense matrix multiplies, and only the hand-off between chunks is sequential. I added a fused kernel for that chunk pipeline.

Rules I use

1. Count only attention layers when sizing the cache of a hybrid. The KV formula still holds; n_attn_layers is just smaller.
2. Budget host RAM for checkpoints if you serve multi-turn sessions on a hybrid. Prefix reuse is no longer free.
3. Test speculative decoding on hybrids specifically. Rollback is where engine bugs hide.
4. Evaluate exact recall at length. It’s the thing the fixed-size state is worst at, and the reason the attention layers exist.

Sources: Katharopoulos et al., Transformers are RNNs (ICML 2020). Schlag et al., Linear Transformers Are Secretly Fast Weight Programmers (ICML 2021). Yang et al., Parallelizing Linear Transformers with the Delta Rule (NeurIPS 2024). Yang, Kautz and Hatamizadeh, Gated Delta Networks (ICLR 2025). The fixes are in my fork.