RNNs, from LSTM to linear attention: why recurrence lost and how it came back

Before transformers, sequence models were recurrent: they read one token at a time and carried a fixed-size state forward. Transformers replaced them almost completely, and then the best ideas from recurrence quietly came back. The model on my inference box is a hybrid with mostly recurrent layers. This is part 4 of the architecture series, after decoder-only, encoder-only and encoder-decoder.

The recurrent update

A vanilla RNN keeps a hidden state h and updates it with every input:

h_t = tanh( W_hh · h_{t−1}  +  W_xh · x_t  +  b )

The state is fixed-size whatever the sequence length, so inference memory is constant and each step costs the same. That’s the property transformers gave up, and the reason recurrence keeps coming back.

Why it was hard to train

Training uses backpropagation through time: the gradient of a late output with respect to an early input passes through every step in between, multiplying the step Jacobians together. Roughly, it scales like a per-step factor raised to the number of steps:

0.9¹⁰⁰ ≈ 2.7 × 10⁻⁵        vanishing: early inputs stop influencing learning
1.1¹⁰⁰ ≈ 13,781            exploding: training blows up

Bengio, Simard and Frasconi (1994) and Hochreiter (1991) analysed this. A plain RNN can’t reliably learn dependencies over more than a few dozen steps. Exploding gradients have a cheap fix, gradient clipping. Vanishing gradients needed a new architecture.

LSTM: a gated highway for the gradient

The LSTM (Hochreiter and Schmidhuber, 1997, with the forget gate added by Gers et al., 2000) adds a cell state c that’s updated additively, under the control of learned gates:

f, i, o = σ(…)                      forget, input, output gates (0…1)
c_t = f ⊙ c_{t−1}  +  i ⊙ candidate_t
h_t = o ⊙ tanh(c_t)

When the forget gate is near 1, the cell state, and its gradient, pass through a step almost unchanged. The gradient no longer has to survive a hundred multiplications by a squashing function. The GRU (Cho et al., 2014) simplifies the same idea to two gates.

LSTMs powered machine translation, speech recognition and handwriting recognition for most of the 2010s. They’re still a sensible choice for small time-series problems; my lstm repo grid-searches one for forecasting.

Why transformers won

The fatal limitation wasn’t quality, it was parallelism. h_t depends on h_{t−1}, so a 1,000-token sequence takes 1,000 sequential steps to train, no matter how many GPU cores are idle. A transformer processes all positions of a training sequence at once, and attention gives every token a direct, one-step path to every other token, with no long chain for the gradient to cross. On GPUs that’s decisive: a lot more data for the same compute.

The transformer’s price was memory at inference: the KV cache grows with every token, and every decode step reads it (the KV cache note has the numbers).

Recurrence comes back, trainable in parallel

The modern revival keeps the constant-memory state but makes the recurrence linear in the state, which allows parallel training:

• State-space models. S4 (Gu et al., 2021) and Mamba (Gu and Dao, 2023) use structured linear recurrences. They train with parallel scans or convolutions and run as an RNN at inference. Mamba makes the recurrence depend on the input (selective), which is what made it competitive on language.
• RWKV (Peng et al., 2023): an RNN formulation with transformer-style training.
• Linear attention and DeltaNet: attention rewritten as a recurrent state update, trained in chunks. Gated DeltaNet adds decay and a delta-rule update (the hybrid models note has the math).

A growing number of recent models, such as Jamba and Qwen3-Next, use these as hybrids: many recurrent layers for cheap, constant-memory context, plus a few full-attention layers for exact recall. That’s the shape of the model I run every day, and why its 262K-token context fits on one 24 GB card at all.

Rules I use

1. Constant memory per token is the reason to want recurrence. If context length isn’t your constraint, plain attention is simpler.
2. Expect hybrids, not pure RNNs, for language at scale. Exact recall still needs some attention.
3. For small sequential data, an LSTM is still a cheap, strong baseline before reaching for a transformer.

Sources: Hochreiter, Untersuchungen zu dynamischen neuronalen Netzen (1991). Bengio, Simard and Frasconi, Learning long-term dependencies with gradient descent is difficult (1994). Hochreiter and Schmidhuber, Long Short-Term Memory (1997). Gers, Schmidhuber and Cummins, Learning to Forget (2000). Cho et al., GRU (2014). Gu et al., S4 (2021). Gu and Dao, Mamba (2023). Peng et al., RWKV (2023).