Encoder-decoder transformers: cross-attention, T5, Whisper, and when they still win

The transformer in Attention Is All You Need (Vaswani et al., 2017) wasn’t decoder-only. It was an encoder and a decoder joined by cross-attention, built for translation. The layout lost the general-LLM race, but it’s still the right tool for a class of jobs, and the speech model you probably use daily is one. This is part 3 of the architecture series, after decoder-only and encoder-only.

Two stacks and a bridge

• The encoder reads the whole input with bidirectional attention, like BERT, and produces one representation per input position.
• The decoder generates the output left to right, with causal self-attention over what it has written so far.
• Cross-attention connects the two. In every decoder block, the queries come from the decoder and the keys and values come from the encoder’s output. Each generated token can look at any part of the input.

decoder block:
  x = x + CausalSelfAttention(Norm(x))
  x = x + CrossAttention(Norm(x), encoder_output)     the bridge
  x = x + MLP(Norm(x))

Where the parameters go

The cross-attention adds another set of four projections to every decoder block, so a decoder block is about 16·d² against an encoder block’s 12·d². T5-base (Raffel et al., 2020) has 12 encoder blocks and 12 decoder blocks at d = 768:

encoder     12 × 12 × 768²  =  84.9M
decoder     12 × 16 × 768²  = 113.2M
embeddings  32,128 × 768    =  24.7M
total                       ≈ 222.9M    (published: ~220M)

Training: span corruption

T5 trains with span corruption. About 15% of the input tokens are dropped in contiguous spans, with a mean span length of 3, and each span is replaced with a sentinel token. The decoder’s job is to output the missing spans, each after its sentinel. It’s denoising: the encoder sees a damaged input, and the decoder writes back only what’s missing, which keeps targets short. BART (Lewis et al., 2020) uses a related family of noising schemes and reconstructs the full original text.

The serving advantage: read once

At inference, the encoder runs exactly once over the input. Its outputs, and therefore the cross-attention keys and values, are fixed for the whole generation and can be cached. The decoder then only grows its own, much shorter, self-attention cache.

That’s a real efficiency when the input is long and the output short:

• Speech recognition. Whisper (Radford et al., 2022) encodes 30 seconds of audio once, then decodes a few dozen text tokens from it.
• Translation. NLLB, MarianMT and much of production machine translation are encoder-decoder models.
• Summarisation and extraction with a long input and a short output.

The encoder also reads the input bidirectionally, which a decoder-only model never does with its prompt.

Why decoder-only won anyway

For general-purpose models, the advantages of the decoder-only design added up:

• one stack instead of two, and one objective, next-token prediction, that uses every token of raw text;
• no fixed split between “input” and “output”, so chat, tools and multi-turn context are just more sequence;
• a serving stack (prefix caching, speculative decoding, continuous batching) built around a single cache.

With enough scale, the bidirectional encoder’s edge stopped paying for the extra complexity. In domains with a clear input–output asymmetry it never stopped paying.

Rules I use

1. Long input, short output, clear task: consider an encoder-decoder model. Speech and translation are the textbook cases.
2. Open-ended, multi-turn, tool-using work: decoder-only, for the serving stack alone.
3. Count cross-attention when comparing sizes. An encoder-decoder’s decoder is a third heavier per block than an encoder’s.

Sources: Vaswani et al., Attention Is All You Need (2017). Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5, 2020). Lewis et al., BART (2020). Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper, 2022). Parameter counts are computed from the T5-base config.