Attention Is All You Need: the paper, the idea, and what it changed
The 2017 Google paper that introduced the transformer: the problem it set out to solve, what it actually proposed (scaled dot-product attention, multi-head attention, sinusoidal positions, an encoder-decoder for translation), the numbers it reported, what later models changed, and how a translation paper became the basis of modern AI.
- deep dive
- transformer
- attention
- architecture
- history
In June 2017, eight researchers at Google posted a paper about machine translation with a title that riffs on a Beatles song. It reported a better BLEU score on two translation benchmarks at a fraction of the usual training cost. That paper, Attention Is All You Need by Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin, introduced the transformer. Almost every model this site is about descends from it.
The problem it set out to solve
In 2017, the best translation systems were recurrent sequence-to-sequence models. An encoder RNN read the source sentence token by token, and a decoder RNN wrote the translation token by token. Attention already existed: Bahdanau, Cho and Bengio (2014) had added a mechanism that let the decoder look back at all encoder states, instead of squeezing the sentence into one vector. But it was an add-on to recurrence.
Recurrence had two costs. Training couldn’t be parallelised across positions, because step t needed step t − 1. And information between distant positions had to travel through every step in between, so long-range dependencies were hard to learn (the RNN note covers why). Convolutional alternatives such as ByteNet and ConvS2S parallelised better, but still needed many layers to connect distant positions.
The paper’s bet is in its title: drop recurrence and convolution entirely, and build the whole model out of attention.
What it proposed
Scaled dot-product attention. Each position produces a query, a key and a value. A position’s output is a weighted sum of all values, weighted by how well its query matches each key:
Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · VThe √d_k scaling is a small, important detail. Without it, dot products grow with the dimension, push the softmax into regions with tiny gradients, and training stalls.
Multi-head attention. Instead of one attention with full-width vectors, run h attentions in parallel on lower-dimensional projections, then concatenate the results. Each head can attend to a different kind of relationship. The base model used 8 heads of 64 dimensions each.
Positional encodings. Attention is order-blind, so the paper added fixed sine and cosine waves of different frequencies to the input embeddings. That gave each position a unique signature and made relative offsets a linear function of the encodings. Today’s RoPE is a descendant of the same frequency idea (the RoPE note has the details).
The architecture. An encoder of 6 identical layers, each self-attention plus a feed-forward network, and a decoder of 6 layers with masked self-attention, cross-attention into the encoder, and a feed-forward network. Residual connections and layer normalisation wrap every sub-layer (the encoder-decoder note walks through this layout).
The training recipe. Adam with a warm-up schedule (the learning rate rises for 4,000 steps, then decays with the inverse square root of the step), dropout of 0.1, and label smoothing of 0.1. The recipe mattered as much as the architecture; transformers are notoriously sensitive to it.
The argument, in one table
The paper compares layer types on three properties for a sequence of length n with width d:
| Layer | Work per layer | Sequential steps | Longest path between two positions |
|---|---|---|---|
| Self-attention | O(n² · d) | O(1) | O(1) |
| Recurrent | O(n · d²) | O(n) | O(n) |
| Convolutional | O(k · n · d²) | O(1) | O(log_k n) |
Self-attention connects any two positions in one step and has no sequential dependency at all. Its cost is quadratic in sequence length, but for 2017’s sentence-length inputs, where n was smaller than d, it was actually cheaper per layer than recurrence. At n = d = 512 the two are equal; at n = 4096 attention does eight times the work. The quadratic term is the price that every long-context technique since has been trying to reduce.
What it reported
- English–German (WMT 2014): 28.4 BLEU for the big model, beating the previous best results, including ensembles, by more than 2 BLEU.
- English–French: 41.8 BLEU, a new single-model state of the art.
- Training cost: the base model trained in 12 hours on 8 NVIDIA P100 GPUs; the big model in 3.5 days. The paper points out that this was a small fraction of the training cost of the best competing models.
- Sizes: about 65M parameters for the base model and 213M for the big one.
It also showed the architecture transferring to English constituency parsing with little task-specific tuning, an early hint that this wasn’t just a translation model.
What later models changed
The skeleton survived; many details didn’t:
| 2017 paper | Typical today |
|---|---|
| Encoder-decoder | Decoder-only for general models (decoder-only note) |
| Sinusoidal absolute positions | Rotary position embeddings (RoPE) |
| Post-norm (normalise after the residual add) | Pre-norm, often RMSNorm, for stable deep training |
| ReLU feed-forward | Gated feed-forward (SwiGLU) |
| Full multi-head attention | Grouped-query or latent attention, to shrink the KV cache |
| Dense layers | Often mixture-of-experts (dense vs MoE note) |
| Sentence-length context | Hundreds of thousands of tokens |
The impact
Within about a year, the transformer had escaped translation:
- 2018: GPT (decoder-only) and BERT (encoder-only) showed that pre-training a transformer on raw text, then adapting it, beat task-specific models across the board.
- 2019–2020: GPT-2, T5 and GPT-3 established scaling: more parameters, more data, predictably better models. The Vision Transformer (2020) showed images could be treated as sequences of patches.
- 2021–2022: AlphaFold 2 used attention-based blocks to predict protein structures. Whisper did the same for speech. Codex turned the architecture on code.
- 2022 onward: ChatGPT, and the whole open-model ecosystem, from Llama to Qwen to DeepSeek.
The paper is now one of the most cited in computer science, with well over a hundred thousand citations. All eight authors eventually left Google, and several founded companies that are part of this story, among them Cohere, Character.AI, Essential AI, Inceptive, Sakana AI and NEAR.
Its deepest legacy isn’t any one component. It’s that a single, simple, parallelisable block, repeated and scaled, turned out to be general enough to learn language, code, images, audio and proteins. That made AI progress mostly a matter of data, compute and engineering, which is the kind of problem engineers can work on.
Reading it today
The paper is short and still worth reading. Three things to notice:
- How much is training recipe: warm-up, label smoothing and dropout placement are given as much care as the architecture.
- The complexity argument: it’s the clearest statement of why attention beat recurrence, and of the quadratic cost that long-context research still fights.
- The attention visualisations at the end: heads that track syntax and resolve pronouns, the first glimpse of the interpretability questions people still study.
Sources: Vaswani et al., Attention Is All You Need (NeurIPS 2017; arXiv 1706.03762). Bahdanau, Cho and Bengio, Neural Machine Translation by Jointly Learning to Align and Translate (2014). Radford et al., GPT (2018). Devlin et al., BERT (2019). Dosovitskiy et al., ViT (2020). Jumper et al., AlphaFold 2 (2021). Complexities and results are as reported in the paper.