Encoder-only transformers: BERT, embeddings and rerankers

Encoder-only models don’t write anything, and they’re everywhere. Most of the embedding models behind RAG search and the rerankers that sort its results are encoders. This is part 2 of the architecture series, after decoder-only.

Bidirectional attention

An encoder block is the same as a decoder block, attention plus an MLP with residuals and norms, with one difference: there’s no causal mask. Every token attends to every other token, before and after it. The word “bank” in “the river bank flooded” sees “river” on the left and “flooded” on the right in the same layer.

That makes encoders very good at understanding a whole input at once, and unable to generate left to right. A token’s representation depends on tokens that wouldn’t exist yet during generation.

Training: masked language modelling

You can’t train a bidirectional model to predict the next token, because it can already see it. BERT (Devlin et al., 2019) instead hides some tokens and asks the model to recover them:

• 15% of positions are selected;
• of those, 80% are replaced with a [MASK] token, 10% with a random token, and 10% are left unchanged;
• the loss is computed only on the selected positions.

The 80/10/10 split matters. The model never sees [MASK] at inference, so it can’t learn that only masked positions need attention. Every token has to carry a useful representation. BERT also used next-sentence prediction; RoBERTa (Liu et al., 2019) showed it could be dropped and the model trained longer on more data instead.

The cost is efficiency: only 15% of tokens produce a training signal, where a decoder learns from every position.

Where the parameters live

The same arithmetic as decoders, 12·L·d² plus embeddings. BERT-base, with L = 12, d = 768 and a vocabulary of 30,522:

blocks           12 × 12 × 768²  =  84.9M
embeddings  (30,522 + 512 + 2) × 768  =  23.8M
total                              ≈ 108.8M    (published: 110M, with biases and norms)

A tenth of a small decoder model, which is part of the point. Encoders are cheap to run over millions of documents.

What encoders are for

• Embeddings. Pool the token representations, usually with a mean or a [CLS] token, into one vector per text. Train it so similar texts land close together. These vectors drive dense retrieval (the RAG note covers that side).
• Rerankers. A cross-encoder reads the query and a candidate passage together, in one bidirectional pass, and outputs a relevance score. Every query token can attend to every passage token, which is why it’s far more accurate than comparing two independent embeddings, and far more expensive.
• Classification and extraction. Intent detection, sentiment, named entities and PII detection are cheap, fast jobs where reading the whole input both ways is exactly what you want.

For those jobs, a 100–400M-parameter encoder is often more accurate per millisecond than prompting a 27B decoder.

Modern encoders

The original BERT stopped at 512 tokens with learned positions. ModernBERT (Warner et al., 2024) brought decoder-era improvements back: rotary positions, an 8,192-token context, alternating global and local attention for efficiency, and training on far more data including code. Encoders didn’t go away; they caught up quietly.

Rules I use

1. Use an encoder for scoring, matching and classification, and a decoder for generation.
2. Retrieve with a bi-encoder, rerank with a cross-encoder. Cheap recall, then expensive precision.
3. Fine-tune small encoders on your own labels. For narrow classification tasks, a few thousand examples on a 100M-class encoder often beat a large model prompted zero-shot.

Sources: Devlin et al., BERT (NAACL 2019). Liu et al., RoBERTa (2019). Reimers and Gurevych, Sentence-BERT (2019). Warner et al., ModernBERT (2024). Parameter counts are computed from the published BERT-base config.