Tokenizers: BPE, byte-level models, and what a token costs

Everything about an LLM is counted in tokens: context length, KV cache size, price, speed. But “token” isn’t a fixed unit. It’s whatever the tokenizer decided, and that decision has costs everywhere downstream.

BPE: a vocabulary built from merges

Byte-pair encoding, as adapted for neural translation by Sennrich et al. (2016), starts from single symbols and repeatedly merges the most frequent adjacent pair into a new symbol, until the vocabulary reaches its target size. Common words become single tokens. Rare words are spelled out from smaller pieces. Encoding new text replays the learned merges in order.

Byte-level BPE (GPT-2, Radford et al., 2019) starts the merges from the 256 possible bytes instead of characters. Any string, in any language, emoji or binary garbage, can be encoded, because in the worst case it falls back to raw bytes. There’s no “unknown token”. Most current open models use a byte-level BPE variant. SentencePiece, with BPE or Unigram models (Kudo, 2018), is the other common family.

Vocabulary size is a trade

A bigger vocabulary means:

• Fewer tokens per text. Each token covers more characters, so the same document is a shorter sequence: less KV cache, less attention work and more text per context window.
• More parameters at both ends. The embedding matrix and the output projection are each V × d. With a vocabulary of 152,064 and a hidden size of 4096, that’s 623M parameters per matrix; at a hidden size of 5120, 779M. Unless the two are tied, a model pays that twice. For a small model, that’s a big share of the total budget.
• Thinner training per token. Rare tokens are seen less often, so their embeddings are learned from less data.

That’s why vocabularies grew from about 32K tokens in early Llama models to 128K–152K now: longer contexts and multilingual text made sequence length the more expensive side.

Not all text costs the same

A tokenizer trained mostly on English packs English efficiently and everything else less so. Polish is a good example. Diacritics are two bytes each in UTF-8: “Zażółć gęślą jaźń” is 17 characters but 26 bytes, where “The quick brown fox” is 19 characters and 19 bytes. Polish inflection also creates many surface forms of each word, so fewer of them become single tokens. The same meaning costs more tokens, which means more money, more KV cache and less effective context than the same text in English.

Code has its own quirks. Indentation, long identifiers and symbol-heavy syntax tokenise unevenly, and how a tokenizer handles runs of spaces changes code token counts a lot. When you budget context for an agent working on code, measure tokens on your actual files, not on prose.

Models without a tokenizer

Byte-level models drop the learned vocabulary and work on raw bytes:

• ByT5 (Xue et al., 2022): bytes all the way, which makes sequences much longer.
• MegaByte (Yu et al., 2023): groups bytes into fixed patches.
• Byte Latent Transformer (Pagnoni et al., 2024): sizes patches dynamically by how predictable the next byte is. Hard regions get more compute, easy ones less.

The appeal is robustness: no tokenisation artefacts, uniform handling of every language, and no vocabulary to bolt on. The cost is sequence length and the engineering to make patching efficient. I wrote blt, a high-performance byte-level tokenizer in Rust. This layer is where a lot of efficiency is won or lost before the model ever sees the input.

Rules I use

1. Measure tokens on your own data before sizing context or estimating cost. Ratios measured on English prose don’t transfer to Polish or to code.
2. Never mix tokenizers between training and serving. A model is only defined together with its vocabulary.
3. Count the embeddings when comparing small models. A big vocabulary can be a third of a small model’s parameters.
4. Watch the edges in tool calls. Whitespace and special tokens around JSON are where tokenisation bugs show up first.

Sources: Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units (2016). Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2, 2019). Kudo, Subword Regularization (2018). Xue et al., ByT5 (2022). Yu et al., MegaByte (2023). Pagnoni et al., Byte Latent Transformer (2024). Byte counts are computed for the strings shown.