RoPE from first principles, and how YaRN stretches a model's context

Attention on its own has no idea where a token is: permute the inputs and you permute the outputs. Position has to be added. Almost every current open model does it with rotary position embeddings (RoPE). How RoPE works decides how far a model can be pushed past the context it was trained on.

Position as rotation

RoPE (Su et al., 2021) splits each query and key vector into pairs of dimensions and rotates pair i by an angle proportional to the token’s position m:

θ_i = base^(−2i/d)          i = 0 … d/2 − 1,   d = head dimension
[x_2i, x_2i+1] → rotate by m · θ_i

The useful property: rotating q by mθ and k by nθ makes their dot product depend only on m − n. Attention sees relative position for free, through the angle between two vectors. No position vector is added, and nothing has to be learned per position.

Every dimension has a wavelength

Pair i completes a full turn every λ_i = 2π / θ_i = 2π · base^(2i/d) tokens. With a head dimension of 128:

Base	Fastest pair	Slowest pair	Pairs slower than 32K tokens
10,000	6.3 tokens	54,410 tokens	4 of 64
1,000,000	6.3 tokens	5,063,256 tokens	24 of 64

The fast pairs encode fine local order: is this token 3 or 4 positions back? The slow pairs encode coarse, long-range position.

The extrapolation problem lives in the slow pairs. If a pair’s wavelength is longer than the training context, the model has only ever seen a fraction of its rotation. Past the training length, that pair produces angles the model has never seen, and attention degrades. That’s why newer models raise the base: Llama 3 uses 500,000 and Qwen2.5 uses 1,000,000. It spreads the slow dimensions out, at the cost of more of them never completing a turn during training.

Three ways to stretch it

Position interpolation (Chen et al., 2023). Divide every position by the scale factor s = L_new / L_train, so a 262K position looks like a 32K one. It’s simple, but it also compresses the fast pairs, which costs local resolution: tokens 3 and 4 positions back now look closer together than the model learned. It needs fine-tuning to recover.

NTK-aware scaling. Change the base instead of the positions: base' = base · s^(d / (d − 2)). The fast pairs barely change and the slow pairs stretch the most. It works noticeably better than plain interpolation without fine-tuning, but it treats every dimension with one formula.

YaRN (Peng et al., 2023). Treat each pair according to its wavelength relative to the training length:

• pairs that complete many turns within the training context are left alone, because they already generalise;
• pairs that complete less than about one turn are interpolated by the full factor s;
• pairs in between get a linear ramp.

YaRN also adds an attention temperature. Queries and keys are scaled by √(1/t) = 0.1 · ln(s) + 1, because a longer context spreads attention over more tokens and flattens the softmax. Going from 32K to 262K gives s = 8, a factor of 1.208, so attention logits are multiplied by about 1.46.

How models actually get to 262K

The usual recipe is to pre-train at a shorter length, extend with YaRN-style scaling, and then train further on long documents so the model learns to use the extra range. Published configs show the scaling explicitly: several Qwen releases ship a native 32K window with a YaRN factor of 4 to reach 131K. Newer ones advertise 262,144 tokens natively.

In llama.cpp the scaling usually comes from the GGUF metadata. You can override it with --rope-scaling yarn, --rope-scale and --yarn-orig-ctx.

Two things RoPE scaling does not change:

• KV cache cost. The cache grows linearly with context no matter how positions are encoded. The KV cache note has the per-token numbers.
• Whether the model can reason over all of it. Finding a needle at 200K is not the same as using information from 200K tokens back. Evaluate long context on your own tasks.

Rules I use

1. Don’t push past the advertised context without a scaling method the model was trained or tested with. “It loads” isn’t “it works”.
2. Check the GGUF metadata for the RoPE base and scaling before blaming the model for long-context failures.
3. Measure at the length you’ll actually run. Quality, speed and cache size all change with context.
4. Prefer models trained long over models stretched long, when the task needs it.

Sources: Su et al., RoFormer (2021). Chen et al., Extending Context Window of Large Language Models via Positional Interpolation (2023). Peng et al., YaRN (2023). NTK-aware scaling was first described by the open-source community (bloc97, 2023). Wavelength numbers are computed from the formula above.