Sampling math: temperature, top-p, min-p, and why agents loop

The model doesn’t output text. It outputs a score for every token in the vocabulary, the logits, and a chain of samplers turns those into one choice. Most “the model is dumb” bug reports I see are really sampler bugs. Here’s what each step does, on real numbers.

From logits to probabilities

Softmax turns logits z into probabilities, and temperature T divides the logits first:

p_i = exp(z_i / T) / Σ_j exp(z_j / T)

Take five candidate tokens with logits [5.0, 4.2, 3.9, 2.0, 0.5]:

Token	T = 1.3	T = 1.0	T = 0.6
A	0.476	0.543	0.699
B	0.257	0.244	0.184
C	0.204	0.181	0.112
D	0.047	0.027	0.005
E	0.015	0.006	0.0004

Lower temperature sharpens the distribution toward the top token; higher temperature flattens it. At T → 0 you get greedy decoding: always the top token, fully deterministic.

Truncation: top-k, top-p, min-p

Temperature reshapes the distribution. Truncation samplers cut off its tail, so that a 1-in-200 token can never be picked:

• Top-k keeps the k most likely tokens. It’s blunt: k = 40 is too many when the model is sure and too few when it isn’t.
• Top-p, or nucleus sampling (Holtzman et al., 2020), keeps the smallest set of tokens whose probabilities sum to at least p. At T = 1 and p = 0.9, the example keeps A, B and C, which cover 0.967.
• Min-p (Nguyen et al., 2024) keeps every token whose probability is at least min_p × p_max. With min_p = 0.05 and p_max = 0.543, the threshold is 0.027, so again A, B and C survive.

Min-p’s advantage is that the threshold scales with the model’s confidence. When the top token is at 0.9, anything below 0.045 is cut. When the model is genuinely unsure and the top token is at 0.2, the bar drops to 0.01 and more alternatives survive. Top-p moves the cut by mass instead, which behaves worse on very flat or very peaked distributions.

Order matters

Samplers run as a chain, and llama.cpp lets you set the order with --samplers. Order changes the result:

• Truncate, then apply temperature: the candidate set is decided by the model’s raw confidence, and temperature only reshapes what survives.
• Apply temperature, then truncate: a high temperature flattens the distribution first and lets more of the tail through the cut.

Neither order is wrong, but the same settings mean different things in different engines. When you move a configuration between engines, check the chain, not just the numbers.

Repetition penalties, and why they break code

The CTRL-style repetition penalty (Keskar et al., 2019) scales down the logits of tokens that already appear in a recent window: positive logits are divided by θ, negative ones multiplied. Frequency and presence penalties instead subtract α × count + β × [count > 0].

That’s fine for prose. It’s poison for code and JSON, where repetition is the syntax: braces, quotes, commas, indentation, the same key names on every line. Penalise those and you get malformed tool calls and broken indentation. For agents and code, I keep repetition penalties off and solve repetition elsewhere.

Why agents loop

An agent loop is a feedback system. The model sees a tool result, chooses an action, and sees a result again. If an action produced the same result as last time, the context is now nearly identical to the previous step. A deterministic or near-deterministic sampler then picks the same action again. The repetition is self-reinforcing: every repeat makes the next repeat more likely.

Two fixes, at different layers:

1. Sampling. A bit of entropy breaks the symmetry. On my inference box, moving to --temp 0.6 --top-p 0.95 --min-p 0.05 stopped the tool-call loops. It also took decode from 62 to 82 tok/s, because sampling settings change how many speculative draft tokens are accepted (the decode note has that math).
2. The harness. Detecting repeated calls, refusing to run the identical call twice and forcing a different plan is more reliable than tuning temperature. amber-bench has a whole loop-discipline suite for this, and the benchmark note shows how widely models differ on it.

Rules I use

1. Agents and code: temperature around 0.6, with top-p and min-p as guard rails, and no repetition penalty. Loops are handled in the harness.
2. Evals: fixed seed and low temperature when comparing models, then repeat runs to see the spread. A single sampled run isn’t a measurement.
3. Check the chain order when moving settings between engines.
4. Constrain the format, not the content. If you need valid JSON, use constrained decoding instead of lowering the temperature until the model stops being creative.

Sources: Holtzman et al., The Curious Case of Neural Text Degeneration (ICLR 2020). Nguyen et al., Turning Up the Heat: Min-p Sampling (2024). Keskar et al., CTRL (2019). The 62 → 82 tok/s measurement is from my fork’s production configuration.