Agent eval statistics: is 816 vs 808 a real difference?

My own leaderboard puts Qwen3.6-27B at 816.2 and Qwopus3.6-27B-v2-MTP at 808.0. It’s tempting to call the first one better. Before anyone does, here’s the statistics. It applies to every agent benchmark, mine included.

A pass rate is an estimate

Passing 44 of 49 scenarios doesn’t mean the model has an 89.8% success rate. It means 89.8% is our best estimate from 49 draws. With so few scenarios, the uncertainty is wide. The Wilson score interval gives a 95% range for the true rate:

Model	Passed	Pass rate	95% interval
Qwen3.6-27B (dense)	44/49	0.898	0.782 – 0.956
Qwopus3.6-27B-v2-MTP	42/49	0.857	0.733 – 0.929
Gemma-4-31B	38/49	0.776	0.641 – 0.870
Ornith-1.0-9B	27/49	0.551	0.413 – 0.681

The top three intervals overlap heavily, so on pass counts alone the benchmark can’t order them. The bottom model is a different story: its upper bound, 0.681, sits below the best model’s lower bound, 0.782. That gap is real.

So the honest reading of my table is that the 27B dense models and Gemma-4-31B form a top group, and the small model is clearly worse. “816 beats 808” is not supported by this data.

How many scenarios would it take?

To detect a true difference between 85% and 90% with 80% power at the usual 5% significance level, treating the models as independent samples, you need about:

n ≈ (z_α/2 + z_β)² · (p₁(1−p₁) + p₂(1−p₂)) / (p₁ − p₂)²
  ≈ (1.96 + 0.84)² · (0.1275 + 0.09) / 0.05²   ≈ 683 scenarios per model

Six hundred scenarios to resolve five points. Most agent benchmarks have tens to a few hundred, which is why leaderboard order at the top is mostly noise.

Paired tests buy back power

There’s a better design. Every model in my benchmark runs the same scenarios, so the right question isn’t “whose pass rate is higher?” but “on the scenarios where they disagree, who wins?” Scenarios both pass, or both fail, carry no information about the difference.

McNemar’s test uses only the disagreements. Suppose two models disagree on 8 scenarios, and model A wins 6 of them:

exact two-sided p (6 vs 2) ≈ 0.29        not significant
exact two-sided p (10 vs 1) ≈ 0.012      significant

Pairing strips out the shared difficulty of each scenario, so it detects real differences with far fewer scenarios than comparing two pass rates. It requires per-scenario results, which is one more reason to keep raw run records, not just totals. My benchmark keeps them in bench/results/.

Run it more than once

Agents sample, so the same model on the same scenario can pass on one run and fail on the next. One run per scenario hides that variance completely. Run each scenario several times, at the temperature you’ll use in production, and report the spread. If two models’ run-to-run spread overlaps, the ranking isn’t settled.

pass@k vs pass^k

Code benchmarks often report pass@k: the chance that at least one of k attempts succeeds. That suits a human picking the best of k suggestions. It’s the wrong metric for an agent that runs unattended, where every run has to succeed. τ-bench (Yao et al., 2024) introduced pass^k: the chance that all k independent attempts succeed.

The gap between them is large:

Per-run success	pass@4	pass^4	pass^8
0.9	0.9999	0.656	0.430
0.8	0.9984	0.410	0.168

A model that succeeds 90% of the time looks perfect under pass@4. It still fails at least one of every four runs more than a third of the time. For production agents, pass^k is the number that predicts the support tickets.

Rules I use

1. Report intervals, not just ranks. A leaderboard without uncertainty invites over-reading, mine included.
2. Compare models paired, on the same scenarios, and keep per-scenario records so you can.
3. Repeat runs at the production temperature, and report the spread.
4. Use pass^k for agents. Reliability is the product; best-of-k is a demo metric.
5. Fix the harness before ranking models. On my benchmark, harness fixes moved scores by 52.5 points, more than the gaps between the top models. The amber-bench note has the details.

Sources: Wilson, Probable Inference, the Law of Succession, and Statistical Inference (1927). McNemar, Note on the sampling error of the difference between correlated proportions (1947). Chen et al., Evaluating Large Language Models Trained on Code (2021) for pass@k. Yao et al., τ-bench (2024) for pass^k. Intervals and p-values are computed from the counts shown; the model results are from my benchmark.