amber-bench: benchmarking the harness, not the model
A model leaderboard can't tell you whether your agent loop is broken. I built a benchmark that scores the harness and the model as a pair. The first thing it found was bugs in my own harness.
Every agent result is a pair: a model and the harness driving it. The tool schemas, the system prompt, the loop control and the way tool output is parsed and fed back all change what the model does. Swap the harness and the same model scores differently. Leaderboards score the model alone, so they can’t tell you which half of the pair is failing.
amber is my C++17 agent runtime. amber-bench is the benchmark I built next to it, to measure the harness itself: every architecture change should show up as a number, not as a feeling after a good afternoon.
What it runs
There are 49 scenarios in 10 suites:
| Suite | Scenarios | Examples |
|---|---|---|
| Coding | 5 | FizzBuzz, sorting, ring buffer, LCS, graph BFS |
| Prompt adherence | 5 | envelope format, refusing a banned tool, verify before claiming |
| Refactoring | 4 | extract method, polymorphism over switch, adapter, strategy |
| Terminal work | 6 | count files, grep a value, compile and run C++, pipelines |
| Tool selection | 6 | search vs grep, read vs cat, write vs rewrite |
| Agent failures | 6 | redundant calls, recovering from a mistake, mid-stream dropout, empty replies |
| Loop-live | 6 | plan design, plan adherence, repeat discipline, termination |
| Review | 8 | architecture, data structures, patterns, code quality |
| Repo | 2 | repository navigation and search |
| Delegate | 1 | sub-agent delegation |
How it scores
Each scenario earns 0–100 from four sub-scores:
- Correctness: did it reach the right result? Checked against the expected answer and a hidden test.
- Efficiency: steps, wasted calls and redundant calls, compared with what the task needs.
- Robustness: retries, recoveries and hard stops.
- Adherence: prompt checks passed, minus any use of a forbidden tool.
Scenarios carry a difficulty weight from 1 to 5. The model score is the weighted aggregate, scaled to 1000.
The first thing it found was my bugs
I recorded harness baselines on the same model, Qwopus3.6-27B, as the engine changed:
| Baseline | Scenarios | Score | Pass |
|---|---|---|---|
| v2, first live calibration | 43 | 769.1 | 39/43 |
| v3, after fixes | 43 | 821.6 | 41/43 |
| v4, loop-live suite added | 49 | 788.2 | 42/49 |
The +52.5 points between v2 and v3 didn’t come from a better model or a better prompt. They came from three harness defects that the v2 run surfaced:
- Oracle: path normalisation in the oracle was wrong.
- JSON extraction: extraction failed when a model answered with bare JSON.
- SSE sink: the sink’s lifetime ended too early.
v4 adds six loop-discipline scenarios, so its score isn’t comparable with v3.
For scale: on the current 49-scenario set, the gap between the best and the worst of ten local models is 99.6 points. Harness bugs alone were worth about half of that.
Ten local models, one harness
Every model in my local swap registry runs the full 49 scenarios, serially, on one GPU slot:
| Model | Score | Pass | Loop-live | Wasted | Redundant |
|---|---|---|---|---|---|
| Qwen3.6-27B (dense) | 816.2 | 44/49 | 89.0 | 146 | 12 |
| Qwopus3.6-27B-v2-MTP | 808.0 | 42/49 | 76.2 | 142 | 12 |
| Gemma-4-31B | 806.2 | 38/49 | 75.9 | 108 | 8 |
| Gemma-4-12B Q4 | 780.0 | 37/49 | 75.8 | 120 | 17 |
| Ornith-1.0-35B | 770.0 | 40/49 | 75.6 | 192 | 43 |
| Qwen3.6-27B MTP | 768.8 | 41/49 | 77.1 | 166 | 17 |
| Gemma-4-12B Q8 | 756.8 | 37/49 | 75.8 | 135 | 28 |
| Qwen3.6-35B-A3B (MoE) | 742.2 | 36/49 | 65.9 | 183 | 36 |
| Qwen3.6-27B (pi-reasoning) | 723.3 | 36/49 | 63.2 | 197 | 36 |
| Ornith-1.0-9B | 716.6 | 27/49 | 80.0 | 271 | 122 |
The table above is a snapshot. The live leaderboard, updated with each run: trefon.com/benchmarks.
What I read from it:
- Dense 27B leads. The two dense Qwen3.6-27B variants are the strongest local agentic models I have, at 816 and 808.
- MoE and reasoning modes cost tool discipline. The 35B MoE (742) and the pi-reasoning variant (723) trail models smaller than them.
- Pass rate and tool economy are different axes. Gemma-4-31B wastes the fewest calls (108 wasted, 8 redundant) but passes only 38 of 49.
- Small models fall apart on loops. Ornith-1.0-9B makes 271 wasted and 122 redundant calls.
Running it without a model
There’s a hermetic mode with a fake LLM. It’s deterministic and needs no model and no GPU, so the harness half of the pair gets tested in CI on every change.
What I take from it
- Fix the harness before you pick the model. The cheapest 52 points I found were in my own code.
- Count waste, not just passes. Ornith-1.0-35B passes more scenarios than Gemma-4-31B (40 vs 38) with almost twice the wasted calls (192 vs 108). Wasted calls are what you pay for in tokens and latency.
- Measure loop discipline on its own. It fails differently from correctness, and it’s what turns a capable model into an agent that never stops.
The full tables, the scenario list and the method: jtrefon.github.io/amber/benchmarks. The code: github.com/jtrefon/amber.