The demo passes, production doesn’t.
Your test set is five prompts you wrote yourself. Real tickets are longer, messier and full of tool calls the demo never made.
L3 Harnesses / evals measured proof
Know whether the agent works before your customers do.
An agent that passes the demo and fails real tickets is not a model problem. It is a missing measurement problem. I build the eval suite from your real cases, fix what it exposes in the harness, and wire it into CI so the next model update cannot silently break it.
49
scenarios in 10 suites
+52.5
points from harness fixes
10
models through one harness
§01 the symptom n=3
How this usually shows up.
Every model update is a gamble.
You swap the model, spot-check a few answers and hope. Nothing tells you whether pass rate, tool discipline or latency moved.
The agent loops, and nobody can say why.
Repeated tool calls, dropped context, a retry that never terminates. Without traces and scenario-level scores, the fix is guesswork.
§02 what I change n=5
What a scored harness changes.
- 01
A scenario set from your cases
We turn real tickets, transcripts and failure reports into scored scenarios with a known correct outcome, including the ugly ones.
- 02
A score you can defend
Correctness, efficiency, robustness and prompt adherence, weighted by how much the case matters. One number per run, with the failures attached.
- 03
The harness fixes
Path handling, tool-call parsing, retries, context management, loop breakers. The benchmark usually finds defects in the harness before it finds them in the model.
- 04
A CI gate
Every prompt, tool or model change runs the suite. A regression fails the build instead of reaching a customer.
- 05
Traces you can read
Every scenario keeps its transcript and tool calls, so a failure can be diagnosed instead of argued about.
§03 measured proof
Numbers from my own runs.
- v2 769.1 · 39/43
- v3 821.6 · 41/43
49scenarios
across 10 suites, one GPU slot
+52.5points
from three harness defects, same model
769.1 → 821.6
harness v2 to v3, Qwopus3.6-27B
0GPU
needed for the hermetic CI mode
Source: amber-bench: 49 scenarios, one RTX 4090; the hermetic mode runs in CI with a fake model.
§04 deliverables n=4
What you get.
- An eval suite built from your real cases, versioned with your code
- A scorecard per run: pass rate, wasted calls, loops, latency
- Harness fixes, with the failing scenario as the proof
- A CI gate and a runbook for adding new cases
The math behind this.
- 2026-10-26 amber-bench: benchmarking the harness, not the model
- 2026-10-27 Agent eval statistics: is 816 vs 808 a real difference?
- 2026-10-27 Agent context management: budgets, truncation, compaction, and knowing when to stop
- 2026-10-27 MCP under the hood: JSON-RPC, transports, and capability negotiation
§05 answers n=5
Questions I get asked.
How many test cases do we need?
Enough to cover the jobs the agent actually does, not every sentence a user might type. I start with 30–50 scenarios across your main workflows and failure modes, then grow the set from production failures. amber-bench uses 49.
Can you evaluate an agent we bought?
Yes. If it has an API or a UI I can drive, it can be scored. The harness stays yours, so you can re-run it when the vendor ships a new version.
Do we need GPUs?
Not for the harness. For local models, one GPU is enough to run the suite serially. The hermetic mode runs with a fake model in CI, so the pipeline is tested without hardware.
Will this tell us which model to use?
It tells you which model passes your cases with acceptable tool discipline and latency. That is a better answer than a public leaderboard, which measures someone else’s tasks.
How long does it take?
A first scored suite usually takes two to three weeks. The CI gate and the handover add a week.
Shipping an agent you cannot measure?
Describe your problem30-minute technical call with the engineer who builds it. No sales team, no slides.