Trefon.
amber-bench run

Local models for agentic coding, measured.

Most “best local model” lists repeat the numbers on model cards. These are runs: ten local models through the same agent harness, 49 scenarios, one RTX 4090, serial, one GPU slot. Same harness, same tasks, so the only thing that changes is the model.

local models
10
local models
scenarios in 10 suites
49
scenarios in 10 suites
RTX 4090, 24 GB
1×
RTX 4090, 24 GB
difficulty-weighted score
/1000
difficulty-weighted score
§01 leaderboard

Ten models, one harness.

amber-bench model population. Higher score and pass count are better; fewer wasted and redundant calls are better.
# Model Score Pass Loop-live Wasted Redundant
01 Qwen3.6-27B (dense) 816.2 44/49 89.0 146 12
02 Qwopus3.6-27B-v2-MTP 808.0 42/49 76.2 142 12
03 Gemma-4-31B 806.2 38/49 75.9 108 8
04 Gemma-4-12B Q4 780.0 37/49 75.8 120 17
05 Ornith-1.0-35B 770.0 40/49 75.6 192 43
06 Qwen3.6-27B MTP 768.8 41/49 77.1 166 17
07 Gemma-4-12B Q8 756.8 37/49 75.8 135 28
08 Qwen3.6-35B-A3B (MoE) 742.2 36/49 65.9 183 36
09 Qwen3.6-27B (pi-reasoning) 723.3 36/49 63.2 197 36
10 Ornith-1.0-9B 716.6 27/49 80.0 271 122

amber-bench model population. Higher score and pass count are better; fewer wasted and redundant calls are better. Score bars: 700–850.

Raw tables and method ↗ → The write-up github.com/jtrefon/amber ↗
§02 answers

What the numbers say.

What’s the best local model for agentic coding on a 24 GB GPU?

In this run, Qwen3.6-27B dense: 816.2 and 44 of 49 scenarios passed, with the best loop discipline (89.0). Qwopus3.6-27B-v2-MTP (808.0) and Gemma-4-31B (806.2) are close behind. Qwen3.8-27B wasn’t part of this run.

Are MoE models worse for agents?

Here, yes. Qwen3.6-35B-A3B scored 742.2 with a loop-live score of 65.9, against 89.0 for the dense 27B. It also made 183 wasted calls. MoE routing and reasoning modes cost tool discipline in this harness.

Which model wastes the fewest tool calls?

Gemma-4-31B: 108 wasted and 8 redundant calls, the lowest of the ten. It passes only 38 of 49, though, so tool economy and pass rate are separate axes.

Does the harness matter as much as the model?

On the same model, fixing three harness bugs moved the score from 769.1 to 821.6, +52.5 points. The gap between the best and the worst of these ten models is 99.6 points.

§03 method

How it’s scored.

  1. 01 Each scenario earns 0–100 from four sub-scores: correctness (expected answer plus a hidden test), efficiency (steps, wasted and redundant calls), robustness (retries, recoveries, hard stops) and prompt adherence.
  2. 02 Scenarios carry a difficulty weight from 1 to 5; the model score is the weighted aggregate, scaled to 1000.
  3. 03 Suites: coding, prompt adherence, refactoring, terminal work, tool selection, agent failures, loop-live, review, repo and delegation.
  4. 04 Every model runs the full set serially on one GPU slot, through amber, my C++ agent harness.
fig. 01 amber-bench · score /1000
  • Qwen3.6-27B dense 816.2
  • Qwopus3.6-27B-v2-MTP 808.0
  • Gemma-4-31B 806.2
  • Gemma-4-12B Q4 780.0
  • Ornith-1.0-35B 770.0
  • Qwen3.6-27B MTP 768.8
  • Gemma-4-12B Q8 756.8
  • Qwen3.6-35B-A3B MoE 742.2
  • Qwen3.6-27B pi-reason. 723.3
  • Ornith-1.0-9B 716.6
2026-08-13 · RTX 4090 · 24 GB · axis 700–1000

Want this run on your models and your tasks?

Describe your problem

30-minute technical call with the engineer who builds it. No sales team, no slides.