measured, both arms, every run
Benchmarks.
Same model, same tasks, bare repo vs harness. 3 reps per cell, pre-registered graders. What follows is what came out, including the cells that flatter no one.
All runs below: Apple M4 Mac mini with 16 GB memory (not overclocked) · driver opencode run · harness v0.1.2 (ec6dba3)
kimi-k3 via Nvidia ($0, free tier)
18 runs, 18/18 pass. Wall about tied, harness spent more tokens (its context costs ~5-10k input tokens a run).
nemotron-3-super-120b via Nvidia ($0.43 total)
18 runs, 14/18 pass. T1 answers came back terse and tripped the keyword grader (same grader both arms). T3 is the standout harness cell.
Run 2: slim init payload (same bed, both models)
After run 1, init ships a slim AGENTS.md (724B vs 1948B) and pip admits its limits. T2/T3 on kimi collapsed to noise; nemotron now wins or ties every cell on tokens.
Run 3: nemotron, semantic grader (15/18)
T1's keyword grader failed terse-but-correct answers, so v2 accepts equivalent phrasings (pre-registered, fixture-tested). Full fresh rerun. The three remaining T1 misses are genuinely thin answers, both arms.
Run 4: nemotron, v3 tasks (16/18)
T1 asks for files by name and quoted output. Token deltas all inside noise. T1 still misses once per arm: the model answers thinly ~1/3 of the time regardless of setup.
muse-glimmer-30b (17/18, new driver)
Found in a 30-model probe sweep (6 live). Verbose agent, twice the tool calls. One 6s no-tool flake is the only miss.
Megabox: 200-file bed, kimi-k3
Generated monorepo (seeded, generator committed). Harness wins tokens on build/fix tasks; orientation Q&A sits below this driver's threshold (0/6, one diagnostic near-perfect answer proves the task fair).
Megabox + map, nemotron (12/18)
Treatment adds a deterministic repo index (206 files to 229 lines). It did not move the needle: harness costs more tokens in every measurable cell. Extra context plus extra exploration outweighs the orientation saved, here.
Megabox + enriched map, nemotron (11/18)
Second run: enriched memory, narrowed skill triggers. Same verdict, sharper numbers.
M6 local models: qwen2.5-coder ladder ($0, Ollama)
Apple M6 mini, 12c, 16 GB · bare read-everything (4 inferences) vs harness MAP+memory (2 inferences) · 48 rows · 16/16 fixes both arms. Per-token speed is identical; the harness halves session wall by halving inference count. 14b peaks 9.0 GB resident with ~3 GB swapped: the practical 16 GB ceiling.
Reading
On a small repo the harness context costs more tokens than orientation saves; on a 200-file bed it wins tokens on build/fix tasks (-13/-14%). Wall time lands within variance in most cells. n=3 throughout, free-tier queueing noise visible. Full method, probe log (30 models tried, 6 live), and all raw rows: docs/benchmarks.md.
Run it on your stack
Different model, provider, or hardware? Run both arms on the same tasks, 3 reps each (method in research/plans/ab-perf.md), and submit. Accepted submissions land in this table with your hardware label.