HarnessTax: How Much Does the Harness Matter for Coding Agents?
Summary
Sky Lab's HarnessTax benchmark finds harness choice swings cost up to ~5× at similar success rates, across 21 model–harness pairs (seven models × Claude Code, Codex CLI, and Pi) on SWE-bench Lite and Terminal-Bench 2.0.
Harness tax is mostly cost.
Same model, similar success, up to ~5× token cost across harnesses; average success deltas stay within about ±2–5% on the two benches.
The same model can achieve similar success rates at up to 5x costs.
Pi's four tools reach the frontier.
Read, write, edit, and bash suffice for Pareto-competitive cost and success versus Claude Code and Codex CLI.
Pi reaches the Pareto frontier on both benchmarks by providing just four tools: read, write, edit, and bash
Own-harness is not best.
Across six Anthropic and OpenAI models on both benches, an alternative harness wins highest observed success in nine of twelve comparisons.
an alternative harness achieves the highest observed success rate in nine of twelve comparisons
First-call context seeds the tax.
Claude Code's mean initial context is over 10× Pi's via longer instructions and larger tool schemas, before later turns accumulate.
Claude Code’s mean initial context is over 10× Pi’s, with longer instructions and larger tool schemas.
Measure models across harnesses.
Default agent packaging hides cost; evaluations should hold the model fixed and report cost plus success across common harnesses.
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.