An Empirical Study of Harness Design for Coding Agents
Summary
Across 176 matched settings on SWE-Bench Verified and Terminal-Bench 2.1, this paper isolates planning, action space, and context management inside one fixed coding-agent loop.
Component ablations beat harness bake-offs.
Holding the execution loop fixed reveals when planning, tools, and context policy actually move accuracy versus cost.
Context management is a budget tool.
Its value rises as the window shrinks, mostly by preventing overflow rather than rewriting agent behavior.
Context management becomes increasingly valuable as the context-window budget tightens
Elide, then summarize.
Rule-based elision before selective LLM summarization beats recoverable-elision machinery that models barely use.
Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency
Planning changes role with model strength.
Weaker models need it as an accuracy scaffold; stronger models mainly spend less with little accuracy change.
Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models
Action space should match bash skill.
Predefined tools help weak bash users; bash-fluent models can run bash-only interfaces more cheaply on command-line tasks.
bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.