← @bgSponge

An Empirical Study of Harness Design for Coding Agents

Summary

Across 176 matched settings on SWE-Bench Verified and Terminal-Bench 2.1, this paper isolates planning, action space, and context management inside one fixed coding-agent loop.

  1. Component ablations beat harness bake-offs.

    Holding the execution loop fixed reveals when planning, tools, and context policy actually move accuracy versus cost.

  2. Context management is a budget tool.

    Its value rises as the window shrinks, mostly by preventing overflow rather than rewriting agent behavior.

    Context management becomes increasingly valuable as the context-window budget tightens

  3. Elide, then summarize.

    Rule-based elision before selective LLM summarization beats recoverable-elision machinery that models barely use.

    Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency

  4. Planning changes role with model strength.

    Weaker models need it as an accuracy scaffold; stronger models mainly spend less with little accuracy change.

    Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models

  5. Action space should match bash skill.

    Predefined tools help weak bash users; bash-fluent models can run bash-only interfaces more cheaply on command-line tasks.

    bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →