← @bgSponge

Real-SWE Benchmark — Specific Labs

Summary

Specific Labs releases Real-SWE, a coding-agent benchmark on private production codebases licensed from real companies rather than public or synthetic tasks.

  1. Private production code is the test bed.

    Tasks come from licensed company codebases, so agents cannot lean on public GitHub solutions or internet-leaked answers.

    Can a coding agent actually do the work of a software engineer in the real world?

  2. Score the model-plus-harness people actually use.

    Native harnesses (Claude Code, Codex CLI, Gemini CLI, and peers) are evaluated together instead of models in isolation.

    We use native harnesses to reflect how enterprise engineers work in practice

  3. Business consequences beat toy edits.

    Billing tax rules, migrations, and multi-service changes force agents to respect existing products and operational constraints.

  4. Short prompts still demand wide diffs.

    Median instructions are about 1,742 characters, yet reference solutions edit a median of eleven files—more than FrontierCode or DeepSWE.

  5. Frontier resolution stays low.

    Even the leader clears under two-fifths of tasks at pass@1 averaged over eight runs, showing real enterprise SWE remains hard.

    Resolution rate is equivalent to pass@1, averaged over eight independent runs per task.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →