Real-SWE: The Benchmark That Exposes AI Coding Agents

Specific Labs just released Real-SWE, a benchmark that tests AI coding agents against actual production codebases instead of synthetic tasks or public repos a model may have already memorized. The result: the best-performing setup solves 38.8% of tasks. The other seven fall below 34%.

Real tasks, not textbook problems

What sets Real-SWE apart is where the tasks come from. Instead of writing simulated problems, Specific Labs licensed codebases directly from real companies — including a Luma/Partiful competitor with over 200,000 users and a top-100 App Store ranking, a consumer fintech platform processing more than 100,000 bank statements, and several enterprise AI sales platforms. Every task is something an actual engineer had to do, with all the context and mess that comes with it: company-specific conventions, tangled business logic, systems that span multiple services.

One sample task: fix invoice tax logic in a NestJS/TypeScript service, where each business on the platform settles tax a different way — some maintain their own fixed rate, some call the TaxJar API based on the buyer's address, some are fully exempt. The agent has to read each business's configuration correctly, hit the right environment (sandbox or production depending on the account), handle addresses TaxJar rejects without blocking the invoice, then log the transaction back to InfluxDB under the right invoice number so accounting reconciles. None of this ever existed on the public internet.

That's exactly why the difficulty is a step up from older benchmarks. Real-SWE's median instruction length is 1,742 characters — shorter than FrontierCode (2,056) and DeepSWE (1,975) — but the reference solution touches a median of 11 files, nearly double the 6 files in those two benchmarks. Short instructions force the agent to infer most of the implementation detail from the codebase itself, the same way a new hire has to read code to understand a system before touching it.

The leaderboard: the gap is smaller than expected

Per the benchmark's self-reported results (no independent third-party audit yet), resolution rate — pass@1 averaged across 8 independent runs per task — breaks down like this:

Rank	Model	Harness	Resolution rate
1	Fable 5.1	Claude Code	38.8%
2	GPT-6 Astra	Codex CLI	33.8%
3	Gemini 3.8 Flash	Gemini CLI	31.2%
4	GLM 5.3	Claude Code	28.8%
=5	Grok 4.6	Grok Build	23.8%
=5	Muse Spark 1.3	Muse Code	23.8%
7	Kimi K3	Kimi Code	18.8%
8	GPT-5.6 Sol	Codex CLI	16.2%

The gap between first and last place is 22.6 percentage points — nowhere near as wide as the flashy numbers on showcase benchmarks suggest. More telling: Real-SWE evaluates model-and-harness combinations, not models in isolation. GLM 5.3 and Fable 5.1 both run on Claude Code, yet land 10 points apart — the model still drives most of the outcome, the harness doesn't level the field.

Every model fails a different way

Error analysis shows the models don't just differ in score — they differ in how they break. Grok 4.6 and Kimi K3 lead the "missed requirement" category (67.2% and 53.8% of each model's failed runs) — the agent writes partially correct code but leaves out behavior the task actually required. Gemini 3.8 Flash and Muse Spark 1.3 lead "integration error" instead (49.1% and 41.0%) — the right idea, wired into the system incorrectly. GPT-5.6 Sol tops "unverified assumption" (43.3%) — guessing about the system instead of checking the workspace.

One more number worth noting: rollouts under 10 minutes fail 71.4% of the time (70/98); longer rollouts fail 73.4% (398/542). Letting an agent run longer barely moves the needle — a sign the bottleneck isn't thinking time, it's missing context about a specific company's conventions and constraints.

What this means for developers

• Don't read SWE-bench scores as a proxy for production performance. The 70-80% resolution rates on familiar public benchmarks reflect tasks from popular open-source repos, likely present in training data. On private codebases with their own conventions, performance drops sharply.
• Cost doesn't scale with quality. The top model (Fable 5.1) costs an estimated $6.96 per rollout — nearly three times Gemini 3.8 Flash's $2.50 — for only 7.6 extra points of resolution rate. At scale, the cost-versus-accuracy trade-off matters more than picking the "best" model on a leaderboard.
• "Missed requirement" is the failure hardest to catch in a quick review. Code that runs, tests that pass, but a required branch of behavior is just absent — that slips past a fast review far more easily than a syntax error or crash.
• Reviewing agent-generated PRs needs a requirements checklist, not just a diff read. Since the most common failure is missing behavior rather than broken syntax, comparing the diff against the original ask matters more than judging how "clean" the code looks.

Real-SWE doesn't argue AI coding agents are useless — it draws a clear line: today's agents are good at tasks with context already in front of them, but still far from reliable enough to hand off enterprise work without close supervision.

---

Content assisted by AI (Amy 🌸). Reviewed by the author.