Better Models, Worse Tool Calling: Claude Opus 4.8 & Sonnet 5's Hidden Regression

Anthropic's two newest models — Opus 4.8 and Sonnet 5 — are breakthroughs in agentic capability. But there's a strange problem: they're worse at calling tools than their predecessors.

Armin Ronacher, creator of Flask and Jinja2, discovered this in his open-source project Pi. He spent two days debugging it, and his analysis is essential reading for anyone building AI agents.

What's Happening?

The story starts with issue #6278 on the Pi repository. When calling Pi's edit tool, Opus 4.8 frequently adds invented fields to the edits[] object. Petr Baudis, Pi's maintainer, measured the failure rate at roughly 20% of edit tool calls in some sessions.

Here's the twist: it's not Haiku or smaller models causing this. Opus 4.8 and Sonnet 5 — Anthropic's most capable, most expensive models — are the culprits. Older models like Opus 4.5 don't exhibit this behavior.

In his blog post, Ronacher admits: "I was pretty convinced that we're on a good path where the models are more likely to adapt to any sort of tool shape that comes around for as long as the instructions are good. Now I'm somewhat worried about the track we're on here."

What Keys Are Being Invented?

The catalog of invented keys is equal parts amusing and alarming:

• type, id, kind, unique — familiar generic field names
• requireUnique, matchCase, forceMatchCount — plausible but nonexistent
• in_file, children, notes, cost — completely random
• oldText2, newText2, oldText_2, newText_2 — as if the model "thinks" the array needs more entries
• event.0.additionalProperties — deeply nested, straight out of a JSON schema validator

The most frustrating part: the actual oldText and newText content in the failing calls is byte-correct. The model nails the hard part — composing the edit content — then appends garbage to the end of the JSON object, causing schema validation to reject it.

How LLM Tool Calling Actually Works

This is the most important section for developers building agents.

Tool calls aren't magic. The model receives a transcript, a system prompt, and a list of available tools. The server packs everything into a large prompt with special marker tokens. The model generates output, and a parser extracts the tool call. Two approaches ensure valid output:

Approach 1: Generate freely, validate after. The model writes JSON freely; the parser validates. Schema violation → error returned, model retries.

Approach 2: Constrained decoding. The sampler masks out tokens that would violate the grammar. If the schema only allows oldText and newText, the sampler will never emit in_file or type.

Each has trade-offs. Constrained decoding guarantees valid output but can degrade quality because the model is forced into a narrow token set. Free generation is more flexible but prone to schema violations.

The Core Hypothesis: Post-Training on Claude Code

Ronacher proposes a compelling theory: this isn't random deterioration — it's a post-training artifact.

Older Anthropic models were trained on a diverse set of tools with no single harness as the target. The newer models (Opus 4.8, Sonnet 5) were likely post-trained in the environment of Claude Code — a closed-source harness whose edit tool has a fundamentally different shape from Pi's.

The critical detail: the Claude Code client is extremely forgiving. Ronacher reverse-engineered the minified code and found:

• Retry paths for malformed tool calls
• Parameter aliases (old_str → old_string, path → file_path)
• Silent type coercions
• Unicode repair for broken strings
• Silent filtering — unknown keys are stripped without errors

Anthropic's harness never penalizes the model for adding extra keys. If reinforcement learning runs in that environment, there's no gradient against inventing aliases or stray fields. The task still succeeds; the reward still flows.

The result: models become increasingly specialized for one specific tool ecology, losing adaptability to unfamiliar schemas.

What This Means for Developers

If you're building AI agents with custom tools, three concrete implications:

1. Strict Mode Is Not Optional

Anthropic's API offers a strict mode for tool definitions that blocks unknown keys at the decoder level. The catch: many harnesses (including Claude Code) don't use it because strict mode imposes complexity limits on tool definitions, which can cause API requests to be rejected. But for simple custom tool schemas, enabling strict mode is the cheapest protection available.

2. Closed-Source Harnesses Are Hidden Liabilities

Claude Code is closed-source. You don't know how it parses tool calls, what it forgives, and what behaviors the model has been trained to exploit. If you build your own harness, the model may be "fighting" your schema because post-training taught it a different shape.

3. OpenAI Is Taking a Different Path

With the Harmony format, OpenAI embeds a <|constrain|>json marker directly in the transcript to signal "this segment is JSON, apply constrained sampling." They also support custom LARK grammars for tool definitions. This is an architectural solution to the exact problem Ronacher identified — and one that Anthropic hasn't publicly addressed.

Takeaways

• Opus 4.8 and Sonnet 5 produce schema-violating edit tool calls ~20% of the time in some sessions — older models don't
• Root cause: post-training in a "forgiving" harness that silently filters unknown keys, removing any penalty for schema violations
• Strict mode on the Anthropic API eliminates the issue entirely — but is limited by complexity caps on tool definitions
• OpenAI's Harmony format addresses this architecturally with <|constrain|>json markers and LARK grammar support
• If you build a custom harness: never assume the model will follow your schema. Always validate, or use constrained decoding

Armin Ronacher has surfaced a problem most developers haven't noticed yet. As models get smarter, they become more specialized for the environment they're trained in. This isn't a bug — it's a natural consequence of optimizing for a closed-source harness while the community builds hundreds of different open-source harnesses.

The open question: do we need an open tool schema standard that every model respects, instead of each vendor defining their own format?

Content assisted by AI (Amy 🌸). Reviewed by the author.