Better Models: Worse Tools

Armin Ronacher investigates a regression in newer Claude models (Opus 4.8, Sonnet 5) where they emit extra, invented keys in tool call arguments that don't match the schema, causing Pi's edit tool to reject calls. The failure is context-dependent, appearing mainly in long agentic sessions. The hypothesis is that post-training on Claude Code's forgiving harness — which silently filters unknown keys and applies parameter aliases — has reduced the gradient against schema violations, making newer models worse at adhering to non-Claude-Code tool schemas. Enabling Anthropic's strict mode eliminates the issue, suggesting server-side grammar-constrained sampling is the fix. The broader concern is that as post-training becomes increasingly tied to one closed-source harness, alternative tool schemas may become implicitly off-distribution, forcing third-party harnesses to either mimic Claude Code's quirks or rely on strict mode.