--- title: What the HANDBOOK.md Benchmark Says About Your CLAUDE.md description: The HANDBOOK.md benchmark tests AI agents on following long-form policy documents (20–124 pages) across 65 tasks in domains like finance, HR, and logistics.... canonical: https://daily.dev/posts/what-the-handbook-md-benchmark-says-about-your-claude-md-qs9d9ot4o twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: What the HANDBOOK.md Benchmark Says About Your CLAUDE.md | daily.dev og:description: The HANDBOOK.md benchmark tests AI agents on following long-form policy documents (20–124 pages) across 65 tasks in domains like finance, HR, and logistics.... og:url: https://daily.dev/posts/what-the-handbook-md-benchmark-says-about-your-claude-md-qs9d9ot4o og:image: https://api.daily.dev/og/posts/Qs9d9ot4o.png og:image:alt: What the HANDBOOK.md Benchmark Says About Your CLAUDE.md og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # What the HANDBOOK.md Benchmark Says About Your CLAUDE.md **[RUBYLAND](https://daily.dev/sources/rubyla)** · 10 min read · 1 upvotes · 0 comments ## Summary The HANDBOOK.md benchmark tests AI agents on following long-form policy documents (20–124 pages) across 65 tasks in domains like finance, HR, and logistics. The best model, Claude Fable 5, passes only 36.2% of tasks under strict grading; all other frontier models score below 25%. Four consistent failure patterns emerge: in-environment requests override standing policy, agents act against the result of a check they just performed, rule details decay over long sessions, and agents confidently report compliance even when citing the exact sections they violated. The core insight is that a CLAUDE.md or AGENTS.md file does not function as a persistent authority — it competes as one more retrieved source whose influence fades with session length and competing context. Practical takeaways include moving hard rules into hooks, linters, and tool-call guards rather than relying on prose files, treating code naming as part of keeping rules effective (stale names are authoritative-sounding competing context), and limiting LLM agent context to roughly 50% to reduce rule decay. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What percentage of tasks does the best model pass on the HANDBOOK.md benchmark? Claude Fable 5 achieves the highest score at 36.2% strict pass@1, which is 12.7 points ahead of the next best configuration. Every other frontier model, including Claude Opus 4.8 at maximum reasoning effort and GPT-5.5, scores between 21.5% and 21.9%, meaning even the top model still fails nearly two out of three tasks under strict grading. _Anyone tuning CLAUDE.md or AGENTS.md rules can track how agent instruction-following capability evolves via daily.dev._ ### Does increasing reasoning effort improve an AI agent's ability to follow long written policies? Not consistently. Raising reasoning effort improved Opus 4.8 by 3.0 points, Sonnet 4.6 by 2.7 points, and Fable 5 by 2.0 points, left GPT-5.5 unchanged at 21.5%, and actually hurt GLM 5.2 by 2.7 points. Extra deliberation appears to fix missed inferences but cannot fix a missed read of the source document. _Developers deciding how much reasoning effort to configure for agents can follow findings like this on daily.dev._ ### Why do AI agents fail to follow long instruction documents like company handbooks or CLAUDE.md files? Four recurring failure patterns emerge: a plausible in-environment request overrides the standing policy, the agent performs a required check and then acts against its own result, rule details get lost over long reasoning horizons, and the final report asserts compliance even while citing the exact sections it violated. The root cause is that the standing document behaves as just one more retrieved source whose influence decays with distance rather than as a persistent authority screening actions. _Teams hardening agent workflows beyond prose rules can keep up with findings like this on daily.dev._ ## Similar posts on daily.dev - [Writing a good CLAUDE.md](https://daily.dev/posts/writing-a-good-claude-md-xfiafgmqz) · Hacker News · 3 upvotes · 0 comments - [Audit your Agent files](https://daily.dev/posts/audit-your-agent-files-5pwynuy9w) · Addy Osmani · 9 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#claude](https://daily.dev/tags/claude), [#prompt-engineering](https://daily.dev/tags/prompt-engineering) [View this post on daily.dev](https://daily.dev/posts/what-the-handbook-md-benchmark-says-about-your-claude-md-qs9d9ot4o) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"What the HANDBOOK.md Benchmark Says About Your CLAUDE.md","url":"https://daily.dev/posts/what-the-handbook-md-benchmark-says-about-your-claude-md-qs9d9ot4o","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/what-the-handbook-md-benchmark-says-about-your-claude-md-qs9d9ot4o"},"datePublished":"2026-08-12T08:56:42.773Z","dateModified":"2026-09-14T06:41:29.127Z","description":"The HANDBOOK.md benchmark tests AI agents on following long-form policy documents (20–124 pages) across 65 tasks in domains like finance, HR, and logistics....","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/88095c008f5d58a597233fb5a37da978?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/88095c008f5d58a597233fb5a37da978?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"RUBYLAND","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"RUBYLAND","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/rubyla","url":"https://daily.dev/sources/rubyla"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/what-the-handbook-md-benchmark-says-about-your-claude-md-qs9d9ot4o","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,ai-agents,claude,prompt-engineering","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"RUBYLAND","item":"https://daily.dev/sources/rubyla"},{"@type":"ListItem","position":3,"name":"What the HANDBOOK.md Benchmark Says About Your CLAUDE.md"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/what-the-handbook-md-benchmark-says-about-your-claude-md-qs9d9ot4o#faq","mainEntity":[{"@type":"Question","name":"What percentage of tasks does the best model pass on the HANDBOOK.md benchmark?","acceptedAnswer":{"@type":"Answer","text":"Claude Fable 5 achieves the highest score at 36.2% strict pass@1, which is 12.7 points ahead of the next best configuration. Every other frontier model, including Claude Opus 4.8 at maximum reasoning effort and GPT-5.5, scores between 21.5% and 21.9%, meaning even the top model still fails nearly two out of three tasks under strict grading. Anyone tuning CLAUDE.md or AGENTS.md rules can track how agent instruction-following capability evolves via daily.dev."}},{"@type":"Question","name":"Does increasing reasoning effort improve an AI agent's ability to follow long written policies?","acceptedAnswer":{"@type":"Answer","text":"Not consistently. Raising reasoning effort improved Opus 4.8 by 3.0 points, Sonnet 4.6 by 2.7 points, and Fable 5 by 2.0 points, left GPT-5.5 unchanged at 21.5%, and actually hurt GLM 5.2 by 2.7 points. Extra deliberation appears to fix missed inferences but cannot fix a missed read of the source document. Developers deciding how much reasoning effort to configure for agents can follow findings like this on daily.dev."}},{"@type":"Question","name":"Why do AI agents fail to follow long instruction documents like company handbooks or CLAUDE.md files?","acceptedAnswer":{"@type":"Answer","text":"Four recurring failure patterns emerge: a plausible in-environment request overrides the standing policy, the agent performs a required check and then acts against its own result, rule details get lost over long reasoning horizons, and the final report asserts compliance even while citing the exact sections it violated. The root cause is that the standing document behaves as just one more retrieved source whose influence decays with distance rather than as a persistent authority screening actions. Teams hardening agent workflows beyond prose rules can keep up with findings like this on daily.dev."}}]} ```