--- title: Dayjob: Tax – evaluating AI agents on 50 professional... description: Surge AI's DAYJOB: Tax benchmark tests AI agents on 50 realistic professional tax assignments across eight practice areas. Claude Sonnet 5.5 leads the... canonical: https://daily.dev/posts/dayjob-tax-evaluating-ai-agents-on-50-professional-tax-assignments-tknynvr2r twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Dayjob: Tax – evaluating AI agents on 50 professional tax assignments | daily.dev og:description: Surge AI's DAYJOB: Tax benchmark tests AI agents on 50 realistic professional tax assignments across eight practice areas. Claude Sonnet 5.5 leads the... og:url: https://daily.dev/posts/dayjob-tax-evaluating-ai-agents-on-50-professional-tax-assignments-tknynvr2r og:image: https://api.daily.dev/og/posts/TKNyNvr2R.png og:image:alt: Dayjob: Tax – evaluating AI agents on 50 professional tax assignments og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Dayjob: Tax – evaluating AI agents on 50 professional tax assignments **[Hacker News](https://daily.dev/sources/hn)** · 8 min read · 3 upvotes · 1 comments ## Summary Surge AI's DAYJOB: Tax benchmark tests AI agents on 50 realistic professional tax assignments across eight practice areas. Claude Sonnet 5.5 leads the leaderboard with a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%, using pass@1 at a 95% rubric threshold. The benchmark includes 1,789 workspace files such as PDFs, Excel workbooks, and handwritten receipts. In one case, Astra never identified a deadline that could cost a home seller their exclusion eligibility across five attempts, while Opus identified it every time. In another, Opus assumed an unmarried couple across all five attempts, missing marriage evidence in their own records, while Astra recognized the need to verify marital status in four of five attempts. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Which AI model performs best on the DAYJOB Tax benchmark for professional tax work? Claude Sonnet 5.5 leads the DAYJOB: Tax leaderboard with a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. The benchmark measures pass@1 at a 95% rubric threshold across 50 realistic tax environments covering calculations, supporting evidence, consequential issues, and completed deliverables. _Teams choosing an agent for tax or compliance work can track benchmark results like these on daily.dev._ ### What is the DAYJOB Tax benchmark and what does it test in AI agents? DAYJOB: Tax is a benchmark from Surge AI with 50 environments across eight areas of tax practice, including individual, partnership, estate, and international tax. It tests whether AI agents can navigate messy records, exercise professional judgment, identify issues clients never raised, and produce complete tax returns, workpapers, or advice, not just answer predefined questions. _Developers evaluating agentic AI for professional domains can follow benchmarks like this on daily.dev._ ### Why do AI agents fail at real-world tax advisory tasks even when they know the tax rules? Agents often fail because they don't investigate unverified assumptions or flag deadlines the client never asked about, even when they know relevant rules. In one test, Opus 5.5 assumed a separating couple was unmarried across all five attempts despite records suggesting a valid informal marriage, while Astra correctly flagged the need to verify marital status in four of five attempts. _Anyone weighing AI agent reliability for high-stakes work can follow real benchmark failures on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 118 comments across hackernews (as of 2026-10-29). **TL;DR:** The discussion pivots away from the benchmark itself toward real-world experiences with AI helping (and sometimes failing at) taxes, with many sharing stories of LLMs catching accountant errors but also making serious mistakes of their own; there's notable skepticism about whether a bare-agent benchmark without task-specific tooling says much. **Sentiment:** 25% positive · 45% mixed · 30% skeptical **The case for** - Several people reported LLMs catching real errors in professional-prepared tax returns that cost or would have cost thousands of dollars. - Some found LLMs useful as a 'second pair of eyes' or for cross-checking a preparer's work before filing. - A few described successfully using AI-heavy workflows to prepare complex returns (including cross-border and business situations) at much lower cost than professional fees. **The pushback** - Multiple commenters noted LLMs are weak at reliable arithmetic/math and need deterministic tool-calling for calculations. - Several expressed concern about sharing sensitive financial data with AI companies due to privacy/leak risks. - Some argued the benchmark is of limited value since it tests bare models without tax-specific scaffolding or tools, which isn't representative of how such agents would be deployed. - The 10% pass rate was called out as being within the benchmark's own error bars, making the leaderboard differences questionable. **By community** - hackernews (mixed): Conversation sprawls across liability-for-tax-errors debates, personal anecdotes of AI catching or causing mistakes, and skepticism about the benchmark's methodology, without a dominant consensus. **Hottest debate:** Whether the benchmark's bare-agent, no-scaffolding setup meaningfully measures real-world AI tax capability or is essentially testing random base-model behavior. **Open questions** - What tools, system prompts, or scaffolding were actually provided to the agents during the benchmark tasks? - What is the human professional baseline pass rate for comparison against the ~10% AI pass rate? - How would performance change with a purpose-built tax harness rather than a general-purpose agent given raw documents? **Highlights** > Is this just giving a general-purpose agent a workspace full of documents and an assignment, then seeing what happens? I feel this doesn't really say anything. It appears they mostly dumped all of this in base models with little to nothing in the way of infrastructure to support it. If you just randomly grab some person off the street and say 'do taxes' they would probably mess up just as badly. The interesting question would be what the models could do in a well engineered harness focused on taxes. That would be reflective of the performance boundary and how serious we would have to take AI in that field. > — [bluejay2387 on hackernews · 1 comments](https://news.ycombinator.com/item?id=50282067) > It didn't really do any better, the difference is well within the error bars for their benchmark. > — [WhitneyLand on hackernews](https://news.ycombinator.com/item?id=50281908) > I'm not sure LLMs will ever be good at doing taxes primarily because they're very bad at even simple math unless it's explained in words like in proofs. A good illustration of this is asking an LLM something like this: "If I put a container on a food scale, tare it, add 100g of peanuts, tare it, weigh 100g of almonds and then remove the container from the scale, what does the scale read?" LLMs completely puke all over themselves on this one. They do not account for not taring after the almonds and they do not see the weight of the container needs to be subtracted. The correct answer is minus (100g + container weight). Although this particular question also exploits another LLM weakness which is physical world concepts. LLM are just guessing. For an LLM to do taxes well, it would need to be trained on many scenarios and it would need a math service that it can use as a skill. > — [bodeadly on hackernews · 3 comments](https://news.ycombinator.com/item?id=50279446) > I sent my business and personal returns through Claude this year before signing, and it turns out they were a horror show of unforgivably stupid mistakes. Pretty infuriating considering what my now former CPA was charging me. He amortized the goodwill of businesses I acquired over 5 years instead of 15 (incredibly basic stuff, which Claude flagged with *CRITICAL ERROR*). When I asked about it, he said he had caught the error in his review but forgot to save after he fixed it. I don't know if there's a good response to making that kind of a mistake, but if there is, it's definitely not that one. > — [idopmstuff on hackernews · 1 comments](https://news.ycombinator.com/item?id=50281907) > As a veteran tax CPA, I think LLMs can be very good at preparing taxes but I would not trust them to complete a tax return start to finish. An LLM that can take data from tax documents and input that data into the correct boxes of tax preparation software has no need for math, it just has to identify data in a pdf, spreadsheet, or handwritten note and transcribe that data to the correct box in professional tax software that then does the math. In your typical local CPA firm, an LLM could replace the unlicensed preparers/interns/early career stage CPAs in filling out the return for the experienced professional to review and adjust according to their specialized knowledge. As with many industries, I think the LLMs are suited at replacing entry level automated turks but lack the skills necessary to fully complete the project. > — [rbhelt on hackernews](https://news.ycombinator.com/item?id=50281324) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50275626) · 186 points · 118 comments ## Community discussion Top comments from developers on daily.dev. **@kartiknvj** · 0 upvotes > Astra missed Anna's August 2027 deadline in all five attempts, and Opus assumed the couple was unmarried in all five. I read a miss that repeats five out of five as a blind spot, and a single pass rate like Sonnet's 10.8% doesn't show which model has which one. At a 95% rubric threshold, could an attempt that skips the deadline still pass if everything else is right? ## Similar posts on daily.dev - [Claude did best on a new benchmark for agents that build agents. It still passed fewer than a quarter of the tests.](https://daily.dev/posts/claude-did-best-on-a-new-benchmark-for-agents-that-build-agents-it-still-passed-fewer-than-a-quarte-rrx7ttpl4) · The New Stack · 2 upvotes · 0 comments - [Maybe AI agents can be lawyers after all](https://daily.dev/posts/maybe-ai-agents-can-be-lawyers-after-all-xbt85nir0) · TechCrunch · 0 upvotes · 0 comments - [The Agent Said It Was Done. The Database Disagreed.](https://daily.dev/posts/the-agent-said-it-was-done-the-database-disagreed--th0ihs2vh) · Hugging Face · 2 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#ai-agents](https://daily.dev/tags/ai-agents), [#claude](https://daily.dev/tags/claude), [#fintech](https://daily.dev/tags/fintech) [View this post on daily.dev](https://daily.dev/posts/dayjob-tax-evaluating-ai-agents-on-50-professional-tax-assignments-tknynvr2r) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Dayjob: Tax – evaluating AI agents on 50 professional tax assignments","url":"https://daily.dev/posts/dayjob-tax-evaluating-ai-agents-on-50-professional-tax-assignments-tknynvr2r","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/dayjob-tax-evaluating-ai-agents-on-50-professional-tax-assignments-tknynvr2r"},"datePublished":"2026-10-28T18:15:12.689Z","dateModified":"2026-10-29T00:11:24.853Z","description":"Surge AI's DAYJOB: Tax benchmark tests AI agents on 50 realistic professional tax assignments across eight practice areas. Claude Sonnet 5.5 leads the...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f34927dfbe6495f6076287abcd1199ca?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f34927dfbe6495f6076287abcd1199ca?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/dayjob-tax-evaluating-ai-agents-on-50-professional-tax-assignments-tknynvr2r","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"ai,ai-agents,claude,fintech","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Dayjob: Tax – evaluating AI agents on 50 professional tax assignments"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/dayjob-tax-evaluating-ai-agents-on-50-professional-tax-assignments-tknynvr2r","comment":[{"@type":"Comment","text":"Astra missed Anna’s August 2027 deadline in all five attempts, and Opus assumed the couple was unmarried in all five. I read a miss that repeats five out of five as a blind spot, and a single pass rate like Sonnet’s 10.8% doesn’t show which model has which one. At a 95% rubric threshold, could an attempt that skips the deadline still pass if everything else is right?","datePublished":"2026-10-29T13:51:12.407Z","url":"https://daily.dev/posts/TKNyNvr2R#c-29TlDNsbm","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/dayjob-tax-evaluating-ai-agents-on-50-professional-tax-assignments-tknynvr2r#faq","mainEntity":[{"@type":"Question","name":"Which AI model performs best on the DAYJOB Tax benchmark for professional tax work?","acceptedAnswer":{"@type":"Answer","text":"Claude Sonnet 5.5 leads the DAYJOB: Tax leaderboard with a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. The benchmark measures pass@1 at a 95% rubric threshold across 50 realistic tax environments covering calculations, supporting evidence, consequential issues, and completed deliverables. Teams choosing an agent for tax or compliance work can track benchmark results like these on daily.dev."}},{"@type":"Question","name":"What is the DAYJOB Tax benchmark and what does it test in AI agents?","acceptedAnswer":{"@type":"Answer","text":"DAYJOB: Tax is a benchmark from Surge AI with 50 environments across eight areas of tax practice, including individual, partnership, estate, and international tax. It tests whether AI agents can navigate messy records, exercise professional judgment, identify issues clients never raised, and produce complete tax returns, workpapers, or advice, not just answer predefined questions. Developers evaluating agentic AI for professional domains can follow benchmarks like this on daily.dev."}},{"@type":"Question","name":"Why do AI agents fail at real-world tax advisory tasks even when they know the tax rules?","acceptedAnswer":{"@type":"Answer","text":"Agents often fail because they don't investigate unverified assumptions or flag deadlines the client never asked about, even when they know relevant rules. In one test, Opus 5.5 assumed a separating couple was unmarried across all five attempts despite records suggesting a valid informal marriage, while Astra correctly flagged the need to verify marital status in four of five attempts. Anyone weighing AI agent reliability for high-stakes work can follow real benchmark failures on daily.dev."}}]} ```