--- title: Five agent frameworks billed you for fewer tokens than... description: A widespread bug exists across at least five independent AI agent frameworks (Pipecat, LiveKit Agents, Haystack, mcp-use, LlamaIndex) where token cost... canonical: https://daily.dev/posts/five-agent-frameworks-billed-you-for-fewer-tokens-than-anthropic-charged-them-jtsb8vgyj twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Five agent frameworks billed you for fewer tokens than Anthropic charged them | daily.dev og:description: A widespread bug exists across at least five independent AI agent frameworks (Pipecat, LiveKit Agents, Haystack, mcp-use, LlamaIndex) where token cost... og:url: https://daily.dev/posts/five-agent-frameworks-billed-you-for-fewer-tokens-than-anthropic-charged-them-jtsb8vgyj og:image: https://api.daily.dev/og/posts/jTsB8vGYJ.png og:image:alt: Five agent frameworks billed you for fewer tokens than Anthropic charged them og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Five agent frameworks billed you for fewer tokens than Anthropic charged them **[Agentic Engineering](https://daily.dev/sources/agentic)** · [@arthiarumugam](https://daily.dev/arthiarumugam) · 2 min read · 0 upvotes · 1 comments ## Summary A widespread bug exists across at least five independent AI agent frameworks (Pipecat, LiveKit Agents, Haystack, mcp-use, LlamaIndex) where token cost calculations omit Anthropic's prompt cache counters. Anthropic reports `input_tokens` net of the cache, with cached tokens on separate `cache_read_input_tokens` and `cache_creation_input_tokens` fields. Code that sums only `input_tokens + output_tokens` can under-report costs by an order of magnitude, with the error worsening as cache hit rates improve. A CLI tool called `cachecheck` (installable via `uvx cachecheck .`) statically detects this pattern with no dependencies, exits with code 1 on findings, and can serve as a CI gate. It correctly ignores OpenAI-shaped files where cached tokens are nested inside `prompt_tokens` to avoid false positives. ## Content Anthropic reports `input_tokens` **net of the prompt cache**. The cached tokens come back on separate counters, `cache_read_input_tokens` and `cache_creation_input_tokens`, and there is no field holding the total. So this line is wrong, and it is everywhere: ``` total = usage.input_tokens + usage.output_tokens ``` It is not wrong in an obvious way. It returns a number. The number looks plausible. Nothing raises. **What it costs.** A voice agent re-reads its conversation history on every turn, which is close to the ideal case for a prompt cache and close to the worst case for this bug. On a turn where the provider billed 2,100 input tokens with 2,000 served from cache, `input_tokens` on its own reads 100, and the framework reported a total of 150 against the 2,100 Anthropic actually billed. Anything costing off that span under-reports by an order of magnitude, and the error grows with cache hit rate, so it is worst exactly where caching is working best. **Five independent codebases, same shape.** Pipecat (Daily), merged. LiveKit Agents, merged by their co-founder. Haystack at deepset, merged. mcp-use, open. LlamaIndex, closed because that subsystem is being retired. Finding it once is luck. Finding it in five codebases that do not share a line of code is a pattern, and a pattern is something you can check for. ```bash uvx cachecheck . ``` No dependencies. Exit code 1 when it finds something, so it works as a CI gate. Three rules: a total of input plus output in a file that never reads the cache counter, computing with `input_tokens` while never reading the cache at all, and reading cache reads but not cache writes, which Anthropic bills above the base rate. **What it refuses to flag.** OpenAI puts `cached_tokens` inside `prompt_tokens`. Adding it there double counts, so an OpenAI-shaped file is correct as written and is left alone. Getting that backwards would be the same error in the opposite direction. An earlier version flagged a type declaration and a docstring showing sample output. Both name a field rather than compute with it. A checker that cries wolf on correct code gets switched off, and then it catches nothing. It is patterns, not a type checker. Read the evidence line before acting on a finding. https://github.com/arthi-arumugam-git/cachecheck ``` ## Similar posts on daily.dev - [Monitor prompt caching to optimize your token usage](https://daily.dev/posts/monitor-prompt-caching-to-optimize-your-token-usage-bwzwycrgn) · Datadog · 0 upvotes · 0 comments - [The Hidden Cache You’re Paying For But Not Using](https://daily.dev/posts/the-hidden-cache-you-re-paying-for-but-not-using-0w5qklioj) · Medium · 0 upvotes · 0 comments - [Anthropic quietly nerfed Claude Code's 1-hour cache, and your token budget is paying the price](https://daily.dev/posts/anthropic-quietly-nerfed-claude-code-s-1-hour-cache-and-your-token-budget-is-paying-the-price-1jobhvzkb) · XDA Developers · 1 upvotes · 0 comments - [Prompt Caching for Anthropic and OpenAI Models: Building Cost-Efficient AI Systems](https://daily.dev/posts/prompt-caching-for-anthropic-and-openai-models-building-cost-efficient-ai-systems-mjzh92xkq) · DigitalOcean · 2 upvotes · 0 comments --- Tags: [#anthropic](https://daily.dev/tags/anthropic), [#ai-inference](https://daily.dev/tags/ai-inference), [#llmops](https://daily.dev/tags/llmops) [View this post on daily.dev](https://daily.dev/posts/five-agent-frameworks-billed-you-for-fewer-tokens-than-anthropic-charged-them-jtsb8vgyj) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/five-agent-frameworks-billed-you-for-fewer-tokens-than-anthropic-charged-them-jtsb8vgyj","headline":"Five agent frameworks billed you for fewer tokens than Anthropic charged them","text":"A widespread bug exists across at least five independent AI agent frameworks (Pipecat, LiveKit Agents, Haystack, mcp-use, LlamaIndex) where token cost calculations omit Anthropic's prompt cache counters. Anthropic reports `input_tokens` net of the cache, with cached tokens on separate `cache_read_input_tokens` and `cache_creation_input_tokens` fields. Code that sums only `input_tokens + output_tokens` can under-report costs by an order of magnitude, with the error worsening as cache hit rates improve. A CLI tool called `cachecheck` (installable via `uvx cachecheck .`) statically detects this pattern with no dependencies, exits with code 1 on findings, and can serve as a CI gate. It correctly ignores OpenAI-shaped files where cached tokens are nested inside `prompt_tokens` to avoid false positives.","url":"https://daily.dev/posts/five-agent-frameworks-billed-you-for-fewer-tokens-than-anthropic-charged-them-jtsb8vgyj","datePublished":"2026-08-08T08:56:26.801Z","dateModified":"2026-09-13T21:09:51.326Z","author":{"@type":"Person","name":"Arthi Arumugam","url":"https://daily.dev/arthiarumugam","image":"https://avatars.githubusercontent.com/u/62213284?v=4","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":20}},"interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/squads/agentic","name":"Agentic Engineering"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Agentic Engineering","item":"https://daily.dev/squads/agentic"},{"@type":"ListItem","position":3,"name":"Five agent frameworks billed you for fewer tokens than Anthropic charged them"}]} ```