--- title: Building a Unified Pipeline for AI Token Spend | daily.dev description: Ramp's engineering team built a real-time AI token spend tracking pipeline to help finance teams monitor and control AI costs across providers. The system... canonical: https://daily.dev/posts/building-a-unified-pipeline-for-ai-token-spend-ivnfwpnyk twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Building a Unified Pipeline for AI Token Spend | daily.dev og:description: Ramp's engineering team built a real-time AI token spend tracking pipeline to help finance teams monitor and control AI costs across providers. The system... og:url: https://daily.dev/posts/building-a-unified-pipeline-for-ai-token-spend-ivnfwpnyk og:image: https://api.daily.dev/og/posts/iVNfWPNYk.png og:image:alt: Building a Unified Pipeline for AI Token Spend og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Building a Unified Pipeline for AI Token Spend **[Ramp Engineering](https://daily.dev/sources/ramp_eng)** · 9 min read · 0 upvotes · 0 comments ## Summary Ramp's engineering team built a real-time AI token spend tracking pipeline to help finance teams monitor and control AI costs across providers. The system ingests usage events from LiteLLM and OpenRouter via webhooks, streams them through Kafka, and stores them in ClickHouse using a ReplacingMergeTree for deduplication. The post details the full architecture including OAuth2 validation, row-level security for multi-tenancy, and a normalized schema with high-precision cost fields. It also covers practical cost leaks discovered in production: phantom reasoning tokens from a LiteLLM upgrade, oversized models for simple tasks, runaway agent loops, geo pricing premiums, and prompt bloat. Attribution dimensions include per-user, per-team, per-project, and per-use-case breakdowns, enabling CFO-level reporting beyond raw token counts. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [AI Cost Visibility: How to Track and Optimize Token Spend Before the Invoice Arrives](https://daily.dev/posts/ai-cost-visibility-how-to-track-and-optimize-token-spend-before-the-invoice-arrives-133h7kulz) · Telerik · 1 upvotes · 1 comments - [5 Open-Source Tools to Control Your AI API Costs at the Code Level](https://daily.dev/posts/5-open-source-tools-to-control-your-ai-api-costs-at-the-code-level-wncyok6uu) · finout · 0 upvotes · 0 comments - [Five keys to controlling AI token costs](https://daily.dev/posts/five-keys-to-controlling-ai-token-costs-hu2lr27uv) · InfoWorld · 0 upvotes · 0 comments - [AI and Token Cost Management on Kubernetes: Attributing Inference Spend to Teams](https://daily.dev/posts/ai-and-token-cost-management-on-kubernetes-attributing-inference-spend-to-teams-b1jtrhm4i) · Cast AI · 0 upvotes · 0 comments - [The part of AI nobody posts about: the invoice](https://daily.dev/posts/the-part-of-ai-nobody-posts-about-the-invoice-s9lbth87q) · Medium · 1 upvotes · 0 comments --- Tags: [#kafka](https://daily.dev/tags/kafka), [#opentelemetry](https://daily.dev/tags/opentelemetry), [#clickhouse](https://daily.dev/tags/clickhouse) [View this post on daily.dev](https://daily.dev/posts/building-a-unified-pipeline-for-ai-token-spend-ivnfwpnyk) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Building a Unified Pipeline for AI Token Spend","url":"https://daily.dev/posts/building-a-unified-pipeline-for-ai-token-spend-ivnfwpnyk","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/building-a-unified-pipeline-for-ai-token-spend-ivnfwpnyk"},"datePublished":"2026-07-08T19:14:48.613Z","dateModified":"2026-07-08T19:15:33.223Z","description":"Ramp's engineering team built a real-time AI token spend tracking pipeline to help finance teams monitor and control AI costs across providers. The system...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0a35f01a520a823373531c4e05dac383?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0a35f01a520a823373531c4e05dac383?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Ramp Engineering","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Ramp Engineering","logo":"https://media.daily.dev/image/upload/s--4x5H_pLU--/f_auto/v1704280953/logos/ramp.jpg","url":"https://daily.dev/sources/ramp_eng"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/building-a-unified-pipeline-for-ai-token-spend-ivnfwpnyk","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kafka,opentelemetry,clickhouse","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Ramp Engineering","item":"https://daily.dev/sources/ramp_eng"},{"@type":"ListItem","position":3,"name":"Building a Unified Pipeline for AI Token Spend"}]} ```