--- title: Cutting the LLM bill by 78% made our queue grow five... description: Switching an LLM enrichment call from gpt-5.4-nano to gpt-oss-120b on DeepInfra cut the per-call cost from $11.8/day to $2.6/day while eval scores held steady.... canonical: https://daily.dev/posts/cutting-the-llm-bill-by-78-made-our-queue-grow-five-hours-a-day-wpu5l6fur twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Cutting the LLM bill by 78% made our queue grow five hours a day | daily.dev og:description: Switching an LLM enrichment call from gpt-5.4-nano to gpt-oss-120b on DeepInfra cut the per-call cost from $11.8/day to $2.6/day while eval scores held steady.... og:url: https://daily.dev/posts/cutting-the-llm-bill-by-78-made-our-queue-grow-five-hours-a-day-wpu5l6fur og:image: https://api.daily.dev/og/posts/WpU5L6FuR.png og:image:alt: Cutting the LLM bill by 78% made our queue grow five hours a day og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Cutting the LLM bill by 78% made our queue grow five hours a day **[AlphAI](https://daily.dev/sources/alphai)** · [@mikhailmakeev](https://daily.dev/mikhailmakeev) · 2 min read · 1 upvotes · 2 comments ## Summary Switching an LLM enrichment call from gpt-5.4-nano to gpt-oss-120b on DeepInfra cut the per-call cost from $11.8/day to $2.6/day while eval scores held steady. But the new model's latency roughly doubled (8s to 16.7s), and because the worker processed articles strictly serially, throughput dropped below the daily inflow rate, causing the enrichment queue to grow by about five hours of lag per day. The fix was scaling the worker to 3 replicas (concurrency was already safe via select_for_update with skip_locked and an upsert keyed on article id), which restored throughput to about 590 articles/hour and drained the queue. The lesson: evals that measure quality and price per call must also account for latency multiplied by daily volume, and a flat hourly processing rate (instead of following diurnal input patterns) signals a worker pinned at its throughput ceiling. ## Content We run one LLM call per ingested news article: tickers, a category, a relevance score. On the 19th I moved that call from gpt-5.4-nano to gpt-oss-120b on DeepInfra. The eval said quality held and the bill for that call went from about $11.8 a day to about $2.6. I watched the first 40 minutes, which were quiet, and went to bed. Next morning the day's enriched count was 5,126 rows against 6,849 the day before, with more articles coming in, not fewer. The queue held 924 articles, the oldest six hours old. Median enrichment lag was 353 minutes and growing by about five hours per day. Nothing was broken. The worker is strictly serial: one article, one call, one save. Its throughput is one over call latency. The old model answered in about 8 seconds, the new one in about 16.7, so the ceiling dropped to roughly 210 articles an hour against an inflow of about 6,500 a day. The eval measured quality and price per call. Nobody multiplied latency by daily volume. The fix was scaling the worker to 3 replicas, no code change. Concurrency was already safe: a row is claimed with select_for_update(skip_locked=True) and the save is an upsert keyed on the article id, so a double enrichment cannot produce a duplicate. Twelve minutes after the scale the rate was about 590 an hour and the queue was draining. The tell I now check after any provider swap: a healthy consumer's hourly rate follows the shape of its input, which for news is strongly diurnal. A flat line around the clock means the worker is pinned at its ceiling and the queue is absorbing the peaks. ## Community discussion Top comments from developers on daily.dev. **@trevorsuna** · 1 upvotes > The cost-per-call eval missed the system bottleneck because latency multiplied by daily volume was the real constraint. A small load model covering arrival rate, service time, concurrency, and backlog age should be part of every provider swap, even when quality and unit price look better. ## Similar posts on daily.dev - [Process a Million Documents Overnight: Batch Inference End-to-End](https://daily.dev/posts/process-a-million-documents-overnight-batch-inference-end-to-end-fjrmed2sj) · DigitalOcean Community · 0 upvotes · 0 comments - [How To Reduce LLM Token Costs by 70–90%](https://daily.dev/posts/how-to-reduce-llm-token-costs-by-70-90--lqaosovr2) · C\# Corner · 1 upvotes · 0 comments - [The LLM Inference Trilemma: Throughput, Latency, Cost](https://daily.dev/posts/the-llm-inference-trilemma-throughput-latency-cost-ws3duzsii) · DigitalOcean · 0 upvotes · 0 comments - [How I massively improved my AI inference performance without buying new hardware](https://daily.dev/posts/how-i-massively-improved-my-ai-inference-performance-without-buying-new-hardware-jgyj1odfm) · Red Hat Developer · 0 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#llm](https://daily.dev/tags/llm), [#django](https://daily.dev/tags/django), [#finops](https://daily.dev/tags/finops) [View this post on daily.dev](https://daily.dev/posts/cutting-the-llm-bill-by-78-made-our-queue-grow-five-hours-a-day-wpu5l6fur) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/cutting-the-llm-bill-by-78-made-our-queue-grow-five-hours-a-day-wpu5l6fur","headline":"Cutting the LLM bill by 78% made our queue grow five hours a day","text":"Switching an LLM enrichment call from gpt-5.4-nano to gpt-oss-120b on DeepInfra cut the per-call cost from $11.8/day to $2.6/day while eval scores held steady. But the new model's latency roughly doubled (8s to 16.7s), and because the worker processed articles strictly serially, throughput dropped below the daily inflow rate, causing the enrichment queue to grow by about five hours of lag per day. The fix was scaling the worker to 3 replicas (concurrency was already safe via select_for_update with skip_locked and an upsert keyed on article id), which restored throughput to about 590 articles/hour and drained the queue. The lesson: evals that measure quality and price per call must also account for latency multiplied by daily volume, and a flat hourly processing rate (instead of following diurnal input patterns) signals a worker pinned at its throughput ceiling.","url":"https://daily.dev/posts/cutting-the-llm-bill-by-78-made-our-queue-grow-five-hours-a-day-wpu5l6fur","datePublished":"2026-08-24T13:10:52.067Z","dateModified":"2026-09-13T21:29:37.899Z","author":{"@type":"Person","name":"Mikhail Makeev","url":"https://daily.dev/mikhailmakeev","image":"https://lh3.googleusercontent.com/a/ACg8ocIjcvDU1gmJ9-KJDeTYqY2rMdl5QxcWYtLTGSXrj8ubF4nwar_w=s96-c","description":"Owner of alphai.io","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":3600}},"image":"https://media.daily.dev/image/upload/s--BKm8Hvih--/f_auto/v1789277110/posts/WpU5L6FuR?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":2}],"comment":[{"@type":"Comment","text":"The cost-per-call eval missed the system bottleneck because latency multiplied by daily volume was the real constraint. A small load model covering arrival rate, service time, concurrency, and backlog age should be part of every provider swap, even when quality and unit price look better.","datePublished":"2026-08-25T14:51:59.011Z","url":"https://daily.dev/posts/WpU5L6FuR#c-Dl5xYHtjs","author":{"@type":"Person","name":"Trevor Suna","url":"https://daily.dev/trevorsuna","image":"https://media.daily.dev/image/upload/s--dZ7gXxpp--/f_auto/v1784081551/avatars/avatar_EMoP47rpuw8DNjhp6R1b6?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/squads/alphai","name":"AlphAI"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AlphAI","item":"https://daily.dev/squads/alphai"},{"@type":"ListItem","position":3,"name":"Cutting the LLM bill by 78% made our queue grow five hours a day"}]} ```