--- title: Process a Million Documents Overnight: Batch Inference... description: Running a million LLM requests through a real-time API is slow, expensive, and fragile. Batch inference — packaging all requests into JSONL files, submitting... canonical: https://daily.dev/posts/process-a-million-documents-overnight-batch-inference-end-to-end-fjrmed2sj twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Process a Million Documents Overnight: Batch Inference End-to-End | daily.dev og:description: Running a million LLM requests through a real-time API is slow, expensive, and fragile. Batch inference — packaging all requests into JSONL files, submitting... og:url: https://daily.dev/posts/process-a-million-documents-overnight-batch-inference-end-to-end-fjrmed2sj og:image: https://api.daily.dev/og/posts/FJRmEd2sj.png og:image:alt: Process a Million Documents Overnight: Batch Inference End-to-End og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Process a Million Documents Overnight: Batch Inference End-to-End **[DigitalOcean Community](https://daily.dev/sources/do_community)** · 22 min read · 0 upvotes · 0 comments ## Summary Running a million LLM requests through a real-time API is slow, expensive, and fragile. Batch inference — packaging all requests into JSONL files, submitting them to the platform, and collecting results when done — uses a separate quota, costs up to half the real-time rate, and handles retries automatically. The post walks through a complete Python pipeline: sizing and splitting 1M support tickets into 40 batch files of 25,000 requests each, submitting with idempotent request IDs, polling for completion, and parsing/validating results. A cost model shows GPT-5 mini batch at $387.50 vs $775 real-time for the same job, while GPT-5 nano drops the bill to $77.50 with an 8% category-violation rate that a validation-and-retry pass can handle. Measured evaluation data (50 docs, two models, two prompt versions) backs the model-selection recommendation. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much cheaper is batch inference vs real-time API for processing a million documents with GPT-5 mini? Batch inference on DigitalOcean costs up to half the real-time rate. For a million documents with ~1,500 input and ~200 output tokens each, GPT-5 mini batch totals $387.50 ($187.50 input + $200 output) versus $775 at real-time rates — a saving of $387.50. GPT-5 nano batch drops the same job to $77.50, though it showed an 8% category-violation rate in a 50-document evaluation. _Teams deciding between batch and real-time LLM pricing for large document jobs track model cost comparisons like these on daily.dev._ ### What are the file size and request limits for DigitalOcean batch inference jobs? Each batch input file is capped at 200 MB and 50,000 requests, and the default token quota is 10 billion tokens per model per account. At roughly 6.5 KB per JSONL line (1,200-token document plus prompt and wrapper), the 200 MB file size limit fills first at about 31,500 documents — well below the 50,000-request cap. Using 25,000 requests per file gives comfortable headroom, requiring 40 files for one million documents. _Engineers splitting large LLM workloads into batch files find the latest limit changes for DigitalOcean inference on daily.dev._ ### How often does GPT-5 nano return an out-of-list category value when doing JSON classification with a strict prompt? In a 50-document evaluation, GPT-5 nano returned an invalid category (one not on the specified list) 4 times out of 50 — an 8% violation rate — both before and after prompt tightening. GPT-5 mini dropped from 2 violations to 0 after the prompt was tightened with an explicit 'never invent another category' instruction. The nano violation rate was unchanged by the prompt fix, making a code-level validation check essential regardless of prompt wording. _Developers choosing between nano and mini for classification pipelines at scale follow model behavior findings like this on daily.dev._ ## Similar posts on daily.dev - [We Classified 250,000 Records with DigitalOcean Batch Inference for $7.04](https://daily.dev/posts/we-classified-250-000-records-with-digitalocean-batch-inference-for-7-04-eaxahugw2) · DigitalOcean Community · 0 upvotes · 0 comments - [Comparing token economics across 42 AI models](https://daily.dev/posts/comparing-token-economics-across-42-ai-models-htlm2dnb1) · Neon · 2 upvotes · 0 comments - [From 4 Weeks to 45 Minutes: Designing a Document Extraction System for 4,700\+ PDFs](https://daily.dev/posts/from-4-weeks-to-45-minutes-designing-a-document-extraction-system-for-4-700-pdfs-boven7x5o) · Towards Data Science · 1 upvotes · 0 comments - [Cheapest Batch Inference for Low-Cost AI Inference: 6 Providers Compared \(2026\)](https://daily.dev/posts/cheapest-batch-inference-for-low-cost-ai-inference-6-providers-compared-2026--urh3juuuq) · DigitalOcean Community · 0 upvotes · 0 comments - [LLM Inference Cost Optimization: Run AI Inference for Less](https://daily.dev/posts/llm-inference-cost-optimization-run-ai-inference-for-less-syqrz0vbv) · Cast AI · 0 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#llm](https://daily.dev/tags/llm), [#finops](https://daily.dev/tags/finops), [#digitalocean](https://daily.dev/tags/digitalocean) [View this post on daily.dev](https://daily.dev/posts/process-a-million-documents-overnight-batch-inference-end-to-end-fjrmed2sj) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Process a Million Documents Overnight: Batch Inference End-to-End","url":"https://daily.dev/posts/process-a-million-documents-overnight-batch-inference-end-to-end-fjrmed2sj","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/process-a-million-documents-overnight-batch-inference-end-to-end-fjrmed2sj"},"datePublished":"2026-08-11T09:47:43.645Z","dateModified":"2026-08-11T09:48:59.049Z","description":"Running a million LLM requests through a real-time API is slow, expensive, and fragile. Batch inference — packaging all requests into JSONL files, submitting...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8f2172bba436e9bce87684915dfd91c6?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8f2172bba436e9bce87684915dfd91c6?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"DigitalOcean Community","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DigitalOcean Community","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/c1b9d07730e34ea388c39a498a753d6c","url":"https://daily.dev/sources/do_community"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/process-a-million-documents-overnight-batch-inference-end-to-end-fjrmed2sj","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,llm,finops,digitalocean","timeRequired":"PT22M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DigitalOcean Community","item":"https://daily.dev/sources/do_community"},{"@type":"ListItem","position":3,"name":"Process a Million Documents Overnight: Batch Inference End-to-End"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/process-a-million-documents-overnight-batch-inference-end-to-end-fjrmed2sj#faq","mainEntity":[{"@type":"Question","name":"How much cheaper is batch inference vs real-time API for processing a million documents with GPT-5 mini?","acceptedAnswer":{"@type":"Answer","text":"Batch inference on DigitalOcean costs up to half the real-time rate. For a million documents with ~1,500 input and ~200 output tokens each, GPT-5 mini batch totals $387.50 ($187.50 input + $200 output) versus $775 at real-time rates — a saving of $387.50. GPT-5 nano batch drops the same job to $77.50, though it showed an 8% category-violation rate in a 50-document evaluation. Teams deciding between batch and real-time LLM pricing for large document jobs track model cost comparisons like these on daily.dev."}},{"@type":"Question","name":"What are the file size and request limits for DigitalOcean batch inference jobs?","acceptedAnswer":{"@type":"Answer","text":"Each batch input file is capped at 200 MB and 50,000 requests, and the default token quota is 10 billion tokens per model per account. At roughly 6.5 KB per JSONL line (1,200-token document plus prompt and wrapper), the 200 MB file size limit fills first at about 31,500 documents — well below the 50,000-request cap. Using 25,000 requests per file gives comfortable headroom, requiring 40 files for one million documents. Engineers splitting large LLM workloads into batch files find the latest limit changes for DigitalOcean inference on daily.dev."}},{"@type":"Question","name":"How often does GPT-5 nano return an out-of-list category value when doing JSON classification with a strict prompt?","acceptedAnswer":{"@type":"Answer","text":"In a 50-document evaluation, GPT-5 nano returned an invalid category (one not on the specified list) 4 times out of 50 — an 8% violation rate — both before and after prompt tightening. GPT-5 mini dropped from 2 violations to 0 after the prompt was tightened with an explicit 'never invent another category' instruction. The nano violation rate was unchanged by the prompt fix, making a code-level validation check essential regardless of prompt wording. Developers choosing between nano and mini for classification pipelines at scale follow model behavior findings like this on daily.dev."}}]} ```