Process a Million Documents Overnight: Batch Inference End-to-End
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Running a million LLM requests through a real-time API is slow, expensive, and fragile. Batch inference — packaging all requests into JSONL files, submitting them to the platform, and collecting results when done — uses a separate quota, costs up to half the real-time rate, and handles retries automatically. The post walks through a complete Python pipeline: sizing and splitting 1M support tickets into 40 batch files of 25,000 requests each, submitting with idempotent request IDs, polling for completion, and parsing/validating results. A cost model shows GPT-5 mini batch at $387.50 vs $775 real-time for the same job, while GPT-5 nano drops the bill to $77.50 with an 8% category-violation rate that a validation-and-retry pass can handle. Measured evaluation data (50 docs, two models, two prompt versions) backs the model-selection recommendation.
Table of contents
The job we are runningWhy real-time inference is the wrong tool hereWhat you need before startingPlanning around the limitsBuilding the input filesUploading files and creating jobsMonitoring 40 jobsHandling failuresRetrieving and joining resultsThe itemized billWhen batch beats real-time, and when to self-hostClosingReferencesQuestions this post answers
How much cheaper is batch inference vs real-time API for processing a million documents with GPT-5 mini?
Batch inference on DigitalOcean costs up to half the real-time rate. For a million documents with ~1,500 input and ~200 output tokens each, GPT-5 mini batch totals $387.50 ($187.50 input + $200 output) versus $775 at real-time rates — a saving of $387.50. GPT-5 nano batch drops the same job to $77.50, though it showed an 8% category-violation rate in a 50-document evaluation. Teams deciding between batch and real-time LLM pricing for large document jobs track model cost comparisons like these on daily.dev.
What are the file size and request limits for DigitalOcean batch inference jobs?
Each batch input file is capped at 200 MB and 50,000 requests, and the default token quota is 10 billion tokens per model per account. At roughly 6.5 KB per JSONL line (1,200-token document plus prompt and wrapper), the 200 MB file size limit fills first at about 31,500 documents — well below the 50,000-request cap. Using 25,000 requests per file gives comfortable headroom, requiring 40 files for one million documents. Engineers splitting large LLM workloads into batch files find the latest limit changes for DigitalOcean inference on daily.dev.
How often does GPT-5 nano return an out-of-list category value when doing JSON classification with a strict prompt?
In a 50-document evaluation, GPT-5 nano returned an invalid category (one not on the specified list) 4 times out of 50 — an 8% violation rate — both before and after prompt tightening. GPT-5 mini dropped from 2 violations to 0 after the prompt was tightened with an explicit 'never invent another category' instruction. The nano violation rate was unchanged by the prompt fix, making a code-level validation check essential regardless of prompt wording. Developers choosing between nano and mini for classification pipelines at scale follow model behavior findings like this on daily.dev.