A detailed cost breakdown for running a retrieval-augmented generation (RAG) pipeline over a 100,000-document corpus on DigitalOcean shows that embedding and vector storage are nearly free — about $6.86 one-time and roughly $1/month respectively — while reranking and answer generation account for over 99% of ongoing model spend, scaling linearly with query traffic. At 1,000 queries/day the whole system costs about $98/month, rising to roughly $4,000/month at 100,000 queries/day. The three highest-leverage cost levers identified are the choice of generation model (a 4x price swing between gpt-oss-120b and Llama 3.3 70B), the number of chunks reranked per query, and the crossover point (~$3,200/month) where switching from serverless to dedicated GPU inference becomes cheaper. The piece also walks through building the full pipeline: chunking, embeddings via Qwen3 Embedding 0.6B, storage in Managed PostgreSQL with pgvector and HNSW indexing, LLM-based reranking with DeepSeek V4 Flash, and answer generation with gpt-oss-120b, including optional batch-inference chunk enrichment.
Table of contents
The cost modelWhy the bill looks like thisHow the economics compare to running this elsewhereThe pipeline behind the numbersThe managed alternativeConclusionReferencesQuestions this post answers
What does it cost to run a RAG pipeline over 100,000 documents in production?
Roughly $98 per month at 1,000 queries per day, rising to about $496 at 10,000 queries per day and $4,000 at 100,000 queries per day, using Qwen3 Embedding 0.6B, DeepSeek V4 Flash for reranking, and gpt-oss-120b for answer generation on DigitalOcean. Embedding the full 100,000-document corpus is a one-time $6.86, and vector storage adds about $1 per month; reranking and generation make up over 99% of ongoing model spend and scale with query volume. daily.dev surfaces cost-breakdown deep dives like this for teams budgeting a RAG deployment.
Why is the reranking step the most expensive part of a RAG pipeline at high query volume?
The reranker reads all 20 retrieved candidate chunks per query, about 10,400 input tokens, roughly four times more tokens than the answer-generation model sees over its top 5 chunks. At high traffic this token volume makes reranking the single largest line item, even though model price per token is relatively low; reducing candidates to 10, reranking only ambiguous queries, or using a cheaper dedicated reranker (like BGE Reranker v2 m3) can cut this cost significantly. Track token-cost tradeoffs like this reranking bottleneck on daily.dev before scaling a RAG service.
When does dedicated GPU inference become cheaper than serverless inference for a RAG pipeline?
A single NVIDIA H100 dedicated instance costs a flat $4.41 per hour, about $3,220 per month, regardless of query volume, while serverless inference pricing scales linearly with traffic. Once monthly serverless spend on generation and reranking approaches that flat rate, roughly around the 100,000-queries-per-day tier in this cost model, dedicated capacity for an open-weight model starts winning on both price and latency consistency. Developers comparing serverless versus dedicated inference costs can find breakdowns like this on daily.dev.