Self-hosted LLM economics come down to two levers: reducing operating cost and increasing tokens actually served. Cost per token equals total operating cost (hardware, software, people, facilities) divided by tokens processed, and the piece walks through right-sizing GPU instances, autoscaling replicas, consolidating workloads to fill idle GPU cycles, and measuring real-world throughput versus theoretical maximum with tools like GuideLLM. It also covers how model size and quantization (e.g., FP8) affect hardware requirements, how input, cached, and output tokens carry different effective GPU costs, and how vLLM and llm-d features like KV cache offloading, intelligent routing, and speculative decoding can improve throughput and effective cost per token.
Table of contents
A simple formulaSelf-hosted LLM cost breakdownTokens processedTuning with vLLM and llm-dThe impact of model choiceInput tokens, cached tokens, and output tokensFinal thoughtsQuestions this post answers
How do you calculate the actual cost per token for a self-hosted LLM deployment?
Cost per token equals total operating cost divided by tokens actually processed over the same period, not the theoretical maximum throughput. For example, $50,000 in monthly infrastructure cost divided by 500 million tokens processed works out to about 10 cents per million tokens. Reducing spend or increasing real usage both lower this figure. daily.dev surfaces practical breakdowns like this for engineers sizing GPU budgets against real token volume.
Why are output tokens more expensive to generate than input tokens on a GPU-hosted LLM?
Output tokens are generated sequentially, one at a time, which is far less parallelizable than processing input tokens in a single pass. Input tokens benefit from continuous batching and are relatively efficient, while cached tokens reused from the KV cache are cheap since they're looked up rather than recomputed, making output-heavy workloads the priciest per token. Understanding GPU token economics helps teams budgeting inference costs; daily.dev tracks these serving optimization techniques.
How much smaller GPU hardware do you need to serve Llama-3.1-8B-Instruct versus Llama-3.3-70B-Instruct?
Llama-3.1-8B-Instruct fits on a single H100 or even a budget GPU like an L40S, while Llama-3.3-70B-Instruct can require up to four H100s per instance. Choosing the smaller model when quality is sufficient cuts hardware cost sharply and increases achievable throughput per dollar. Comparing model sizes against GPU budgets is a recurring decision daily.dev helps engineers stay informed on.