GPU utilization in production Kubernetes fleets averages just 5%, making idle capacity the primary driver of LLM inference costs. Five concrete optimization levers are covered: (1) GPU sharing via NVIDIA MIG partitions or time-slicing for multi-tenant workloads; (2) continuous batching, speculative decoding, and prefix caching in vLLM for 3-5x throughput gains; (3) quantization format selection by GPU architecture (FP8 on H100/H200, AWQ on A100, INT4 on T4/L4) with specific accuracy trade-offs; (4) right-sizing GPU requests using DCGM metrics rather than worst-case assumptions; and (5) inference-aware autoscaling with KEDA using queue depth and tokens-per-second signals, including scale-to-zero for idle endpoints. Benchmark data shows continuous batching at batch size 8 reduces Llama 3.1 70B inference cost from $0.60-0.80 to $0.15-0.25 per million tokens on an H100.
Table of contents
Lever 1: GPU Sharing with MIG and Time-SlicingLever 2: Continuous Batching, Speculative Decoding, and Prefix CachingLever 3: Quantization and llm inference cost Per TokenLever 4: Right-Sizing GPU Requests to Reduce Inference CostLever 5: Autoscaling and Scale-to-ZeroHow Cast AI Reduces LLM Inference Cost at ScaleFrequently Asked QuestionsQuestions this post answers
How much does continuous batching reduce LLM inference cost compared to single-stream mode?
Continuous batching at batch size 8 reduces Llama 3.1 70B inference cost from $0.60-0.80 per million tokens (single-stream) to $0.15-0.25 per million tokens — a 3-4x reduction with no additional hardware. These figures are from benchmark testing on an H100 80GB SXM5 with vLLM 0.5+, 512-token prompt plus 256-token completion, at H100 spot pricing of approximately $2-4/hr. The gain applies to contexts up to ~2K tokens. Teams optimizing LLM serving costs track batching benchmarks and GPU pricing changes on daily.dev.
Can I run a 70B parameter model on a single A100 80GB GPU?
Yes, using AWQ (4-bit) quantization. A 70B model quantized with AWQ requires approximately 35-37GB for weights, leaving around 40GB for KV cache on a single 80GB device. INT8 keeps each parameter at 1 byte, so a 70B INT8 model weighs ~70GB and fits an A100 80GB but leaves minimal KV cache headroom. FP8 requires an H100 or H200 — the A100 lacks native FP8 Tensor Cores. Developers choosing between A100 and H100 for 70B model deployments find hardware trade-off discussions on daily.dev.
When does speculative decoding help reduce LLM generation latency and when does it not?
Speculative decoding is most effective at batch size 1-2 for latency-sensitive chat or copilot workloads, where it can reduce generation latency by 30-50% when a suitable small draft model is available. At high concurrency or large batch sizes, running the draft model in parallel reduces the net benefit. The gain comes because verifying a batch of draft tokens costs nearly the same compute as verifying one token in autoregressive generation. Engineers tuning vLLM serving configurations for production chat workloads follow inference optimization threads on daily.dev.