Inference, Serving & Optimization
The metrics that matter
- TTFT (Time To First Token). Latency until the first token appears. Dominated by the prefill (prompt-processing) phase and prompt length. Drives perceived responsiveness in chat.
- TPOT / ITL (Time Per Output Token / inter-token latency). Speed of the decode phase. It determines how fast text streams.
- Throughput. Total tokens/sec across all concurrent requests. The economic metric (tokens per dollar of GPU time).
- Latency vs throughput is a trade-off: batching more requests raises throughput but can raise per-request latency. Serving pushes both via smarter scheduling.
Inference has two distinct phases: prefill (compute-bound, processes the whole prompt in parallel) and decode (memory-bandwidth-bound, one token at a time). Most optimizations target one or the other. Some 2026 stacks even physically disaggregate prefill and decode onto different hardware pools.
Core optimization techniques
- Quantization. Store/compute weights (and sometimes activations/KV-cache) at lower precision (FP8, INT8, INT4). Shrinks memory and boosts throughput/latency at some accuracy cost. Weight-only INT4 (AWQ, GPTQ) is common for squeezing big models onto small GPUs. FP8 is increasingly used for near-lossless speedups on Hopper/Blackwell. See also Local & Self-Hosting Stack for GGUF/MLX formats.
- KV cache. Attention caches the keys/values of all prior tokens so each new token doesn't re-process the sequence. It is essential for speed but grows with batch × sequence length and often becomes the memory bottleneck. PagedAttention (vLLM's signature idea) manages KV cache in non-contiguous "pages" like OS virtual memory, which cuts fragmentation and enables far larger effective batches. Compressing the KV cache (quantized KV, GQA/MLA attention) is a major 2025 to 2026 lever.
- Continuous (in-flight) batching. Instead of waiting for a fixed batch, the scheduler adds/removes requests every step as sequences finish, which keeps the GPU saturated. The single biggest throughput win in modern serving.
- Speculative decoding. A small fast "draft" model proposes several tokens. The big model verifies them in one forward pass and accepts the run that matches. Same output distribution, with materially lower latency when acceptance is high. Variants: self-speculation, Medusa/EAGLE-style multi-head drafting, n-gram/lookahead.
- Distillation (Dataset Engineering) and prompt/prefix caching (reuse KV for shared prompt prefixes, valuable for system prompts and few-shot templates) round out the toolbox.
Serving engines (as of July 2026)
| Engine | Origin | Best at | One gotcha | Key fact |
|---|---|---|---|---|
| vLLM | UC Berkeley → community | High-throughput general serving; the de facto open default | Peak per-GPU latency can trail TensorRT-LLM on NVIDIA | Pioneered PagedAttention; OpenAI-compatible server; broad model + hardware support |
| SGLang | Community | Structured generation, agentic/multi-turn, high throughput | Younger ecosystem than vLLM | RadixAttention for automatic prefix-cache reuse; strong on complex prompting workloads |
| TGI (Text Generation Inference) | Hugging Face | Tight HF ecosystem integration, easy deploy | Historically trailed vLLM/SGLang on raw throughput in some benchmarks | Production-hardened, OpenAI-compatible, good tooling |
| TensorRT-LLM | NVIDIA | Absolute lowest latency / highest throughput on NVIDIA GPUs | NVIDIA-only; build/compile step and steeper setup | Compiles model-specific optimized engines; FP8/FP4 on Hopper/Blackwell |
| LMDeploy | InternLM / community | High-throughput serving (TurboMind), strong on Qwen/InternLM/DeepSeek | Smaller community than vLLM/SGLang | Routinely benchmarked head-to-head with vLLM/SGLang; good mixed-precision support |
Rules of thumb: vLLM is the safe default for most teams; SGLang is strong for agent/structured/high-concurrency workloads and prefix reuse; TensorRT-LLM when you're all-NVIDIA and squeezing every last millisecond/dollar and can afford the build complexity; TGI when you live in the Hugging Face stack. At datacenter scale, NVIDIA Dynamo is the emerging framework for disaggregated prefill/decode serving across GPU pools. Benchmarks flip frequently with each release, so always test on your model, hardware, and traffic shape. For single-GPU/local, see llama.cpp/Ollama in Local & Self-Hosting Stack.