A deep-dive podcast episode with Baseten engineers Philip Kiely and Ali Taha covering the full lifecycle of inference engineering for large language models. Topics include cache-aware routing, disaggregated prefill and decode, speculative decoding, quantization strategies (including how quantization errors can cancel out for better fidelity), retrofitting vision encoders onto existing LLMs, nondeterministic failures caused by GPU kernel race conditions, model parallelism, NVIDIA Dynamo, diffusion vs. autoregressive video generation, and the convergence of training and inference. Key insight: Baseten demonstrated 20% better throughput than NVIDIA's own quantization by mathematically selecting layers whose quantization errors cancel each other out, validated via KL divergence on logit distributions.
