A deep-dive into the architectural foundations of distributed LLM inference, covering the prefill/decode phase split and its impact on TTFT and TPOT KPIs, plus a detailed breakdown of the five parallelism dimensions: tensor (TP), pipeline (PP), expert (EP), data (DP), and context parallelism (PCP/DCP). Uses Alibaba's Qwen3.5 and Qwen3.6 model families as concrete examples across dense and MoE architectures. Includes hardware sizing guidance, configuration tables, and decision heuristics for when to apply each parallelism strategy. Also notes recent ecosystem developments including llm-d's CNCF Sandbox acceptance, KV cache innovation projects (NIXL, LMCache, Mooncake), and EAGLE 3.1 speculative decoding improvements.
59 Impressions