Amazon SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD), an inference optimization that separates the prefill and decode phases of LLM inference onto dedicated GPU pools. KV cache is transferred between pools over EFA using GPU-Direct RDMA. This eliminates resource contention where long-context requests stall token generation for concurrent requests. DPD enables independent scaling of prefill and decode capacity, delivers more consistent per-token latency, and higher goodput at strict latency SLOs. An intelligent router automatically directs long-context requests through the disaggregated path while sending shorter prompts directly to the decoder. It is enabled via a `pdSpec` section in the existing `InferenceEndpointConfig` custom resource and is composable with existing KV cache offloading and routing features. Available on EKS-orchestrated HyperPod clusters on EFA-capable instances across all supported AWS regions.
