A walkthrough of Kubernetes resources created when deploying LLM inference workloads using Red Hat AI Inference on Amazon EKS. Covers two deployment scenarios: a basic LLMInferenceService without routing and one with the llm-d router block enabled. Uses kubectl tree to trace all generated resources including Deployments, ReplicaSets, Services, EndpointSlices, InferencePool, HTTPRoute, and DestinationRules. Explains how the scheduler, route, and gateway subfields in the router configuration map to specific Kubernetes objects, and how intelligent routing based on queue depth and cache locality differs from simple round-robin load balancing.
Table of contents
Deploying an inference serviceServing the modelAdding llm-dUp next: Deep dive into endpoint picking and traffic flowLearn more52 Impressions