--- title: Microsoft Three-Layer LLM Routing Architecture for AI... description: Microsoft has published a reference architecture for routing LLM traffic from AI agents on Azure Kubernetes Service (AKS), decomposing the problem into three... canonical: https://daily.dev/posts/microsoft-three-layer-llm-routing-architecture-for-ai-agents-on-aks-67wgdmok9 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Microsoft Three-Layer LLM Routing Architecture for AI Agents on AKS | daily.dev og:description: Microsoft has published a reference architecture for routing LLM traffic from AI agents on Azure Kubernetes Service (AKS), decomposing the problem into three... og:url: https://daily.dev/posts/microsoft-three-layer-llm-routing-architecture-for-ai-agents-on-aks-67wgdmok9 og:image: https://api.daily.dev/og/posts/67WgdmOK9.png og:image:alt: Microsoft Three-Layer LLM Routing Architecture for AI Agents on AKS og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Microsoft Three-Layer LLM Routing Architecture for AI Agents on AKS **[InfoQ](https://daily.dev/sources/infoq)** · 4 min read · 1 upvotes · 0 comments ## Summary Microsoft has published a reference architecture for routing LLM traffic from AI agents on Azure Kubernetes Service (AKS), decomposing the problem into three layers: semantic model selection (RouteLLM), policy and governance (agentgateway), and GPU-aware replica placement (Kubernetes Gateway API Inference Extension). The design targets agentic workloads where a single task can trigger hundreds of LLM calls, most of which don't require a frontier model. RouteLLM uses a matrix-factorization router to predict when a cheaper model suffices, achieving ~95% of GPT-4's quality while routing only ~26% of calls to the strong model, cutting costs by up to 85%. KAITO manages GPU node pools and runs vLLM, exposing KV-cache and queue metrics for intelligent replica selection. The architecture is modular — teams can adopt individual layers incrementally — but key components are still maturing, with CRD fields and API names changing between releases. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is Microsoft's reference architecture for routing LLM agent traffic on Azure Kubernetes Service? It combines three layers: RouteLLM for semantic routing between a strong and weak model, agentgateway as an AI proxy for authentication, rate limits and cost tracking, and the Kubernetes Gateway API Inference Extension's Endpoint Picker for GPU-aware placement based on vLLM's KV-cache occupancy and queue depth. KAITO provisions GPU node pools and runs vLLM, with Azure OpenAI serving the strong-model path. All three connect through one OpenAI-compatible endpoint. _daily.dev surfaces practical breakdowns like this for engineers designing agent routing on Kubernetes._ ### How much cost savings did RouteLLM achieve by routing only some calls to GPT-4 instead of all of them? RouteLLM's matrix-factorization router achieved about 95% of GPT-4's MT-Bench quality while sending only around 26% of calls to GPT-4, saving up to 85% in costs compared to routing every call to the strong model. Microsoft cautions this figure is tied to the specific model pair RouteLLM was trained on and doesn't transfer automatically to other pairings like phi-4-mini and GPT-5.1. _Teams weighing model routing tradeoffs can find grounded benchmarks like this through daily.dev._ ### Why can't the Gateway API Inference Extension's Endpoint Picker be replaced with a fully managed Azure service? Because GPU-aware placement requires reading live per-pod metrics like vLLM's KV-cache occupancy and queue depth, which only exist inside the cluster where the GPUs run. Microsoft's Foundry model router offers a managed version of RouteLLM's semantic routing layer, but there is no managed equivalent for the Endpoint Picker, so it must run in-cluster regardless of which gateway is used. _daily.dev helps engineers track which parts of an infra stack stay self-managed versus go managed._ ## Similar posts on daily.dev - [How to route external and local LLMs with Models-as-a-Service](https://daily.dev/posts/how-to-route-external-and-local-llms-with-models-as-a-service-ufwzx55fl) · Red Hat Developer · 0 upvotes · 0 comments - [LLM Router Pattern \(2026\): Model Routing, Fallbacks, Cost Control](https://daily.dev/posts/llm-router-pattern-2026-model-routing-fallbacks-cost-control-8txalf5nf) · Alex CloudStar · 1 upvotes · 0 comments - [Monitor LLM routing with the Kubernetes Inference Extension](https://daily.dev/posts/monitor-llm-routing-with-the-kubernetes-inference-extension-2xwttjkou) · Datadog · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#ai-agents](https://daily.dev/tags/ai-agents), [#azure](https://daily.dev/tags/azure), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/microsoft-three-layer-llm-routing-architecture-for-ai-agents-on-aks-67wgdmok9) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Microsoft Three-Layer LLM Routing Architecture for AI Agents on AKS","url":"https://daily.dev/posts/microsoft-three-layer-llm-routing-architecture-for-ai-agents-on-aks-67wgdmok9","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/microsoft-three-layer-llm-routing-architecture-for-ai-agents-on-aks-67wgdmok9"},"datePublished":"2026-07-29T12:02:41.224Z","dateModified":"2026-09-14T06:03:41.367Z","description":"Microsoft has published a reference architecture for routing LLM traffic from AI agents on Azure Kubernetes Service (AKS), decomposing the problem into three...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/485ac713bcb7896d946259893443463c?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/485ac713bcb7896d946259893443463c?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"InfoQ","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"InfoQ","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/afc3bced3e1e4b188dd9127017a60e0c","url":"https://daily.dev/sources/infoq"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/microsoft-three-layer-llm-routing-architecture-for-ai-agents-on-aks-67wgdmok9","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,ai-agents,azure,vllm","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"InfoQ","item":"https://daily.dev/sources/infoq"},{"@type":"ListItem","position":3,"name":"Microsoft Three-Layer LLM Routing Architecture for AI Agents on AKS"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/microsoft-three-layer-llm-routing-architecture-for-ai-agents-on-aks-67wgdmok9#faq","mainEntity":[{"@type":"Question","name":"What is Microsoft's reference architecture for routing LLM agent traffic on Azure Kubernetes Service?","acceptedAnswer":{"@type":"Answer","text":"It combines three layers: RouteLLM for semantic routing between a strong and weak model, agentgateway as an AI proxy for authentication, rate limits and cost tracking, and the Kubernetes Gateway API Inference Extension's Endpoint Picker for GPU-aware placement based on vLLM's KV-cache occupancy and queue depth. KAITO provisions GPU node pools and runs vLLM, with Azure OpenAI serving the strong-model path. All three connect through one OpenAI-compatible endpoint. daily.dev surfaces practical breakdowns like this for engineers designing agent routing on Kubernetes."}},{"@type":"Question","name":"How much cost savings did RouteLLM achieve by routing only some calls to GPT-4 instead of all of them?","acceptedAnswer":{"@type":"Answer","text":"RouteLLM's matrix-factorization router achieved about 95% of GPT-4's MT-Bench quality while sending only around 26% of calls to GPT-4, saving up to 85% in costs compared to routing every call to the strong model. Microsoft cautions this figure is tied to the specific model pair RouteLLM was trained on and doesn't transfer automatically to other pairings like phi-4-mini and GPT-5.1. Teams weighing model routing tradeoffs can find grounded benchmarks like this through daily.dev."}},{"@type":"Question","name":"Why can't the Gateway API Inference Extension's Endpoint Picker be replaced with a fully managed Azure service?","acceptedAnswer":{"@type":"Answer","text":"Because GPU-aware placement requires reading live per-pod metrics like vLLM's KV-cache occupancy and queue depth, which only exist inside the cluster where the GPUs run. Microsoft's Foundry model router offers a managed version of RouteLLM's semantic routing layer, but there is no managed equivalent for the Endpoint Picker, so it must run in-cluster regardless of which gateway is used. daily.dev helps engineers track which parts of an infra stack stay self-managed versus go managed."}}]} ```