An Azure AI Landing Zone should give teams a pre-approved, fast path for building AI workloads while keeping centralized controls where they genuinely matter. The architecture separates concerns across three layers: Azure Policy governs resources (regions, tagging, model deployment approval), Azure API Management governs runtime consumption (token limits, quotas, routing, telemetry via policies like llm-token-limit and llm-emit-token-metric), and workload teams own application logic, prompts, and evaluations. Guidance covers cost control through environment-specific defaults, model selection profiles, quota ownership, telemetry collection without wholesale prompt logging, and a phased rollout starting with one complete governed path before adding caching or failover.
Table of contents
An AI Landing Zone should remove repeated platform decisionsKeep platform and workload ownership separateAPIM should govern the runtime contractAzure Policy and APIM policies solve different problemsAI cost control belongs in the architectureUse different defaults in each environmentGive teams a sensible model starting pointQuotas need something identifiable behind themCollect useful telemetry without collecting everythingThe platform should be able to attribute AI usage to enough context that someone can act on it.Do not turn APIM into the applicationStart with one complete governed pathWhere Azure AI Landing Zones tend to go wrongQuestions this post answers
What is the difference between Azure Policy and Azure API Management for governing AI workloads?
Azure Policy governs Azure resources themselves, such as approved regions, allowed model deployments, required tags, and public network access, and enforces this before or at deployment time. Azure API Management governs runtime consumption once traffic is flowing, controlling things like token allowances, rate limits, backend routing, and per-consumer quotas. Azure Policy cannot stop an application from flooding a model endpoint with requests; only APIM policies can. Architects weighing Azure Policy against APIM for AI governance can track this kind of platform guidance on daily.dev.
How can I enforce token rate limits for an Azure OpenAI API exposed through Azure API Management?
Use the llm-token-limit policy in Azure API Management, which enforces token rate limits or quotas by a calculated key such as consumer identity. Pair it with llm-emit-token-metric to send token consumption metrics and custom dimensions to Application Insights, and llm-content-safety to check prompts and responses through Azure AI Content Safety. Policy availability varies by APIM tier, so check the support matrix before choosing a tier. Teams building an AI gateway on APIM can follow platform architecture patterns like this via daily.dev.
Should every team share one Azure OpenAI or Microsoft Foundry resource, or should each team have its own?
There is no blanket rule; shared resources work well when teams have similar data boundaries, regional requirements, capacity needs, and operational expectations. Workload-owned resources become preferable once teams need independent scaling, separate deployment lifecycles, or stronger isolation. A landing zone should define governance for both patterns rather than forcing every team through one central AI resource. Platform teams deciding between shared versus dedicated AI resources can weigh trade-offs like these on daily.dev.
3.2K Impressions1 Comment