--- title: "APIM Policy Patterns for AI Governance: Part 1 – Rate Limits, Token Quotas & Observability" url: https://daily.dev/posts/apim-policy-patterns-for-ai-governance-part-1-rate-limits-token-quotas-observability-3kredwnuo source_url: https://thomasthornton.cloud/apim-policy-patterns-for-ai-governance-part-1-rate-limits-token-quotas-observability type: article source: "Thomas Thornton" published: 2026-08-24T12:20:30.532Z updated: 2026-08-24T12:20:54.665Z tags: ["azure", "observability", "ai-governance", "azure-openai"] reading_time: 11 upvotes: 1 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # APIM Policy Patterns for AI Governance: Part 1 – Rate Limits, Token Quotas & Observability **[Thomas Thornton](https://daily.dev/sources/thomasthornton)** · 11 min read · 1 upvotes · 0 comments ## Summary A practitioner's approach to enforcing runtime governance for shared AI/LLM access through Azure API Management, covering how request-rate limits and token quotas solve different problems, using rate-limit-by-key tied to trusted consumer identities, the llm-token-limit policy for token-per-minute and monthly quotas (429 vs 403 responses), why the built-in APIM cache should not be treated as a financial ledger, emitting token telemetry via llm-emit-token-metric and request-level tracing, and handling backend throttling with retry-based primary/fallback routing. Includes a checklist of edge cases to test before rolling a policy into production, such as per-gateway (not global) counters, cardinality limits on custom metric dimensions, and buffering retried POST bodies. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the difference between rate-limit-by-key and llm-token-limit in Azure API Management? Rate-limit-by-key throttles based on the number of HTTP requests within a time window, rejecting calls before they reach the backend, while llm-token-limit constrains actual model token consumption, enforcing both a short-term tokens-per-minute rate and a longer token-quota over hourly to yearly periods. Exceeding tokens-per-minute returns 429, while exhausting token-quota returns 403, so clients need to treat them as distinct failure conditions. _Teams building AI gateways can compare governance patterns like this on daily.dev before designing their own policies._ ### Why shouldn't I use the APIM built-in cache to track AI spending budgets? Microsoft describes the built-in APIM cache as volatile and best-effort, and in classic tiers it is cleared progressively during service updates, making it unreliable as an authoritative financial ledger. Token quotas via llm-token-limit are also imprecise because APIM can't know final completion size until concurrent requests are processed, so usage data should instead flow into a proper FinOps or reporting pipeline for chargeback. _Anyone weighing cost-control options for shared AI platforms can dig into these tradeoffs on daily.dev._ ### Are rate-limit-by-key and llm-token-limit supported on the Azure APIM Consumption tier? No, both policies are unsupported on the Consumption tier. They work on Developer, Basic, Basic v2, Standard, Standard v2, Premium and Premium v2 tiers, so choosing Consumption for a shared AI gateway rules out these runtime rate and token controls entirely. _Developers picking an APIM tier for AI workloads can track policy support details like this on daily.dev._ ## Similar posts on daily.dev - [From Azure Policy to APIM: Implementing Azure AI Guardrails](https://daily.dev/posts/from-azure-policy-to-apim-implementing-azure-ai-guardrails-jsddgfd4m) · Thomas Thornton · 0 upvotes · 0 comments - [Rate limiting for LLM applications: Why it matters and how to implement it](https://daily.dev/posts/rate-limiting-for-llm-applications-why-it-matters-and-how-to-implement-it-xodmsr6nv) · portkey · 0 upvotes · 0 comments - [Manage AI resource use with TokenRateLimitPolicy](https://daily.dev/posts/manage-ai-resource-use-with-tokenratelimitpolicy-j8dnqhfgj) · Red Hat Developer · 1 upvotes · 0 comments --- Tags: [#azure](https://daily.dev/tags/azure), [#observability](https://daily.dev/tags/observability), [#ai-governance](https://daily.dev/tags/ai-governance), [#azure-openai](https://daily.dev/tags/azure-openai) [View this post on daily.dev](https://daily.dev/posts/apim-policy-patterns-for-ai-governance-part-1-rate-limits-token-quotas-observability-3kredwnuo)