A practitioner's approach to enforcing runtime governance for shared AI/LLM access through Azure API Management, covering how request-rate limits and token quotas solve different problems, using rate-limit-by-key tied to trusted consumer identities, the llm-token-limit policy for token-per-minute and monthly quotas (429 vs 403 responses), why the built-in APIM cache should not be treated as a financial ledger, emitting token telemetry via llm-emit-token-metric and request-level tracing, and handling backend throttling with retry-based primary/fallback routing. Includes a checklist of edge cases to test before rolling a policy into production, such as per-gateway (not global) counters, cardinality limits on custom metric dimensions, and buffering retried POST bodies.
Table of contents
Request rate and token consumption are different controlsRate limit against a trusted consumerUse llm-token-limit rather than building your own counterI would not make the APIM cache a billing ledgerEmit useful token telemetry before choosing quotasBackend throttling needs a separate planThe details I would test before sharing the policyI would start with telemetry, then enforce limitsQuestions this post answers
What is the difference between rate-limit-by-key and llm-token-limit in Azure API Management?
Rate-limit-by-key throttles based on the number of HTTP requests within a time window, rejecting calls before they reach the backend, while llm-token-limit constrains actual model token consumption, enforcing both a short-term tokens-per-minute rate and a longer token-quota over hourly to yearly periods. Exceeding tokens-per-minute returns 429, while exhausting token-quota returns 403, so clients need to treat them as distinct failure conditions. Teams building AI gateways can compare governance patterns like this on daily.dev before designing their own policies.
Why shouldn't I use the APIM built-in cache to track AI spending budgets?
Microsoft describes the built-in APIM cache as volatile and best-effort, and in classic tiers it is cleared progressively during service updates, making it unreliable as an authoritative financial ledger. Token quotas via llm-token-limit are also imprecise because APIM can't know final completion size until concurrent requests are processed, so usage data should instead flow into a proper FinOps or reporting pipeline for chargeback. Anyone weighing cost-control options for shared AI platforms can dig into these tradeoffs on daily.dev.
Are rate-limit-by-key and llm-token-limit supported on the Azure APIM Consumption tier?
No, both policies are unsupported on the Consumption tier. They work on Developer, Basic, Basic v2, Standard, Standard v2, Premium and Premium v2 tiers, so choosing Consumption for a shared AI gateway rules out these runtime rate and token controls entirely. Developers picking an APIM tier for AI workloads can track policy support details like this on daily.dev.