--- title: How Spotify Deployed Kong's AI Gateway to Power... description: Spotify faced fragmented LLM integrations across hundreds of internal teams, each solving the same authentication, rate limiting, cost tracking, and failover... canonical: https://daily.dev/posts/how-spotify-deployed-kong-s-ai-gateway-to-power-generative-ai-at-scale-faonlx9mr twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How Spotify Deployed Kong's AI Gateway to Power Generative AI at Scale | daily.dev og:description: Spotify faced fragmented LLM integrations across hundreds of internal teams, each solving the same authentication, rate limiting, cost tracking, and failover... og:url: https://daily.dev/posts/how-spotify-deployed-kong-s-ai-gateway-to-power-generative-ai-at-scale-faonlx9mr og:image: https://api.daily.dev/og/posts/FaonlX9mr.png og:image:alt: How Spotify Deployed Kong's AI Gateway to Power Generative AI at Scale og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How Spotify Deployed Kong's AI Gateway to Power Generative AI at Scale **[Spotify Labs](https://daily.dev/sources/spotify)** · 9 min read · 5 upvotes · 1 comments ## Summary Spotify faced fragmented LLM integrations across hundreds of internal teams, each solving the same authentication, rate limiting, cost tracking, and failover problems independently. To unify this, they deployed Kong's AI Gateway in May 2025 on GKE in hybrid mode, with a control plane backed by PostgreSQL and stateless data plane proxies. The gateway uses a plugin chain covering multi-provider routing (OpenAI, Vertex AI, first-party Hendrix models), Prometheus-based token-level telemetry, OpenTelemetry distributed tracing, and AI content guardrails. Over 1,000 internal services have onboarded, reducing new team integration from weeks to an afternoon config change. Future plans include A/B model testing via Spotify's Confidence platform and expanded support for agentic workflows. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does Spotify route requests to multiple LLM providers like OpenAI, Vertex AI, and its own Hendrix models through one API? Spotify uses Kong's ai-proxy-advanced plugin as a single API format on the client side, with provider-specific authentication and request/response transformation happening on the backend. Teams integrate once and the gateway transparently routes to Hendrix, OpenAI, or Vertex AI, letting teams switch providers without changing their code. _Teams weighing a unified LLM routing layer can follow real-world gateway architectures like this on daily.dev._ ### What performance overhead does Kong's AI Gateway add to LLM requests? Kong added roughly 40ms of routing overhead in Spotify's benchmarks and was expected to scale beyond 1,000 requests per second per instance, which was more than sufficient for its traffic patterns. This performance, combined with Kong being fully self-hosted so data stays within company infrastructure, drove the decision to build on Kong rather than a custom proxy. _Engineers evaluating gateway latency trade-offs can track benchmarks like these on daily.dev._ ### Why did Spotify move from Kong OSS to Kong Enterprise for its AI Gateway? Spotify started with Kong OSS but moved to Kong Enterprise because business-critical LLM traffic required enterprise-grade security and support that weren't available in the open-source version. The switch preserved the same hybrid control-plane/data-plane architecture and GitOps workflow while adding features needed to run generative AI infrastructure at scale. _Teams weighing open-source versus enterprise infrastructure for AI workloads can compare real decisions like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@pdfopsdev** · 0 upvotes > A gateway hop costing ~40ms is a dealbreaker for normal API traffic but rounding error against multi-second LLM completions — that latency asymmetry is why 'put everything behind the gateway' actually works for AI traffic when it never quite did for microservices. The per-consumer token attribution is the sleeper feature here; most orgs still can't answer 'which team spent what on LLMs last month'. ## Similar posts on daily.dev - [KubeCon NA 2025 - Erica Hughberg and Alexa Griffith on Tools for the Age of GenAI](https://daily.dev/posts/kubecon-na-2025---erica-hughberg-and-alexa-griffith-on-tools-for-the-age-of-genai-pavyctafs) · InfoQ · 0 upvotes · 0 comments --- Tags: [#observability](https://daily.dev/tags/observability), [#api-gateway](https://daily.dev/tags/api-gateway), [#ai-gateway](https://daily.dev/tags/ai-gateway) [View this post on daily.dev](https://daily.dev/posts/how-spotify-deployed-kong-s-ai-gateway-to-power-generative-ai-at-scale-faonlx9mr) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How Spotify Deployed Kong's AI Gateway to Power Generative AI at Scale","url":"https://daily.dev/posts/how-spotify-deployed-kong-s-ai-gateway-to-power-generative-ai-at-scale-faonlx9mr","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-spotify-deployed-kong-s-ai-gateway-to-power-generative-ai-at-scale-faonlx9mr"},"datePublished":"2026-07-14T17:04:54.240Z","dateModified":"2026-09-13T20:53:21.457Z","description":"Spotify faced fragmented LLM integrations across hundreds of internal teams, each solving the same authentication, rate limiting, cost tracking, and failover...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1a7b61a277a87c204a11503f432d2ffc?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1a7b61a277a87c204a11503f432d2ffc?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Spotify Labs","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Spotify Labs","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/spotify","url":"https://daily.dev/sources/spotify"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/how-spotify-deployed-kong-s-ai-gateway-to-power-generative-ai-at-scale-faonlx9mr","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":5},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"observability,api-gateway,ai-gateway","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Spotify Labs","item":"https://daily.dev/sources/spotify"},{"@type":"ListItem","position":3,"name":"How Spotify Deployed Kong's AI Gateway to Power Generative AI at Scale"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/how-spotify-deployed-kong-s-ai-gateway-to-power-generative-ai-at-scale-faonlx9mr","comment":[{"@type":"Comment","text":"A gateway hop costing ~40ms is a dealbreaker for normal API traffic but rounding error against multi-second LLM completions — that latency asymmetry is why ‘put everything behind the gateway’ actually works for AI traffic when it never quite did for microservices. The per-consumer token attribution is the sleeper feature here; most orgs still can’t answer ‘which team spent what on LLMs last month’.","datePublished":"2026-07-14T18:05:15.875Z","url":"https://daily.dev/posts/FaonlX9mr#c-el78YxsC1","author":{"@type":"Person","name":"PDFops","url":"https://daily.dev/pdfopsdev","image":"https://media.daily.dev/image/upload/s---8isRBKc--/f_auto/v1782922291/avatars/avatar_orjMeK8QKaaVZwGq7ScPz?_a=BAMAMicg0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-spotify-deployed-kong-s-ai-gateway-to-power-generative-ai-at-scale-faonlx9mr#faq","mainEntity":[{"@type":"Question","name":"How does Spotify route requests to multiple LLM providers like OpenAI, Vertex AI, and its own Hendrix models through one API?","acceptedAnswer":{"@type":"Answer","text":"Spotify uses Kong's ai-proxy-advanced plugin as a single API format on the client side, with provider-specific authentication and request/response transformation happening on the backend. Teams integrate once and the gateway transparently routes to Hendrix, OpenAI, or Vertex AI, letting teams switch providers without changing their code. Teams weighing a unified LLM routing layer can follow real-world gateway architectures like this on daily.dev."}},{"@type":"Question","name":"What performance overhead does Kong's AI Gateway add to LLM requests?","acceptedAnswer":{"@type":"Answer","text":"Kong added roughly 40ms of routing overhead in Spotify's benchmarks and was expected to scale beyond 1,000 requests per second per instance, which was more than sufficient for its traffic patterns. This performance, combined with Kong being fully self-hosted so data stays within company infrastructure, drove the decision to build on Kong rather than a custom proxy. Engineers evaluating gateway latency trade-offs can track benchmarks like these on daily.dev."}},{"@type":"Question","name":"Why did Spotify move from Kong OSS to Kong Enterprise for its AI Gateway?","acceptedAnswer":{"@type":"Answer","text":"Spotify started with Kong OSS but moved to Kong Enterprise because business-critical LLM traffic required enterprise-grade security and support that weren't available in the open-source version. The switch preserved the same hybrid control-plane/data-plane architecture and GitOps workflow while adding features needed to run generative AI infrastructure at scale. Teams weighing open-source versus enterprise infrastructure for AI workloads can compare real decisions like this on daily.dev."}}]} ```