Spotify faced fragmented LLM integrations across hundreds of internal teams, each solving the same authentication, rate limiting, cost tracking, and failover problems independently. To unify this, they deployed Kong's AI Gateway in May 2025 on GKE in hybrid mode, with a control plane backed by PostgreSQL and stateless data plane proxies. The gateway uses a plugin chain covering multi-provider routing (OpenAI, Vertex AI, first-party Hendrix models), Prometheus-based token-level telemetry, OpenTelemetry distributed tracing, and AI content guardrails. Over 1,000 internal services have onboarded, reducing new team integration from weeks to an afternoon config change. Future plans include A/B model testing via Spotify's Confidence platform and expanded support for agentic workflows.
Table of contents
The Challenge: AI Adoption Across SpotifyWhy Kong: Building on Proven API InfrastructureArchitecture: A Gateway Purpose-Built for AI TrafficThe Plugin Chain: Infrastructure for Unified AI GovernanceImpact: What ChangedWhat's NextConclusion4.6K Impressions1 Comment