--- title: I thought an LLM gateway was unnecessary. Then my... description: As AI applications grow beyond a single model call, complexity accumulates around provider management — fallbacks, retries, rate limits, cost tracking, and... canonical: https://daily.dev/posts/i-thought-an-llm-gateway-was-unnecessary-then-my-application-needed-a-second-model-provider--g8yxjiacr twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: I thought an LLM gateway was unnecessary. Then my application needed a second model provider! | daily.dev og:description: As AI applications grow beyond a single model call, complexity accumulates around provider management — fallbacks, retries, rate limits, cost tracking, and... og:url: https://daily.dev/posts/i-thought-an-llm-gateway-was-unnecessary-then-my-application-needed-a-second-model-provider--g8yxjiacr og:image: https://api.daily.dev/og/posts/G8YxjIAcr.png og:image:alt: I thought an LLM gateway was unnecessary. Then my application needed a second model provider! og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # I thought an LLM gateway was unnecessary. Then my application needed a second model provider! **[Swapnoneel Saha](https://daily.dev/sources/ldi7vqbpkryv936wnukjd)** · [@swapnoneel](https://daily.dev/swapnoneel) · 3 min read · 1 upvotes · 0 comments ## Summary As AI applications grow beyond a single model call, complexity accumulates around provider management — fallbacks, retries, rate limits, cost tracking, and observability. Bifrost is an open-source AI gateway written in Go that sits between applications and model providers, handling this infrastructure in one place. It uses an OpenAI-compatible API supporting 20+ providers, making migration low-friction. The project reports under 100 microseconds of gateway overhead at 5,000 requests per second, treating latency as a primary engineering concern rather than an afterthought. ## Content The first version of an AI application is usually simple. You get an API key. You call a model. You get a response. For some time, this works very well. Then the application becomes important. Now you want to use Claude for one task and an OpenAI model for another. You add another provider because you do not want one outage to stop your application. Then you need retries. Then you need fallbacks. Then you need rate limits. Then someone asks how much each team spends. Then you need logs because a request failed yesterday and nobody knows why. At some point, the difficult part is no longer calling the model. The difficult part is everything around the model. That was the part of AI infrastructure that made me interested in Bifrost. Bifrost is an open-source AI gateway. It sits between your application and model providers. Instead of making every application understand every provider, the application talks to one gateway. The gateway does the provider-specific work. This sounds like a small abstraction. I think its value becomes clear only when the system grows. Consider a simple case. Your application uses Provider A. Provider A has a problem. Without a gateway, your application must detect the error, select Provider B, convert the request if necessary, retry it, and record what happened. You can write this code. The problem is that you must write it again for each application. The gateway changes where this complexity lives. Your application sends one request. [Bifrost ](https://github.com/maximhq/bifrost)can route that request to different providers and models. It can also apply fallback rules, load balancing, caching, budgets, and observability in one place. There was another detail that I found interesting. Bifrost uses an OpenAI-compatible API and supports more than 20 model providers. This means the abstraction is not: “Here is another SDK that you must adopt.” It is closer to: “Keep your existing API format and change where the request goes.” For many existing applications, that is a much smaller migration. The project is mostly written in Go. This also makes sense for this type of system. A gateway is on the request path. Each extra operation adds latency. A gateway can have many features, but it becomes less useful if the gateway itself becomes the bottleneck. The Bifrost team publishes benchmarks for this reason. The project reports less than 100 microseconds of gateway overhead at 5,000 requests per second in its benchmark environment. I would not treat one benchmark as a universal result. Hardware, request patterns, providers, network conditions, and configuration all matter. But I like that gateway performance is treated as a primary engineering problem instead of an afterthought. The larger lesson for me was not specifically about Bifrost. It was about where complexity moves as AI applications become real systems. At first, model access is an API problem. Later, it becomes an infrastructure problem. You start to care about: provider failures, cost, routing, keys, rate limits, logs, caching, security, and which models each application can use. You can put all of this logic inside every application. Or you can move it into a common layer. That is the idea behind Bifrost. I am curious how other teams handle this. Do you put provider routing and fallback logic inside each application? Do you use a gateway? Or is the gateway another layer that you prefer not to operate? ## Similar posts on daily.dev - [Top 5 LLM Gateways in 2026: A Deep-Dive Comparison for Production Teams](https://daily.dev/posts/top-5-llm-gateways-in-2026-a-deep-dive-comparison-for-production-teams-izylo6on6) · DEV · 3 upvotes · 0 comments - [AI gateways: why and how](https://daily.dev/posts/ai-gateways-why-and-how-j91suhnef) · A Java geek · 1 upvotes · 0 comments --- Tags: [#golang](https://daily.dev/tags/golang), [#observability](https://daily.dev/tags/observability), [#ai-infrastructure](https://daily.dev/tags/ai-infrastructure) [View this post on daily.dev](https://daily.dev/posts/i-thought-an-llm-gateway-was-unnecessary-then-my-application-needed-a-second-model-provider--g8yxjiacr) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/i-thought-an-llm-gateway-was-unnecessary-then-my-application-needed-a-second-model-provider--g8yxjiacr","headline":"I thought an LLM gateway was unnecessary. Then my application needed a second model provider!","text":"As AI applications grow beyond a single model call, complexity accumulates around provider management — fallbacks, retries, rate limits, cost tracking, and observability. Bifrost is an open-source AI gateway written in Go that sits between applications and model providers, handling this infrastructure in one place. It uses an OpenAI-compatible API supporting 20+ providers, making migration low-friction. The project reports under 100 microseconds of gateway overhead at 5,000 requests per second, treating latency as a primary engineering concern rather than an afterthought.","url":"https://daily.dev/posts/i-thought-an-llm-gateway-was-unnecessary-then-my-application-needed-a-second-model-provider--g8yxjiacr","datePublished":"2026-08-10T18:24:37.732Z","dateModified":"2026-09-13T21:03:59.130Z","author":{"@type":"Person","name":"Swapnoneel Saha","url":"https://daily.dev/swapnoneel","image":"https://media.daily.dev/image/upload/s--7WuNOXsT--/f_auto/v1783760285/avatars/avatar_Ldi7VQbPKrYV936wNUKJD?_a=BAMAMicg0","description":"Technical Growth Operator","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":2450}},"interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/ldi7vqbpkryv936wnukjd","name":"Swapnoneel Saha"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Swapnoneel Saha","item":"https://daily.dev/sources/ldi7vqbpkryv936wnukjd"},{"@type":"ListItem","position":3,"name":"I thought an LLM gateway was unnecessary. Then my application needed a second model provider!"}]} ```