--- title: Why We're Building a Self-Hosted LLM Gateway | daily.dev description: Valymux is a self-hosted LLM gateway written in Rust that provides a single API endpoint for multiple AI providers (OpenAI, Anthropic, Google Gemini). It... canonical: https://daily.dev/posts/why-we-re-building-a-self-hosted-llm-gateway-c5qkasuvm twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why We're Building a Self-Hosted LLM Gateway | daily.dev og:description: Valymux is a self-hosted LLM gateway written in Rust that provides a single API endpoint for multiple AI providers (OpenAI, Anthropic, Google Gemini). It... og:url: https://daily.dev/posts/why-we-re-building-a-self-hosted-llm-gateway-c5qkasuvm og:image: https://api.daily.dev/og/posts/C5qKAsUVm.png og:image:alt: Why We're Building a Self-Hosted LLM Gateway og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why We're Building a Self-Hosted LLM Gateway **[Lay Sheth aka CLoaKY](https://daily.dev/sources/c4osc1it7lkxz2zazfkyz)** · [@cloaky233](https://daily.dev/cloaky233) · 5 min read · 1 upvotes · 0 comments ## Summary Valymux is a self-hosted LLM gateway written in Rust that provides a single API endpoint for multiple AI providers (OpenAI, Anthropic, Google Gemini). It addresses pain points like scattered API keys, inconsistent SDKs, and credential sharing by offering virtual keys with scoped access, capability-aware request validation, and AES-256-GCM credential encryption. Built on Tokio and Axum, it targets predictable low-latency proxying without GC pauses. The project is pre-MVP, open source under AGPL-3.0, and targeting an end-of-April 2026 launch. It positions itself against LiteLLM (Python-based), Portkey (SaaS-first), and Bifrost (enterprise-focused) as a self-hostable, developer-friendly alternative. ## Content We're building [Valymux](https://www.valymux.xyz/) — a self-hosted LLM gateway that gives you one API endpoint for multiple AI providers. It's open source, written in Rust, and we're targeting an end-of-April launch. This post explains why we started building it and what problem we're trying to solve. --- ## The Problem We Kept Running Into If you're building with multiple AI providers, you've probably dealt with some version of this: **API keys everywhere.** OpenAI key in one `.env` file, Anthropic in another, Google in a third. Some shared over Slack when a teammate needed access. Some copy-pasted into CI secrets and forgotten. No central place to see what keys exist or who has access to what. **Different SDKs, different formats.** OpenAI and Anthropic handle system messages differently. Streaming events have different shapes. Error responses don't match. Switching providers means rewriting integration code, not just changing an API key. **Sharing credentials is awkward.** When someone on the team needs to test against Claude, do you share the production key? Create a new one? There's no clean way to give scoped access without exposing the underlying credential. None of these problems are catastrophic on their own. But they add up. And as we started using more providers — OpenAI for some things, Anthropic for others, Gemini for cost-sensitive workloads — the friction kept growing. --- ## What Existing Tools Do We looked at what's already out there. **LiteLLM** is the most popular option. 40K+ GitHub stars, supports 100+ providers, active community. It's a solid tool. But it's Python-based, which means you're adding a runtime dependency and dealing with GC behavior if latency matters. The March 2026 supply chain incident also made us think harder about what it means to route all your API credentials through a third-party package. **Portkey** is more polished, with good observability and enterprise features. But it's SaaS-first. Your credentials live on their infrastructure. For some teams that's fine; for us, it wasn't. **Bifrost** is fast — they claim microsecond overhead. But it's focused on raw performance and enterprise governance, not developer experience for small teams. There's clearly a market here. These tools exist because the problem is real. But we wanted something different: self-hosted by default, simple to deploy, and something we could audit ourselves. --- ## What We're Building Valymux is a gateway that sits between your application and your AI providers. **One endpoint, multiple providers.** You configure your provider credentials once. Your application hits a single endpoint (`/v1/chat/completions`). Valymux routes to the right provider and translates the response to a consistent format. Switching from GPT-4 to Claude means changing a model string, not rewriting code. **Virtual keys for access control.** Instead of sharing your real API key, you create a "virtual key" that maps to it. You can scope virtual keys to specific models, set expiration dates, and revoke them without touching the underlying credential. The real key stays encrypted inside Valymux. **Capability-aware validation.** Different models have different constraints — temperature ranges, streaming support, tool compatibility. Valymux validates requests against model capabilities before sending them upstream. You get a clear error instead of a confusing failure from the provider. **Self-hosted and auditable.** It runs on your infrastructure. Credentials are encrypted at rest (AES-256-GCM). The code is open source (AGPL-3.0). You don't have to trust us — you can read it. --- ## Why Rust We went with Rust for a few reasons: - **Predictable latency.** No garbage collector means no GC pauses. When you're proxying requests, consistent performance matters more than peak performance. - **Single binary.** One Docker image, no runtime dependencies to manage. - **Memory safety.** The gateway handles credentials. We wanted the compiler catching memory bugs, not production. This isn't a "Rust is better than Python" argument. LiteLLM works well for a lot of teams. We just had different constraints. --- ## Where We Are Now We're pre-MVP. Here's the current status: **Done:** - Core gateway architecture (Tokio + Axum) - Credential encryption (AES-256-GCM) - Virtual key system with scoped access - Google Gemini adapter (streaming and non-streaming) - Request logging with token usage **In progress:** - OpenAI adapter (should be straightforward — it's the native format) - Anthropic adapter (requires message format translation) - Frontend dashboard and configuration UI **Target:** End of April 2026 --- ## What We'd Like From You We're building this for developers who have the same problems we do. If that's you, we'd genuinely like your input. **What's your current setup?** What works? What's frustrating? What would actually make you consider switching tools? → [Share your experience](https://www.valymux.xyz/feedback) **Want early access when we launch?** → [Join the waitlist](https://www.valymux.xyz/waitlist) **Want to follow along or contribute?** → [GitHub](https://github.com/cloaky233/Valymux) --- ## What's Next Over the next few weeks, we'll be finishing the OpenAI and Anthropic adapters, building out the frontend, and getting the Docker Compose deployment polished. We're building in public, so if you're interested in following along or have questions, the GitHub repo is the best place to start. --- *Valymux is open source (AGPL-3.0) and self-hostable. Website: [valymux.xyz](https://www.valymux.xyz)* --- Tags: [#rust](https://daily.dev/tags/rust) [View this post on daily.dev](https://daily.dev/posts/why-we-re-building-a-self-hosted-llm-gateway-c5qkasuvm) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/why-we-re-building-a-self-hosted-llm-gateway-c5qkasuvm","headline":"Why We're Building a Self-Hosted LLM Gateway","text":"Valymux is a self-hosted LLM gateway written in Rust that provides a single API endpoint for multiple AI providers (OpenAI, Anthropic, Google Gemini). It addresses pain points like scattered API keys, inconsistent SDKs, and credential sharing by offering virtual keys with scoped access, capability-aware request validation, and AES-256-GCM credential encryption. Built on Tokio and Axum, it targets predictable low-latency proxying without GC pauses. The project is pre-MVP, open source under AGPL-3.0, and targeting an end-of-April 2026 launch. It positions itself against LiteLLM (Python-based), Portkey (SaaS-first), and Bifrost (enterprise-focused) as a self-hostable, developer-friendly alternative.","url":"https://daily.dev/posts/why-we-re-building-a-self-hosted-llm-gateway-c5qkasuvm","datePublished":"2026-03-29T17:58:10.583Z","dateModified":"2026-03-29T18:04:17.293Z","author":{"@type":"Person","name":"Lay Sheth aka CLoaKY","url":"https://daily.dev/cloaky233","image":"https://avatars.githubusercontent.com/u/147647691?v=4","description":"Meet a Rust and Python aficionado, a final year CS student.","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":90}},"image":"https://media.daily.dev/image/upload/s--DRIeA0MX--/f_auto/v1774807095/posts/C5qKAsUVm?_a=BAMAMiWQ0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/c4osc1it7lkxz2zazfkyz","name":"Lay Sheth aka CLoaKY"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Lay Sheth aka CLoaKY","item":"https://daily.dev/sources/c4osc1it7lkxz2zazfkyz"},{"@type":"ListItem","position":3,"name":"Why We're Building a Self-Hosted LLM Gateway"}]} ```