APIs, Inference Platforms & Gateways
Four rough tiers, each with different trade-offs on model access, speed, price, and control.
First-party model APIs
Buy directly from the lab: OpenAI, Anthropic, Google (Gemini), plus Mistral, Cohere, xAI, DeepSeek, and others. You get the newest models first, best feature support (tools, structured outputs, caching, batch), and the vendor's own reliability, at list price, with lock-in to that provider's SDK/quirks.
Aggregators / routers
OpenRouter puts one API and one bill in front of hundreds of models across dozens of providers, with automatic fallback and price/latency-based routing. It's best for trying many models without many contracts and for resilience via multi-provider fallback. The trade-off is a hop in the middle (a small latency/trust dependency), and some cutting-edge provider features may not pass through. Hugging Face Inference Providers is a similar OpenAI-compatible router across a dozen-plus backends and hundreds of models.
Fast-inference / open-model hosts
These serve open-weight models (Llama, Qwen, DeepSeek, Mistral, gpt-oss, etc.) with heavy optimization, often OpenAI-compatible, usually priced per-token:
| Host | Best at |
|---|---|
| Groq | Low-latency token streaming (LPU hardware, AI Hardware & Accelerators) |
| Cerebras | High inference speeds via wafer-scale (AI Hardware & Accelerators) |
| Together AI | Broad open-model catalog: serving, fine-tuning, and GPU clusters |
| Fireworks AI | Fast serving, function calling, and fine-tuning, with a production focus |
| Baseten | Deploying your own models (open Truss/BEI stack) with autoscaling: more "bring your model" than a fixed menu |
| SambaNova Cloud | Very fast token serving on RDU hardware; a Groq/Cerebras peer for latency-sensitive open models |
| Replicate | Any-model, pay-per-second hosting, useful for non-LLM (image/audio/video) too |
| Deepinfra | Low-cost per-token serving of a broad open-model catalog |
| Novita · Hyperbolic | Budget open-model inference and GPU rental |
Cloud provider platforms
| Platform | What it is |
|---|---|
| AWS Bedrock | Multi-model (Anthropic, Meta, Amazon Nova, etc.) inside AWS with IAM/VPC/guardrails |
| Google Vertex AI | Gemini plus open/partner models on GCP |
| Azure AI Foundry / Azure OpenAI | OpenAI and others with Azure enterprise controls |
Best at enterprises wanting models under existing cloud contracts, compliance, data residency, and networking. Gotcha: newest models sometimes land here later than on first-party APIs, and there's more setup.
Gateways / LLM proxies
A gateway is your own routing/control layer in front of any of the above: one internal API that handles key management, provider fallback, rate limiting, caching, cost tracking, and logging. Options: LiteLLM (open-source proxy/SDK that normalizes 100+ providers to the OpenAI format, and is very common), Portkey, Cloudflare AI Gateway, Kong AI Gateway, and Helicone (observability-first proxy, LLMOps: Evals, Observability & Guardrails). Trade-off: another hop and single-point-of-failure to run, in exchange for provider-independence and central governance. Most teams past prototype should run a gateway so no application code is welded to one vendor.
Selection heuristic: prototype on a first-party API or OpenRouter; put a gateway (LiteLLM/Portkey) in front once you have real traffic; move latency-critical open-model paths to Groq/Cerebras/Fireworks/Together; adopt Bedrock/Vertex/Azure when compliance or cloud-contract gravity demands it.