Ramp's engineering team built an LLM gateway that routes requests across models and service tiers using Thompson Sampling over lognormal latency posteriors. The system tracks failure rates via EWMA and maintains a Normal-Inverse-Gamma prior on log-latency per context slice, updating posteriors stored in Redis. At routing time, each model is scored by combining its failure probability, the probability of exceeding a latency deadline, and relative cost. Deployed first to a reranker use case with a 6-second deadline, the system dynamically shifted traffic away from rate-limited Gemini to GPT Nano Flex (half the price of Standard) when latencies were comparable, achieving over 25% cost savings with no quality degradation. A streaming variant shipped to Ramp Inspect yielded 30% savings.
