Neon AI Gateway serves open-weight models through Databricks Foundation Model APIs, inheriting a production inference stack with continuous batching, KV-cache paging, a TensorRT-LLM-based backend with custom kernels, quantization, and prompt caching. Prompt caching is highlighted as especially relevant for coding agents and multi-step tool loops that repeat the same system prompt and tool schema on every turn, with Databricks reporting a 2.5x throughput gain and 3x latency reduction at only ~30% cache hit rate when the feature launched on gpt-oss. The post explains how MoE architecture, tensor parallelism, FP8 quantization, and per-GPU/per-cloud tuning combine to keep theoretical speed gains intact under real traffic, and notes that calling open-weight models via Neon AI Gateway costs the standard per-token rate with no markup, with free tokens during its beta.
Table of contents
Open-weight models are built to be fast, but bad serving cancels thatHow we optimize performance in open-weight modelsPrompt caching for open-weight modelsCall your LLMs from NeonQuestions this post answers
How much does prompt caching improve LLM inference throughput and latency for open-weight models like gpt-oss?
Prompt caching on gpt-oss in a large production batch-inference pipeline improved per-replica input-token throughput by 2.5x and reduced P50 latency by 3x, even at a relatively low cache hit ratio of around 30%. This works by reusing the KV cache for shared prompt prefixes such as system prompts and tool schemas, skipping redundant prefill computation on cache hits. Agent builders juggling latency budgets can follow prompt caching gains for open-weight models on daily.dev.
Why do Mixture-of-Experts models run faster than dense models of similar size in production serving?
MoE models activate only a few expert sub-networks per token instead of the full parameter set, which lowers active compute per token while keeping total parameters high for quality. At small batch sizes the workload becomes memory-bandwidth bound, so loading only active experts lets MoE models exceed the tokens-per-second of a comparable dense model, as seen with Llama 4 Maverick running over 40% faster than Llama 3.3 70B. Developers comparing MoE and dense model serving costs can track architecture tradeoffs like this on daily.dev.
How much faster did FP8 quantization make Llama 2 70B inference on H100 GPUs compared to FP16?
FP8 quantization of Llama 2 70B on H100 GPUs roughly doubled the maximum batch size and increased throughput by about 2.2x compared to FP16, with roughly 30% better time-to-first-token on prefill from faster FP8 math. Quality checks on the Mosaic Gauntlet eval suite showed no meaningful average quality drop across dozens of benchmarks. Teams weighing quantization tradeoffs for LLM serving can keep up with benchmarks like this on daily.dev.