--- title: Open-weight models are fast on Neon AI Gateway. Here's why description: Neon AI Gateway serves open-weight models through Databricks Foundation Model APIs, inheriting a production inference stack with continuous batching, KV-cache... canonical: https://daily.dev/posts/open-weight-models-are-fast-on-neon-ai-gateway-here-s-why-dtkhuv4up twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Open-weight models are fast on Neon AI Gateway. Here's why | daily.dev og:description: Neon AI Gateway serves open-weight models through Databricks Foundation Model APIs, inheriting a production inference stack with continuous batching, KV-cache... og:url: https://daily.dev/posts/open-weight-models-are-fast-on-neon-ai-gateway-here-s-why-dtkhuv4up og:image: https://api.daily.dev/og/posts/DTkhUv4Up.png og:image:alt: Open-weight models are fast on Neon AI Gateway. Here's why og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Open-weight models are fast on Neon AI Gateway. Here's why **[Neon](https://daily.dev/sources/neontech)** · 7 min read · 0 upvotes · 0 comments ## Summary Neon AI Gateway serves open-weight models through Databricks Foundation Model APIs, inheriting a production inference stack with continuous batching, KV-cache paging, a TensorRT-LLM-based backend with custom kernels, quantization, and prompt caching. Prompt caching is highlighted as especially relevant for coding agents and multi-step tool loops that repeat the same system prompt and tool schema on every turn, with Databricks reporting a 2.5x throughput gain and 3x latency reduction at only ~30% cache hit rate when the feature launched on gpt-oss. The post explains how MoE architecture, tensor parallelism, FP8 quantization, and per-GPU/per-cloud tuning combine to keep theoretical speed gains intact under real traffic, and notes that calling open-weight models via Neon AI Gateway costs the standard per-token rate with no markup, with free tokens during its beta. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much does prompt caching improve LLM inference performance for open-weight models like gpt-oss? Prompt caching improved per-replica input-token throughput by 2.5x and reduced P50 latency by 3x for gpt-oss, even at a relatively low cache hit rate of about 30%, in a large production batch-inference pipeline at Databricks. The optimization skips prefill computation for repeated prompt prefixes such as system prompts and tool schemas, which is especially valuable for agent loops that resend the same context on every turn. _Teams tuning agent loop latency can track serving optimizations like prompt caching on daily.dev._ ### What speedup does FP8 quantization give when serving Llama 2 70B on H100 GPUs compared to FP16? FP8 quantization on H100 GPUs delivered roughly 2x larger maximum batch size and about 2.2x higher throughput versus FP16 for Llama 2 70B, along with roughly 30% better time-to-first-token from faster FP8 math. Quality checks on the Mosaic Gauntlet eval suite showed no meaningful average quality drop across dozens of benchmarks. _Engineers weighing quantization trade-offs for LLM serving can follow benchmarks like this on daily.dev._ ### How much faster is Llama 4 Maverick than Llama 3.3 70B in inference? Llama 4 Maverick delivers over 40% faster inference than Llama 3.3 70B, largely due to its Mixture-of-Experts architecture activating only a subset of expert sub-networks per token while keeping total parameter count high for quality. At small batch sizes this workload is memory-bandwidth bound, so loading only active experts can exceed the tokens-per-second of a comparably sized dense model under a tight latency budget. _Developers comparing MoE and dense model options for inference speed can follow benchmarks on daily.dev._ ## Similar posts on daily.dev - [LLMs belong in your backend](https://daily.dev/posts/llms-belong-in-your-backend-z3fdep2tj) · Neon · 1 upvotes · 0 comments - [The MoE-ification of the Open Model Ecosystem, and What It Means for Your Inference Bill](https://daily.dev/posts/the-moe-ification-of-the-open-model-ecosystem-and-what-it-means-for-your-inference-bill-e4tw8ail5) · DigitalOcean Community · 1 upvotes · 0 comments --- Tags: [#databricks](https://daily.dev/tags/databricks), [#ai-inference](https://daily.dev/tags/ai-inference), [#mixture-of-experts](https://daily.dev/tags/mixture-of-experts) [View this post on daily.dev](https://daily.dev/posts/open-weight-models-are-fast-on-neon-ai-gateway-here-s-why-dtkhuv4up) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Open-weight models are fast on Neon AI Gateway. Here's why","url":"https://daily.dev/posts/open-weight-models-are-fast-on-neon-ai-gateway-here-s-why-dtkhuv4up","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/open-weight-models-are-fast-on-neon-ai-gateway-here-s-why-dtkhuv4up"},"datePublished":"2026-08-20T14:09:49.327Z","dateModified":"2026-09-13T20:59:09.912Z","description":"Neon AI Gateway serves open-weight models through Databricks Foundation Model APIs, inheriting a production inference stack with continuous batching, KV-cache...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3c080994c7ae092f4a802bf86089849e?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3c080994c7ae092f4a802bf86089849e?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Neon","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Neon","logo":"https://media.daily.dev/image/upload/s--65EDghou--/f_auto/v1724339118/logos/neontech","url":"https://daily.dev/sources/neontech"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/open-weight-models-are-fast-on-neon-ai-gateway-here-s-why-dtkhuv4up","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"databricks,ai-inference,mixture-of-experts","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Neon","item":"https://daily.dev/sources/neontech"},{"@type":"ListItem","position":3,"name":"Open-weight models are fast on Neon AI Gateway. Here's why"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/open-weight-models-are-fast-on-neon-ai-gateway-here-s-why-dtkhuv4up#faq","mainEntity":[{"@type":"Question","name":"How much does prompt caching improve LLM inference performance for open-weight models like gpt-oss?","acceptedAnswer":{"@type":"Answer","text":"Prompt caching improved per-replica input-token throughput by 2.5x and reduced P50 latency by 3x for gpt-oss, even at a relatively low cache hit rate of about 30%, in a large production batch-inference pipeline at Databricks. The optimization skips prefill computation for repeated prompt prefixes such as system prompts and tool schemas, which is especially valuable for agent loops that resend the same context on every turn. Teams tuning agent loop latency can track serving optimizations like prompt caching on daily.dev."}},{"@type":"Question","name":"What speedup does FP8 quantization give when serving Llama 2 70B on H100 GPUs compared to FP16?","acceptedAnswer":{"@type":"Answer","text":"FP8 quantization on H100 GPUs delivered roughly 2x larger maximum batch size and about 2.2x higher throughput versus FP16 for Llama 2 70B, along with roughly 30% better time-to-first-token from faster FP8 math. Quality checks on the Mosaic Gauntlet eval suite showed no meaningful average quality drop across dozens of benchmarks. Engineers weighing quantization trade-offs for LLM serving can follow benchmarks like this on daily.dev."}},{"@type":"Question","name":"How much faster is Llama 4 Maverick than Llama 3.3 70B in inference?","acceptedAnswer":{"@type":"Answer","text":"Llama 4 Maverick delivers over 40% faster inference than Llama 3.3 70B, largely due to its Mixture-of-Experts architecture activating only a subset of expert sub-networks per token while keeping total parameter count high for quality. At small batch sizes this workload is memory-bandwidth bound, so loading only active experts can exceed the tokens-per-second of a comparably sized dense model under a tight latency budget. Developers comparing MoE and dense model options for inference speed can follow benchmarks on daily.dev."}}]} ```