--- title: Fixing Tool-Call Accuracy in vLLM | daily.dev description: Enabling constrained decoding in vLLM 0.31 fixes tool-call schema errors when serving Kimi and DeepSeek models. Ubicloud found that serving Kimi K2.6 on vLLM... canonical: https://daily.dev/posts/fixing-tool-call-accuracy-in-vllm-xkddqafvu twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Fixing Tool-Call Accuracy in vLLM | daily.dev og:description: Enabling constrained decoding in vLLM 0.31 fixes tool-call schema errors when serving Kimi and DeepSeek models. Ubicloud found that serving Kimi K2.6 on vLLM... og:url: https://daily.dev/posts/fixing-tool-call-accuracy-in-vllm-xkddqafvu og:image: https://api.daily.dev/og/posts/XkddqAfvU.png og:image:alt: Fixing Tool-Call Accuracy in vLLM og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Fixing Tool-Call Accuracy in vLLM **[Hacker News](https://daily.dev/sources/hn)** · 9 min read · 1 upvotes · 0 comments ## Summary Enabling constrained decoding in vLLM 0.31 fixes tool-call schema errors when serving Kimi and DeepSeek models. Ubicloud found that serving Kimi K2.6 on vLLM 0.28 produced valid tool-call schemas only 92% of the time, causing context pollution and retries. vLLM implements constrained decoding using structural tags from the XGrammar library, restricting token sampling to valid paths during generation. After vLLM disabled constrained decoding by default for broader model compatibility, Ubicloud re-enabled it, reaching 100% schema accuracy on the K2VV ToolCall benchmark. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I fix malformed tool calls when serving Kimi or DeepSeek models with vLLM? Enable constrained decoding, which became a server-side flag in vLLM 0.31. It restricts token sampling to only valid tokens at each generation step using structural tags from the XGrammar library, guaranteeing schema-adherent tool calls instead of relying on the model's best effort. One deployment saw tool-call schema accuracy rise from 92% to 100% after enabling it. _Track vLLM releases like this on daily.dev before a tool-calling bug hits your agent in production._ ### Why does vLLM disable constrained decoding by default even though it fixes tool-call accuracy? vLLM disabled constrained decoding by default because it is a model-agnostic serving engine and support for constrained decoding is a per-model task; maintainers reported problems with several models once it was enabled broadly. vLLM later added a --tool-strict-level flag so operators can opt into stricter tool-call enforcement for specific deployments. _Developers tuning vLLM flags for specific models can follow changes like this on daily.dev._ ### What is the K2VV ToolCall benchmark used to measure LLM tool-calling quality? K2VV ToolCall is a benchmark published by Moonshot AI consisting of 2000 requests that evaluate whether a model triggers tool calls and whether those calls are syntactically correct. It reports two metrics: trigger similarity, comparing triggering decisions to Moonshot AI's own service, and schema accuracy, the fraction of tool calls that adhere to the schema and can be invoked successfully. _Teams benchmarking inference engines for agentic workloads can track findings like this on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 47 comments across hackernews (as of 2026-10-26). **TL;DR:** Discussion centers on how constrained decoding works and its tradeoffs for tool-call accuracy, with the post's authors actively engaging to clarify technical details; most commenters are curious and technically engaged rather than critical, though some push back on whether the framing of 'fixing vLLM' is accurate and debate whether constrained decoding can hurt output quality versus retry-based approaches. **Sentiment:** 35% positive · 55% mixed · 10% skeptical **The case for** - The authors' active participation answering technical questions was well received and praised for engineering rigor and benchmarking. - Some note constrained decoding is clearly becoming an industry trend, with OpenAI and Moonshot both defaulting to it. - One commenter appreciated the article's clarity and didn't find it confusing at all. **The pushback** - Several argue the post's framing is misleading since constrained decoding and constrained tool-call support already existed in vLLM for a long time — it was really a configuration fix, not a vLLM fix. - Debate over whether constrained decoding degrades model output quality by forcing out-of-distribution tokens compared to a retry-based approach. - One commenter criticizes vLLM itself as a confusing, fragile, overcomplicated piece of software despite being powerful once tuned. **By community** - hackernews (mixed): Technically rich thread mixing genuine curiosity about special tokens and constrained decoding mechanics with some pushback on the post's framing and quality tradeoffs, while the authors' direct engagement kept the tone largely constructive. **Hottest debate:** Whether schema-constrained decoding meaningfully degrades model output quality compared to a retry-until-success strategy for tool calls. **Open questions** - Does constrained decoding negatively affect overall model/agent quality compared to retrying failed tool calls, and by how much? - Would this technique work well with speculative/inference optimization methods like DFlash and MTP without excessive overhead? - What would a 'fully-constrained tool call environment' look like given that tool definitions can change mid-run? **Highlights** > The way it works is to disallow invalid next states, and getting backed into a corner is one way to think about one possible failure mode. For a concrete example (happens less nowadays with bigger models, but it still happens, especially later into a context window), models hate long json strings. It's very easy to get a json output like {"name": "Bob", "biography": "Bob is a fantastic builder, having ... Critically, the ellipses terminate the JSON. With schema-constrained decoding, you _must_ eventually terminate that broken string, and it'll typically terminate it right after the ellipses. Basically, by the point you got to the first period the output was borked. You were backed into a corner. Retry as a strategy fixes that. As another example failure mode that's a bit more subtle, you might have a set of "correct" tokens early on in the output which get the KV cache into a bit of a wonky state, causing it to have worse outputs toward the end of your output. Disallowing invalid next states forces you to generate tokens in that worse-output state. Retry lets you fail more often and then hopefully not have those wonky-state early tokens. It's not really that you were backed into a bad state as a hard rule, but that you were backed into a bad state as a probabilistic statement. For a toy model to think about, suppose the entire grammar is [ab, ba], and the LLM's universe of training data is [ab, ab, ba, bb]. The "right" distribution (retry on failure) is {ab: 67%, ba: 33%}. The schema-constrained distribution is {ab: 50%, ba: 50%}. > — [hansvm on hackernews](https://news.ycombinator.com/item?id=50245714) > > but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same? Yes, this is what constrained decoding means. You constrain the model to only output from a set of "allowed" tokens (a.k.a. grammars). So, if you use tool use with constrained decoding, you can be certain that the model will always choose from a list of approved tools (i.e. if tools=['read','write'], it cannot output 'execute', even if in other sessions it could, if execute was in the approved tools). The tricky problem is that certain models work / have been trained in certain ways. And constraining them can lead to "invisible" problems. Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file). > Could something like that also be used in other contexts? Sure. You can build pretty cool grammars and constrain the model to do cool stuff, akin to "tool use" but more free hand. You could, for example, make it always emit a 10 point plan. For any prompt, it will give you a fixed syntax 10 point plan, always. And the plan will be constant with a plan_title and 10{plan_item, item_text}. But, as above, there's no guarantee that the individual items in the plan would make any sense... (i.e. step7 in a pasta making recipe would be "add glue"). > — [NitpickLawyer on hackernews · 1 comments](https://news.ycombinator.com/item?id=50245873) > Constrained decoding has been in vLLM for a long time, and constrained tool call decoding has been in vLLM for a while, too. The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly. To be fair, the course of constrained tool calling configuration in vLLM changed over time in ways that changed behavior, which is the real problem. The Ubicloud team deserves a lot of credit for having benchmarks in place to track performance and catch regressions for investigations. Some of the providers you find on OpenRouter have inexplicably bad performance relative to others and clearly do not care this much. However, if this article left you confused because constrained decoding and constrained tool call schema have been features in vLLM for a long time, jump to the end where they show what configuration changes they made. > — [Aurornis on hackernews · 3 comments](https://news.ycombinator.com/item?id=50245479) > Overall, a fair assessment. I do want to place a footnote here that we did fix vLLM in the sense that its correct behavior for Kimi required code changes: it was impossible to fix it in the configuration. Later, the vLLM team fixed this problem upstream. Now, with the latest version of vLLM, the problem we describe can be addressed by setting the `--tool-strict-level` flag to `parameter`. > — [jesdo on hackernews](https://news.ycombinator.com/item?id=50245660) > > For example, when we served Kimi K2.6 with vLLM version 0.28, we got the expected tool call schema only 92% of the time. In other words, 2 out of every 25 attempted tool calls failed. 100% success is critical for tool calling. The probability that the overall agent iteration is successful would scale very poorly even if we achieved 95%. With p95, if you need 50 tool calls that must execute sequentially to achieve a result, the chances that you won't have an error are only ~8%. There are ways to recover from bad tool calls (error feedback helps a lot), but if the agent is persistently using the wrong tools with the wrong args the overall system will not work very well. > — [bob1029 on hackernews](https://news.ycombinator.com/item?id=50246237) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50242870) · 82 points · 47 comments ## Similar posts on daily.dev - [Chasing 100% Accuracy: A Deep Dive into Debugging Kimi K2's Tool-Calling on vLLM](https://daily.dev/posts/chasing-100-accuracy-a-deep-dive-into-debugging-kimi-k2-s-tool-calling-on-vllm-ipsnwzjkr) · vLLM · 1 upvotes · 0 comments - [Tool calling, open source, and the M×N problem](https://daily.dev/posts/tool-calling-open-source-and-the-m-n-problem-lpmjyqzzd) · Hacker News · 0 upvotes · 0 comments - [Structured Output Reliability at Scale: When JSON Schema Validity Breaks Down Under Concurrency](https://daily.dev/posts/structured-output-reliability-at-scale-when-json-schema-validity-breaks-down-under-concurrency-vhkoljwe3) · DigitalOcean Community · 1 upvotes · 0 comments - [Kimi K3 Performance Optimizations in vLLM: The Road to 2.8× Throughput](https://daily.dev/posts/kimi-k3-performance-optimizations-in-vllm-the-road-to-2-8-throughput-tlqrbg8ni) · vLLM · 0 upvotes · 0 comments - [The biggest local LLM on your machine is useless if it can't call a single tool, no matter how many parameters it has](https://daily.dev/posts/the-biggest-local-llm-on-your-machine-is-useless-if-it-can-t-call-a-single-tool-no-matter-how-many--uauiyyggz) · XDA Developers · 1 upvotes · 0 comments --- Tags: [#open-source](https://daily.dev/tags/open-source), [#llm](https://daily.dev/tags/llm), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/fixing-tool-call-accuracy-in-vllm-xkddqafvu) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Fixing Tool-Call Accuracy in vLLM","url":"https://daily.dev/posts/fixing-tool-call-accuracy-in-vllm-xkddqafvu","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/fixing-tool-call-accuracy-in-vllm-xkddqafvu"},"datePublished":"2026-10-26T15:15:09.522Z","dateModified":"2026-10-26T16:16:12.335Z","description":"Enabling constrained decoding in vLLM 0.31 fixes tool-call schema errors when serving Kimi and DeepSeek models. Ubicloud found that serving Kimi K2.6 on vLLM...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/99417ab814a1edd3217849dae086b382?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/99417ab814a1edd3217849dae086b382?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/fixing-tool-call-accuracy-in-vllm-xkddqafvu","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"open-source,llm,ai-inference,vllm","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Fixing Tool-Call Accuracy in vLLM"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/fixing-tool-call-accuracy-in-vllm-xkddqafvu#faq","mainEntity":[{"@type":"Question","name":"How do I fix malformed tool calls when serving Kimi or DeepSeek models with vLLM?","acceptedAnswer":{"@type":"Answer","text":"Enable constrained decoding, which became a server-side flag in vLLM 0.31. It restricts token sampling to only valid tokens at each generation step using structural tags from the XGrammar library, guaranteeing schema-adherent tool calls instead of relying on the model's best effort. One deployment saw tool-call schema accuracy rise from 92% to 100% after enabling it. Track vLLM releases like this on daily.dev before a tool-calling bug hits your agent in production."}},{"@type":"Question","name":"Why does vLLM disable constrained decoding by default even though it fixes tool-call accuracy?","acceptedAnswer":{"@type":"Answer","text":"vLLM disabled constrained decoding by default because it is a model-agnostic serving engine and support for constrained decoding is a per-model task; maintainers reported problems with several models once it was enabled broadly. vLLM later added a --tool-strict-level flag so operators can opt into stricter tool-call enforcement for specific deployments. Developers tuning vLLM flags for specific models can follow changes like this on daily.dev."}},{"@type":"Question","name":"What is the K2VV ToolCall benchmark used to measure LLM tool-calling quality?","acceptedAnswer":{"@type":"Answer","text":"K2VV ToolCall is a benchmark published by Moonshot AI consisting of 2000 requests that evaluate whether a model triggers tool calls and whether those calls are syntactically correct. It reports two metrics: trigger similarity, comparing triggering decisions to Moonshot AI's own service, and schema accuracy, the fraction of tool calls that adhere to the schema and can be invoked successfully. Teams benchmarking inference engines for agentic workloads can track findings like this on daily.dev."}}]} ```