--- title: Benchmark leaderboards might just be measuring phrasing luck description: IBM researchers introduce BenchDrift, a method that generates meaning-preserving rewrites of benchmark questions along linguistic, referential, pragmatic, and... canonical: https://daily.dev/posts/benchmark-leaderboards-might-just-be-measuring-phrasing-luck-g5suhbquy twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Benchmark leaderboards might just be measuring phrasing luck | daily.dev og:description: IBM researchers introduce BenchDrift, a method that generates meaning-preserving rewrites of benchmark questions along linguistic, referential, pragmatic, and... og:url: https://daily.dev/posts/benchmark-leaderboards-might-just-be-measuring-phrasing-luck-g5suhbquy og:image: https://api.daily.dev/og/posts/G5suhbQuY.png og:image:alt: Benchmark leaderboards might just be measuring phrasing luck og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Benchmark leaderboards might just be measuring phrasing luck **[Trends](https://daily.dev/sources/trends)** · 2 min read · 0 upvotes · 0 comments ## Summary IBM researchers introduce BenchDrift, a method that generates meaning-preserving rewrites of benchmark questions along linguistic, referential, pragmatic, and structural dimensions while keeping the correct answer fixed, then measures how often models flip their answers. The surprising result: phrasing sensitivity doesn't shrink as models improve, it flips sign. Weaker models tend to benefit from rephrasing more often than they're hurt, while stronger, top-ranked models lose far more than they gain, suggesting leaderboard-topping scores may be heavily dependent on the exact wording of test questions rather than true robustness. ## Content Turns out a lot of what we call "model improvement" might just be models getting lucky with wording. A new IBM paper making the rounds, BenchDrift, is forcing an uncomfortable question: how much of the leaderboard is actually signal? The setup is simple. Take existing benchmarks (GSM8K, MMLU, MATH-Hard), generate meaning-preserving rephrasings along linguistic, referential, pragmatic, and structural axes, keep the correct answer fixed, then check how often the model's answer flips anyway. As @rohanpaul_ai put it: "A model can know the answer and still fail because you asked the same question differently." The numbers are rough. Across 8 models and 3 benchmarks, the gap between best-case and worst-case accuracy averaged 74.7 percentage points. Same question, same answer, wildly different scores depending on how you asked. The part that stings for anyone chasing state-of-the-art claims: stronger models are more exposed to this, not less. @omarsar0 summed up the twist: "Phrasing sensitivity does not fade as models improve. It changes sign. Weak models gain more from rephrasing than they lose, while strong models lose far more than they gain, so the top models on a benchmark are the ones whose scores depend most on the wording they happened to receive." In other words, the model at the top of the leaderboard might just be the one that got the friendliest phrasing. Confidence doesn't save you either. Even in the model's highest-confidence answer bucket, 18.5% of correct answers got broken by a meaning-preserving reword. And the fragility itself is consistent across models: the eight models tested largely agreed on *which* rephrasings were the costly ones, even though they disagreed on overall drift. The practical takeaway, per omarsar0: if you're picking a model based on a small benchmark delta, some of that delta belongs to the phrasing, not the model. Test multiple equivalent phrasings or report a range instead of a single score. There's a broader point floating around this thread too, courtesy of Scott Hanselman: "Anything you leave unclear when prompting, the AI will fill with the statistical mean, with mediocrity." BenchDrift is basically that idea turned into a rigorous audit: ambiguity and phrasing quirks aren't noise around the edges of model behavior, they're baked into how these systems answer at all. Worth remembering next time a launch post leads with a benchmark chart. ## Questions this post answers ### How much can rephrasing a benchmark question change an LLM's accuracy on the same question? Rephrasing benchmark questions while preserving meaning can swing accuracy by an average of 74.7 percentage points between best-case and worst-case phrasing, according to IBM's BenchDrift study across 8 models and 3 benchmarks (GSM8K, MMLU, MATH-Hard). Even in a model's highest-confidence answer bucket, 18.5% of correct answers were broken by a meaning-preserving reword. _daily.dev surfaces research like this for anyone weighing benchmark deltas before picking a model._ ### Do stronger LLMs suffer more from phrasing sensitivity than weaker ones on benchmarks? Yes, phrasing sensitivity does not fade as models improve, it changes sign. Weak models tend to gain accuracy from rephrasing while strong models lose more than they gain, meaning the top-ranked model on a leaderboard may simply be the one that received the friendliest phrasing rather than the objectively best model. _model comparisons get easier to trust when developers track findings like this on daily.dev._ ### How should I compare LLM benchmark scores if phrasing affects the results so much? Test multiple equivalent phrasings of the same question and report a range of scores rather than relying on a single benchmark number, since part of any small benchmark delta between models may come from phrasing rather than actual capability. The eight models tested in BenchDrift largely agreed on which rephrasings were costly, suggesting this fragility is systemic, not random noise. _daily.dev helps developers stay skeptical of leaderboard claims when choosing between models._ ## Similar posts on daily.dev - [Goodhart’s Law Comes for Every Benchmark You Trust – Communications of the ACM](https://daily.dev/posts/goodhart-s-law-comes-for-every-benchmark-you-trust-communications-of-the-acm-sm0gukyij) · Hacker News · 1 upvotes · 1 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#llm](https://daily.dev/tags/llm), [#ibm](https://daily.dev/tags/ibm) [View this post on daily.dev](https://daily.dev/posts/benchmark-leaderboards-might-just-be-measuring-phrasing-luck-g5suhbquy) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Benchmark leaderboards might just be measuring phrasing luck","url":"https://daily.dev/posts/benchmark-leaderboards-might-just-be-measuring-phrasing-luck-g5suhbquy","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/benchmark-leaderboards-might-just-be-measuring-phrasing-luck-g5suhbquy"},"datePublished":"2026-08-17T05:59:22.259Z","dateModified":"2026-09-13T20:01:57.200Z","description":"IBM researchers introduce BenchDrift, a method that generates meaning-preserving rewrites of benchmark questions along linguistic, referential, pragmatic, and...","isAccessibleForFree":true,"articleSection":"Trends","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Trends","logo":"https://media.daily.dev/image/upload/s--ZfSp3asX--/f_auto,q_auto/v1780996004/logos/trends?_a=BAMAMiWQ0","url":"https://daily.dev/sources/trends"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/benchmark-leaderboards-might-just-be-measuring-phrasing-luck-g5suhbquy","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,llm,ibm","timeRequired":"PT2M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Trends","item":"https://daily.dev/sources/trends"},{"@type":"ListItem","position":3,"name":"Benchmark leaderboards might just be measuring phrasing luck"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/benchmark-leaderboards-might-just-be-measuring-phrasing-luck-g5suhbquy#faq","mainEntity":[{"@type":"Question","name":"How much can rephrasing a benchmark question change an LLM's accuracy on the same question?","acceptedAnswer":{"@type":"Answer","text":"Rephrasing benchmark questions while preserving meaning can swing accuracy by an average of 74.7 percentage points between best-case and worst-case phrasing, according to IBM's BenchDrift study across 8 models and 3 benchmarks (GSM8K, MMLU, MATH-Hard). Even in a model's highest-confidence answer bucket, 18.5% of correct answers were broken by a meaning-preserving reword. daily.dev surfaces research like this for anyone weighing benchmark deltas before picking a model."}},{"@type":"Question","name":"Do stronger LLMs suffer more from phrasing sensitivity than weaker ones on benchmarks?","acceptedAnswer":{"@type":"Answer","text":"Yes, phrasing sensitivity does not fade as models improve, it changes sign. Weak models tend to gain accuracy from rephrasing while strong models lose more than they gain, meaning the top-ranked model on a leaderboard may simply be the one that received the friendliest phrasing rather than the objectively best model. model comparisons get easier to trust when developers track findings like this on daily.dev."}},{"@type":"Question","name":"How should I compare LLM benchmark scores if phrasing affects the results so much?","acceptedAnswer":{"@type":"Answer","text":"Test multiple equivalent phrasings of the same question and report a range of scores rather than relying on a single benchmark number, since part of any small benchmark delta between models may come from phrasing rather than actual capability. The eight models tested in BenchDrift largely agreed on which rephrasings were costly, suggesting this fragility is systemic, not random noise. daily.dev helps developers stay skeptical of leaderboard claims when choosing between models."}}]} ```