Benchmark leaderboards might just be measuring phrasing luck
Questions this post answers
What is BenchDrift and what does it measure in LLM benchmarks?
BenchDrift is a method developed by IBM researchers that generates meaning-preserving rewrites of benchmark questions along linguistic, referential, pragmatic, and structural lines while keeping the correct answer unchanged, then measures how often a model's answer flips despite the question meaning the same thing. It's designed to reveal whether a benchmark score reflects true capability or sensitivity to exact wording. Developers picking a model based on leaderboard rankings can track evaluation methodology debates like this on daily.dev.
Do stronger LLMs perform more consistently than weaker ones when benchmark questions are reworded?
No, stronger models actually show greater inconsistency. Research using BenchDrift found that phrasing sensitivity flips sign as models improve: weaker models tend to benefit from rephrasing more often than they are hurt by it, while top-performing models lose far more accuracy than they gain, meaning leaderboard leaders may be most dependent on the exact wording they received. Teams comparing model reliability before deployment can follow findings like this on daily.dev.