--- title: BenchMIRT: What are LLM benchmarks actually measuring? description: Ai2 introduces BenchMIRT, a method applying multidimensional Item Response Theory to audit LLM benchmarks at the individual question level. Trained on results... canonical: https://daily.dev/posts/benchmirt-what-are-llm-benchmarks-actually-measuring--kneq3on8p twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: BenchMIRT: What are LLM benchmarks actually measuring? | daily.dev og:description: Ai2 introduces BenchMIRT, a method applying multidimensional Item Response Theory to audit LLM benchmarks at the individual question level. Trained on results... og:url: https://daily.dev/posts/benchmirt-what-are-llm-benchmarks-actually-measuring--kneq3on8p og:image: https://api.daily.dev/og/posts/knEq3on8P.png og:image:alt: BenchMIRT: What are LLM benchmarks actually measuring? og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # BenchMIRT: What are LLM benchmarks actually measuring? **[Hugging Face](https://daily.dev/sources/huggingface)** · 8 min read · 2 upvotes · 0 comments ## Summary Ai2 introduces BenchMIRT, a method applying multidimensional Item Response Theory to audit LLM benchmarks at the individual question level. Trained on results from 100 LLMs across 16 benchmarks and 34K+ questions, it independently discovered two dominant capability dimensions—safety and general reasoning—without being told which benchmarks measured what. The analysis reveals that benchmarks like BBQ and WMDP, though categorized as safety tests, actually correlate more strongly with general reasoning ability. BenchMIRT can also identify the most informative questions, showing that keeping just 10-50% of questions often preserves the same model rankings, and it predicts held-out question performance with 79% accuracy versus 70% for a naive baseline. Limitations include training data only through March 2025 models and the risk that question-level insights could be misused to strip safety questions from evaluations. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does the BBQ bias benchmark correlate more with general reasoning than with safety in LLM evaluations? BenchMIRT's analysis of 100 LLMs found that BBQ, despite being designed and commonly grouped as a safety/bias benchmark, aligns much more strongly with general reasoning ability. This happens because many BBQ questions require tracking who's who in a scenario and reasoning from evidence rather than assumptions, so a low BBQ score may reflect difficulty with reasoning rather than poor safety behavior. _Teams choosing benchmarks to trust for safety claims can follow analyses like this one on daily.dev._ ### How accurately can multidimensional IRT models predict LLM performance on unseen benchmark questions? BenchMIRT correctly predicted whether a model would answer a held-out benchmark question correctly 79% of the time, compared to 70% accuracy for a simpler baseline that assumes a model performs on each question about as well as it does on the benchmark overall. This was measured across training data spanning 100 LLMs, 16 benchmarks, and over 34,000 questions. _Researchers evaluating models more efficiently can track evaluation methodology advances on daily.dev._ ### Why does stronger reasoning ability lead to lower scores on the WMDP dangerous-knowledge benchmark? WMDP tests dual-use knowledge in biology, chemistry, and cybersecurity that could enable misuse, and it scores refusing or failing to provide that knowledge as the desired response. BenchMIRT found WMDP scores correlate more with general reasoning than with safety, but stronger reasoning is associated with lower WMDP scores because more capable models are better at recognizing and providing the dangerous information the benchmark penalizes. _Anyone weighing safety benchmark design trade-offs can follow this kind of research on daily.dev._ ## Similar posts on daily.dev - [Your AI Interviewer Can’t Pass Its Own Test](https://daily.dev/posts/your-ai-interviewer-can-t-pass-its-own-test-ptlhkj7gk) · Medium · 1 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#data-science](https://daily.dev/tags/data-science), [#ai-safety](https://daily.dev/tags/ai-safety) [View this post on daily.dev](https://daily.dev/posts/benchmirt-what-are-llm-benchmarks-actually-measuring--kneq3on8p) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"BenchMIRT: What are LLM benchmarks actually measuring?","url":"https://daily.dev/posts/benchmirt-what-are-llm-benchmarks-actually-measuring--kneq3on8p","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/benchmirt-what-are-llm-benchmarks-actually-measuring--kneq3on8p"},"datePublished":"2026-09-01T21:40:55.547Z","dateModified":"2026-09-01T21:42:27.714Z","description":"Ai2 introduces BenchMIRT, a method applying multidimensional Item Response Theory to audit LLM benchmarks at the individual question level. Trained on results...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ba156733f93bf6fdf953cfd855487d73?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ba156733f93bf6fdf953cfd855487d73?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hugging Face","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hugging Face","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/f1f55c67d81a4330acf5b90b26b0c8e1","url":"https://daily.dev/sources/huggingface"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/benchmirt-what-are-llm-benchmarks-actually-measuring--kneq3on8p","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,data-science,ai-safety","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hugging Face","item":"https://daily.dev/sources/huggingface"},{"@type":"ListItem","position":3,"name":"BenchMIRT: What are LLM benchmarks actually measuring?"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/benchmirt-what-are-llm-benchmarks-actually-measuring--kneq3on8p#faq","mainEntity":[{"@type":"Question","name":"Why does the BBQ bias benchmark correlate more with general reasoning than with safety in LLM evaluations?","acceptedAnswer":{"@type":"Answer","text":"BenchMIRT's analysis of 100 LLMs found that BBQ, despite being designed and commonly grouped as a safety/bias benchmark, aligns much more strongly with general reasoning ability. This happens because many BBQ questions require tracking who's who in a scenario and reasoning from evidence rather than assumptions, so a low BBQ score may reflect difficulty with reasoning rather than poor safety behavior. Teams choosing benchmarks to trust for safety claims can follow analyses like this one on daily.dev."}},{"@type":"Question","name":"How accurately can multidimensional IRT models predict LLM performance on unseen benchmark questions?","acceptedAnswer":{"@type":"Answer","text":"BenchMIRT correctly predicted whether a model would answer a held-out benchmark question correctly 79% of the time, compared to 70% accuracy for a simpler baseline that assumes a model performs on each question about as well as it does on the benchmark overall. This was measured across training data spanning 100 LLMs, 16 benchmarks, and over 34,000 questions. Researchers evaluating models more efficiently can track evaluation methodology advances on daily.dev."}},{"@type":"Question","name":"Why does stronger reasoning ability lead to lower scores on the WMDP dangerous-knowledge benchmark?","acceptedAnswer":{"@type":"Answer","text":"WMDP tests dual-use knowledge in biology, chemistry, and cybersecurity that could enable misuse, and it scores refusing or failing to provide that knowledge as the desired response. BenchMIRT found WMDP scores correlate more with general reasoning than with safety, but stronger reasoning is associated with lower WMDP scores because more capable models are better at recognizing and providing the dangerous information the benchmark penalizes. Anyone weighing safety benchmark design trade-offs can follow this kind of research on daily.dev."}}]} ```