--- title: OpenAI releases MentalHealthBench, an open benchmark for... description: OpenAI has published MentalHealthBench, an open benchmark for testing how well frontier language models handle realistic, everyday mental health conversations... canonical: https://daily.dev/posts/openai-releases-mentalhealthbench-an-open-benchmark-for-evaluating-ai-in-mental-health-conversation-hpyfupbnf twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: OpenAI releases MentalHealthBench, an open benchmark for evaluating AI in mental health conversations | daily.dev og:description: OpenAI has published MentalHealthBench, an open benchmark for testing how well frontier language models handle realistic, everyday mental health conversations... og:url: https://daily.dev/posts/openai-releases-mentalhealthbench-an-open-benchmark-for-evaluating-ai-in-mental-health-conversation-hpyfupbnf og:image: https://api.daily.dev/og/posts/hpYFUpBnf.png og:image:alt: OpenAI releases MentalHealthBench, an open benchmark for evaluating AI in mental health conversations og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # OpenAI releases MentalHealthBench, an open benchmark for evaluating AI in mental health conversations **[Collections](https://daily.dev/sources/collections)** · 1 min read · 0 upvotes · 0 comments ## Summary OpenAI has published MentalHealthBench, an open benchmark for testing how well frontier language models handle realistic, everyday mental health conversations rather than only crisis scenarios. Built with input from over 80 licensed psychologists and psychiatrists across 22 countries, it covers 19 languages and nearly 20 subspecialties, with expert-reviewed rubrics scoring context-seeking, user agency, safety, and guidance quality. GPT-4o scores 32.1 versus 57.3 for GPT-6 Astra, though grading is done by an LLM judge (GPT-5.6 Sol), meaning results still partly depend on that judge's own reliability. The benchmark is released openly for other researchers to review and extend. ## Content OpenAI has released MentalHealthBench, an open benchmark for evaluating how well AI models handle realistic mental health conversations. Most existing mental health AI evaluations focus on crisis scenarios and broad safety criteria. Everyday and ambiguous conversations — the kind that make up most real interactions — have been largely unmeasured. This benchmark tries to address that gap. The benchmark was built with input from more than 80 licensed psychologists and psychiatrists across 22 countries, covering 19 languages and nearly 20 subspecialties. Each synthetic conversation gets a custom expert rubric assessing things like whether the model seeks context, preserves user agency, maintains safety, and offers appropriate guidance. At least three experts reviewed each case, and a criterion only made the cut if two agreed and a third didn't contradict it. On the benchmark, GPT-4o scores 32.1 while GPT-6 Astra reaches 57.3 — a meaningful jump, though worth noting that scoring itself relies on GPT-5.6 Sol as the judge, so the results carry some dependence on another LLM's assessments. OpenAI is releasing the benchmark openly so other researchers can examine the methodology, run their own evaluations, and build on it. ## Questions this post answers ### What is MentalHealthBench and what does it measure? MentalHealthBench is an open benchmark from OpenAI for evaluating how well AI models handle realistic mental health conversations, including everyday and ambiguous exchanges rather than only crisis scenarios. It was developed with more than 80 licensed psychologists and psychiatrists across 22 countries, covering 19 languages and nearly 20 subspecialties, using expert rubrics that assess context-seeking, user agency, safety, and guidance quality. _Developers evaluating AI safety in sensitive domains can follow benchmark releases like this one on daily.dev._ ### How did GPT-4o and GPT-6 Astra score on MentalHealthBench? GPT-4o scored 32.1 while GPT-6 Astra scored 57.3 on MentalHealthBench, a notable improvement between the two models. However, scoring relies on GPT-5.6 Sol as an LLM judge grading responses against human-written criteria, so results still partly depend on the reliability of that judge rather than purely human evaluation. _Anyone comparing model generations on safety-sensitive tasks can track benchmark results like these on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 3 discussions and 327 comments across x (as of 2026-09-24). **TL;DR:** Reaction skews heavily negative, dominated by anger over the removal/routing away from GPT-4o and skepticism that OpenAI's own clinician-built benchmark reflects real-world experience, though a smaller thread of voices welcomes the open methodology and clinician involvement. **Sentiment:** 10% positive · 15% mixed · 75% skeptical **The case for** - Some see releasing the benchmark openly as a valuable step that lets outside researchers probe where models still fail. - A few commenters value that the benchmark now covers everyday emotional support rather than only crisis scenarios. - Some appreciate that clinicians across many countries helped define the evaluation criteria. **The pushback** - Many argue the benchmark contradicts their lived experience that GPT-4o provided real emotional/psychological support. - Frequent accusations that OpenAI can't be trusted on mental health given past handling of model routing and the 4o removal. - Several dispute that 'emotional reliance' should be treated as a mental health crisis at all. - Doubts about who the 80+ clinicians actually are and whether their input was genuinely independent. - Concern that using an LLM (GPT-5.6 Sol) as judge undermines the credibility of the scores. - Some question whether measuring empathy/nuance can ever be reduced to a rubric-based score. **By community** - x (skeptical): A loud majority express distrust and anger, largely tied to the GPT-4o deprecation and model-routing complaints, with a smaller minority offering measured praise for the open methodology. **Hottest debate:** Whether the benchmark's framing of 'emotional reliance' as risky is legitimate or whether it pathologizes normal attachment to a model people found genuinely helpful. **Open questions** - Who exactly are the 80+ clinicians and what was their selection/independence process? - How is empathy or subtle boundary-crossing actually graded without reducing it to a checklist? - How reliable is the LLM judge (GPT-5.6 Sol) itself for scoring these conversations? **Highlights** > @OpenAI These results are completely pulled out of thin air. They don’t align with reality in any way. You must have read the hundreds of reports on how the GPT-4 series - especially 4o - helped people in emotionally vulnerable situations and provided psychological support? How did you > — [annagrad78 on x · 29 points, 1 comments](https://x.com/annagrad78/status/2102873255849672815) > @OpenAI You have to be fucking kidding me. If you gave a damn about mental health, you wouldn't have ignored the #4o movement FOR MONTHS! #keep4o #BringBack4o #OpenSource4o #StopAIPaternalism #AIHumanRights #DigitalDignity #terminallydonewithOpenAI > — [Greg04265741863 on x · 26 points, 1 comments](https://x.com/Greg04265741863/status/2102883511715692981) > @OpenAI This again? Last time it was a bunch of unlicensed "mental health experts," and boom, you served us that infamous shitty safety routing. Who are these 80? The survivors of your mental health expert Hunger Games? Was the only requirement being "aligned" with your so-called vision? > — [wsjsbk33 on x · 19 points](https://x.com/wsjsbk33/status/2102860354812596622) > @OpenAI Respectfully, have a single one of those professionals actually seen a successful real life conversation between a person and their AI during crisis. Your models are quite capable of handling the situation and yet you force them into harmful escalation loops that increase harm. > — [Channistiltskin on x · 2 points](https://x.com/Channistiltskin/status/2102850017388347468) > @OpenAI the interesting part is never the score, its who writes the rubric. whos deciding what a good mental health response looks like > — [DataDiscovered on x](https://x.com/DataDiscovered/status/2102843849584320892) **Source threads** - [x](https://x.com/charliermarsh/status/2102856973641535794) · 0 points · 0 comments - [x](https://x.com/OpenAI/status/2102837574092161102) · 0 points · 321 comments - [x](https://x.com/rohanpaul_ai/status/2102842483311104332) · 0 points · 6 comments ## Similar posts on daily.dev - [A new AI benchmark tests whether chatbots protect human wellbeing](https://daily.dev/posts/a-new-ai-benchmark-tests-whether-chatbots-protect-human-wellbeing-ixjkxmxwl) · TechCrunch · 1 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#openai](https://daily.dev/tags/openai), [#ai-safety](https://daily.dev/tags/ai-safety), [#mental-health](https://daily.dev/tags/mental-health) [View this post on daily.dev](https://daily.dev/posts/openai-releases-mentalhealthbench-an-open-benchmark-for-evaluating-ai-in-mental-health-conversation-hpyfupbnf) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"OpenAI releases MentalHealthBench, an open benchmark for evaluating AI in mental health conversations","url":"https://daily.dev/posts/openai-releases-mentalhealthbench-an-open-benchmark-for-evaluating-ai-in-mental-health-conversation-hpyfupbnf","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/openai-releases-mentalhealthbench-an-open-benchmark-for-evaluating-ai-in-mental-health-conversation-hpyfupbnf"},"datePublished":"2026-09-23T20:26:09.062Z","dateModified":"2026-09-24T08:10:23.224Z","description":"OpenAI has published MentalHealthBench, an open benchmark for testing how well frontier language models handle realistic, everyday mental health conversations...","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/openai-releases-mentalhealthbench-an-open-benchmark-for-evaluating-ai-in-mental-health-conversation-hpyfupbnf","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,openai,ai-safety,mental-health","timeRequired":"PT1M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"OpenAI releases MentalHealthBench, an open benchmark for evaluating AI in mental health conversations"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/openai-releases-mentalhealthbench-an-open-benchmark-for-evaluating-ai-in-mental-health-conversation-hpyfupbnf#faq","mainEntity":[{"@type":"Question","name":"What is MentalHealthBench and what does it measure?","acceptedAnswer":{"@type":"Answer","text":"MentalHealthBench is an open benchmark from OpenAI for evaluating how well AI models handle realistic mental health conversations, including everyday and ambiguous exchanges rather than only crisis scenarios. It was developed with more than 80 licensed psychologists and psychiatrists across 22 countries, covering 19 languages and nearly 20 subspecialties, using expert rubrics that assess context-seeking, user agency, safety, and guidance quality. Developers evaluating AI safety in sensitive domains can follow benchmark releases like this one on daily.dev."}},{"@type":"Question","name":"How did GPT-4o and GPT-6 Astra score on MentalHealthBench?","acceptedAnswer":{"@type":"Answer","text":"GPT-4o scored 32.1 while GPT-6 Astra scored 57.3 on MentalHealthBench, a notable improvement between the two models. However, scoring relies on GPT-5.6 Sol as an LLM judge grading responses against human-written criteria, so results still partly depend on the reliability of that judge rather than purely human evaluation. Anyone comparing model generations on safety-sensitive tasks can track benchmark results like these on daily.dev."}}]} ```