--- title: Embedded AI evaluators won’t be an easy solution,... description: Hugging Face's chief policy officer Irene Solaiman discusses the limits of AI model benchmarking at TechCrunch Disrupt, arguing that reproducibility and... canonical: https://daily.dev/posts/embedded-ai-evaluators-won-t-be-an-easy-solution-hugging-face-s-policy-chief-says-nhkjvfxdt copyright: © 2026 Daily Dev Ltd. terms: https://daily.dev/tos/ twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Embedded AI evaluators won’t be an easy solution, Hugging Face’s policy chief says | daily.dev og:description: Hugging Face's chief policy officer Irene Solaiman discusses the limits of AI model benchmarking at TechCrunch Disrupt, arguing that reproducibility and... og:url: https://daily.dev/posts/embedded-ai-evaluators-won-t-be-an-easy-solution-hugging-face-s-policy-chief-says-nhkjvfxdt og:image: https://api.daily.dev/og/posts/NhkjVfxdT.png og:image:alt: Embedded AI evaluators won’t be an easy solution, Hugging Face’s policy chief says og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Embedded AI evaluators won’t be an easy solution, Hugging Face’s policy chief says **[TechCrunch](https://daily.dev/sources/tc)** · 3 min read · 0 upvotes · 0 comments ## Summary Hugging Face's chief policy officer Irene Solaiman discusses the limits of AI model benchmarking at TechCrunch Disrupt, arguing that reproducibility and cross-test comparability remain unsolved problems. She raises doubts about the emerging trend of 'embedded evaluators' — third parties placed inside AI labs to assess safety — noting that qualifications for evaluators lack standardization, as illustrated by backlash over Anthropic hiring Accenture as its first embedded evaluator. She also points out that evaluations are shifting from testing models alone to testing the full system, including harnesses and agents, which are harder and more expensive to evaluate but increasingly where users get value. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What are embedded AI evaluators and why are labs like Anthropic adopting them? Embedded evaluators are third-party observers placed inside AI labs to independently assess model safety, proposed as an alternative to government regulation. Anthropic hired Accenture as its first embedded evaluator, a move that drew backlash from critics questioning whether a consulting firm has the qualifications to evaluate AI safety effectively. _Teams weighing third-party safety evaluators over in-house review can track how these debates unfold on daily.dev._ ### Why is reproducibility a problem in AI model benchmarking? Reproducibility is difficult because evaluators often lack sufficient access to the model itself, and open-weight models make reproducing results far easier than closed ones. Hugging Face's chief policy officer Irene Solaiman notes benchmarks are an evolving science where there's no agreed way to meaningfully compare results across different evaluations, making trust in any single benchmark score shaky. _Anyone comparing model benchmarks before picking a tool can follow this reproducibility debate on daily.dev._ ### Why are agentic evaluations harder than evaluating a model alone? Agentic evaluations are expensive and difficult because they must assess the entire system around a model, including the harness used to access it and the agent built on top, not just the model itself. A weaker model paired with a better harness can outperform a frontier model, yet far fewer evaluations exist for harnesses and holistic agents than for standalone models. _Developers building or choosing agent harnesses can keep up with evolving evaluation standards on daily.dev._ ## Similar posts on daily.dev - [What Actually Mattered in AI — August 2026](https://daily.dev/posts/what-actually-mattered-in-ai-august-2026-ybkm1vfpo) · Medium · 1 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#data-science](https://daily.dev/tags/data-science), [#anthropic](https://daily.dev/tags/anthropic), [#ai-safety](https://daily.dev/tags/ai-safety) [View this post on daily.dev](https://daily.dev/posts/embedded-ai-evaluators-won-t-be-an-easy-solution-hugging-face-s-policy-chief-says-nhkjvfxdt) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Embedded AI evaluators won’t be an easy solution, Hugging Face’s policy chief says","url":"https://daily.dev/posts/embedded-ai-evaluators-won-t-be-an-easy-solution-hugging-face-s-policy-chief-says-nhkjvfxdt","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/embedded-ai-evaluators-won-t-be-an-easy-solution-hugging-face-s-policy-chief-says-nhkjvfxdt"},"datePublished":"2026-10-19T22:52:53.165Z","dateModified":"2026-10-20T01:13:04.159Z","description":"Hugging Face's chief policy officer Irene Solaiman discusses the limits of AI model benchmarking at TechCrunch Disrupt, arguing that reproducibility and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ae687465478fc48ad9761eac86df9c1f?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ae687465478fc48ad9761eac86df9c1f?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"TechCrunch","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"TechCrunch","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/tc","url":"https://daily.dev/sources/tc"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/embedded-ai-evaluators-won-t-be-an-easy-solution-hugging-face-s-policy-chief-says-nhkjvfxdt","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,data-science,anthropic,ai-safety","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"TechCrunch","item":"https://daily.dev/sources/tc"},{"@type":"ListItem","position":3,"name":"Embedded AI evaluators won’t be an easy solution, Hugging Face’s policy chief says"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/embedded-ai-evaluators-won-t-be-an-easy-solution-hugging-face-s-policy-chief-says-nhkjvfxdt#faq","mainEntity":[{"@type":"Question","name":"What are embedded AI evaluators and why are labs like Anthropic adopting them?","acceptedAnswer":{"@type":"Answer","text":"Embedded evaluators are third-party observers placed inside AI labs to independently assess model safety, proposed as an alternative to government regulation. Anthropic hired Accenture as its first embedded evaluator, a move that drew backlash from critics questioning whether a consulting firm has the qualifications to evaluate AI safety effectively. Teams weighing third-party safety evaluators over in-house review can track how these debates unfold on daily.dev."}},{"@type":"Question","name":"Why is reproducibility a problem in AI model benchmarking?","acceptedAnswer":{"@type":"Answer","text":"Reproducibility is difficult because evaluators often lack sufficient access to the model itself, and open-weight models make reproducing results far easier than closed ones. Hugging Face's chief policy officer Irene Solaiman notes benchmarks are an evolving science where there's no agreed way to meaningfully compare results across different evaluations, making trust in any single benchmark score shaky. Anyone comparing model benchmarks before picking a tool can follow this reproducibility debate on daily.dev."}},{"@type":"Question","name":"Why are agentic evaluations harder than evaluating a model alone?","acceptedAnswer":{"@type":"Answer","text":"Agentic evaluations are expensive and difficult because they must assess the entire system around a model, including the harness used to access it and the agent built on top, not just the model itself. A weaker model paired with a better harness can outperform a frontier model, yet far fewer evaluations exist for harnesses and holistic agents than for standalone models. Developers building or choosing agent harnesses can keep up with evolving evaluation standards on daily.dev."}}]} ```