--- title: Can we trust LLMs to judge whether a research idea is novel? description: Evaluating whether a research idea is novel using an LLM judge is unreliable, especially for AI-generated ideas. Ai2 tested six judges, including GPT-5.1... canonical: https://daily.dev/posts/can-we-trust-llms-to-judge-whether-a-research-idea-is-novel--mzgzldn75 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Can we trust LLMs to judge whether a research idea is novel? | daily.dev og:description: Evaluating whether a research idea is novel using an LLM judge is unreliable, especially for AI-generated ideas. Ai2 tested six judges, including GPT-5.1... og:url: https://daily.dev/posts/can-we-trust-llms-to-judge-whether-a-research-idea-is-novel--mzgzldn75 og:image: https://api.daily.dev/og/posts/mzGzldn75.png og:image:alt: Can we trust LLMs to judge whether a research idea is novel? og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Can we trust LLMs to judge whether a research idea is novel? **[Hugging Face](https://daily.dev/sources/huggingface)** · 9 min read · 0 upvotes · 0 comments ## Summary Evaluating whether a research idea is novel using an LLM judge is unreliable, especially for AI-generated ideas. Ai2 tested six judges, including GPT-5.1 through GPT-5.4 and Claude Sonnet/Opus models, across prompt, reasoning budget, and retrieval variations. Changing the prompt framing shifted GPT-5.4 accuracy on identifying the more novel idea from 93.5% to 40.9%. More reasoning and retrieval did not help: an agentic GPT-5.6 Sol setup scored 0.78 macro-F1, 11 points behind a simple Claude Opus 4.6 prompt at 0.89, while costing 10 times more. Purpose-built evaluators like Idea Novelty Checker and Sakana AI's AI Scientist also scored lower than the simple baseline despite costing far more. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How reliable are LLMs like GPT-5.4 or Claude Opus 4.6 at judging whether a research idea is novel? They are unreliable, especially when judging AI-generated ideas. In one test, GPT-5.4's accuracy at picking the more novel of two ideas dropped from 93.5% to 40.9%, below chance, after a prompt was simplified, with verdicts flipping on more than half the same pairs. Human-authored idea comparisons were more stable but still shifted by up to 21 points for Claude Opus 4.6. _Anyone building AI research tools can track evolving findings on LLM evaluation reliability via daily.dev._ ### Does giving an AI judge more reasoning budget and live literature search improve its novelty judgments? No, extra reasoning and retrieval did not consistently improve results. An agentic GPT-5.6 Sol setup with extra-high reasoning and live arXiv search scored 0.78 macro-F1, 11 points behind a simple Claude Opus 4.6 baseline at 0.89, while costing roughly 10 times more per decision at current API prices. _Teams weighing reasoning budgets against cost can follow benchmarks like this on daily.dev._ ### Are purpose-built novelty evaluation tools like Sakana AI's AI Scientist more accurate than a simple prompted LLM? No, they scored worse despite costing much more. A simple Claude Opus 4.6 prompt with majority voting across three calls achieved 0.854 macro-F1 at $2.59 total cost, while Idea Novelty Checker scored 0.642 at $30.66 and Sakana AI's AI Scientist scored 0.772 at $82.05. _Developers comparing AI evaluation tools can watch cost-versus-accuracy results like these on daily.dev._ ## Similar posts on daily.dev - [Your AI Judge Is Grading Its Own Homework](https://daily.dev/posts/your-ai-judge-is-grading-its-own-homework-tuqmnph49) · Medium · 1 upvotes · 0 comments - [A Better Way To Build LLM-as-a-Judge Pipelines](https://daily.dev/posts/a-better-way-to-build-llm-as-a-judge-pipelines-59gkauzir) · Daily Dose of Data Science \| Avi Chawla \| Substack · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#claude](https://daily.dev/tags/claude), [#gpt](https://daily.dev/tags/gpt) [View this post on daily.dev](https://daily.dev/posts/can-we-trust-llms-to-judge-whether-a-research-idea-is-novel--mzgzldn75) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Can we trust LLMs to judge whether a research idea is novel?","url":"https://daily.dev/posts/can-we-trust-llms-to-judge-whether-a-research-idea-is-novel--mzgzldn75","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/can-we-trust-llms-to-judge-whether-a-research-idea-is-novel--mzgzldn75"},"datePublished":"2026-10-27T16:13:17.828Z","dateModified":"2026-10-30T16:53:53.514Z","description":"Evaluating whether a research idea is novel using an LLM judge is unreliable, especially for AI-generated ideas. Ai2 tested six judges, including GPT-5.1...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7c765bc36fcc92d852733ff7d1bec0f9?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7c765bc36fcc92d852733ff7d1bec0f9?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hugging Face","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hugging Face","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/f1f55c67d81a4330acf5b90b26b0c8e1","url":"https://daily.dev/sources/huggingface"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/can-we-trust-llms-to-judge-whether-a-research-idea-is-novel--mzgzldn75","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,claude,gpt","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hugging Face","item":"https://daily.dev/sources/huggingface"},{"@type":"ListItem","position":3,"name":"Can we trust LLMs to judge whether a research idea is novel?"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/can-we-trust-llms-to-judge-whether-a-research-idea-is-novel--mzgzldn75#faq","mainEntity":[{"@type":"Question","name":"How reliable are LLMs like GPT-5.4 or Claude Opus 4.6 at judging whether a research idea is novel?","acceptedAnswer":{"@type":"Answer","text":"They are unreliable, especially when judging AI-generated ideas. In one test, GPT-5.4's accuracy at picking the more novel of two ideas dropped from 93.5% to 40.9%, below chance, after a prompt was simplified, with verdicts flipping on more than half the same pairs. Human-authored idea comparisons were more stable but still shifted by up to 21 points for Claude Opus 4.6. Anyone building AI research tools can track evolving findings on LLM evaluation reliability via daily.dev."}},{"@type":"Question","name":"Does giving an AI judge more reasoning budget and live literature search improve its novelty judgments?","acceptedAnswer":{"@type":"Answer","text":"No, extra reasoning and retrieval did not consistently improve results. An agentic GPT-5.6 Sol setup with extra-high reasoning and live arXiv search scored 0.78 macro-F1, 11 points behind a simple Claude Opus 4.6 baseline at 0.89, while costing roughly 10 times more per decision at current API prices. Teams weighing reasoning budgets against cost can follow benchmarks like this on daily.dev."}},{"@type":"Question","name":"Are purpose-built novelty evaluation tools like Sakana AI's AI Scientist more accurate than a simple prompted LLM?","acceptedAnswer":{"@type":"Answer","text":"No, they scored worse despite costing much more. A simple Claude Opus 4.6 prompt with majority voting across three calls achieved 0.854 macro-F1 at $2.59 total cost, while Idea Novelty Checker scored 0.642 at $30.66 and Sakana AI's AI Scientist scored 0.772 at $82.05. Developers comparing AI evaluation tools can watch cost-versus-accuracy results like these on daily.dev."}}]} ```