--- title: DeepSeek V4 Flash Found the Answer. The Hard Part Was... description: An open-source framework called LLM-as-a-Verifier, developed by researchers at Stanford, UC Berkeley, and NVIDIA Research, tackles the problem of selecting the... canonical: https://daily.dev/posts/deepseek-v4-flash-found-the-answer-the-hard-part-was-picking-it--jzbmrxlbl twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: DeepSeek V4 Flash Found the Answer. The Hard Part Was Picking It. | daily.dev og:description: An open-source framework called LLM-as-a-Verifier, developed by researchers at Stanford, UC Berkeley, and NVIDIA Research, tackles the problem of selecting the... og:url: https://daily.dev/posts/deepseek-v4-flash-found-the-answer-the-hard-part-was-picking-it--jzbmrxlbl og:image: https://api.daily.dev/og/posts/jzbMrxLBL.png og:image:alt: DeepSeek V4 Flash Found the Answer. The Hard Part Was Picking It. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # DeepSeek V4 Flash Found the Answer. The Hard Part Was Picking It. **[Medium](https://daily.dev/sources/medium_js)** · 4 min read · 0 upvotes · 0 comments ## Summary An open-source framework called LLM-as-a-Verifier, developed by researchers at Stanford, UC Berkeley, and NVIDIA Research, tackles the problem of selecting the best output among multiple LLM-generated attempts. Rather than assigning discrete scores, it examines the probability distribution behind scoring tokens to create finer-grained, continuous verification scores. Using DeepSeek V4 Flash on Terminal-Bench 2.1 with mini-swe-agent trajectories, Best-of-3 selection reached 86.5% versus 79.4% Pass@1, and Best-of-5 reached 88.0% versus 78.7% Pass@1, with Oracle scores reaching 92.1% and 96.6%. The framework also includes a Probabilistic Pivot Tournament for efficient ranking, prefix-cache optimization, and progress-tracking tools that could serve as a dense reward signal for reinforcement learning or a runtime control mechanism for agents. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is LLM-as-a-Verifier and how does it improve Best-of-N selection for AI agents? LLM-as-a-Verifier is an open-source framework from Stanford, UC Berkeley, and NVIDIA Research that uses a model to inspect and score its own multiple attempts rather than picking randomly or by a coarse discrete score. Instead of a single score token like 4 out of 5, it examines the probability distribution over neighboring score tokens to produce a continuous, finer-grained verification signal for ranking candidate outputs. _Anyone building agent pipelines with self-verification can follow developments like this on daily.dev._ ### How much does self-verification improve DeepSeek V4 Flash's performance on Terminal-Bench 2.1? Using DeepSeek V4 Flash to generate multiple mini-swe-agent trajectories and verify them with the same model, Best-of-3 selection reached 86.5% accuracy compared to 79.4% for Pass@1, and Best-of-5 reached 88.0% versus 78.7% Pass@1. Oracle scores, representing the theoretical ceiling if the best trajectory were always chosen, reached 92.1% and 96.6% respectively. _Developers comparing agent benchmarking approaches can track results like these on daily.dev._ ### What is a Probabilistic Pivot Tournament used for in LLM agent evaluation? A Probabilistic Pivot Tournament is a ranking method included in the LLM-as-a-Verifier repository that ranks multiple candidate agent trajectories without requiring every possible pairwise comparison, reducing evaluation cost. It pairs with prefix-cache optimization for long trajectories and progress-tracking tools that can score an agent's work as it unfolds, rather than only at completion. _Those optimizing agent evaluation pipelines can keep up with methods like this via daily.dev._ ## Similar posts on daily.dev - [DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0](https://daily.dev/posts/deepseek-v4-1-flash-on-vllm-5x-agentic-throughput-since-day-0-js5lna7tm) · vLLM · 0 upvotes · 0 comments - [Why not use DeepSeek Flash for everything? · Los Techies](https://daily.dev/posts/why-not-use-deepseek-flash-for-everything-los-techies-sq77ike6a) · LosTechies · 1 upvotes · 1 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#reinforcement-learning](https://daily.dev/tags/reinforcement-learning), [#deepseek](https://daily.dev/tags/deepseek) [View this post on daily.dev](https://daily.dev/posts/deepseek-v4-flash-found-the-answer-the-hard-part-was-picking-it--jzbmrxlbl) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"DeepSeek V4 Flash Found the Answer. The Hard Part Was Picking It.","url":"https://daily.dev/posts/deepseek-v4-flash-found-the-answer-the-hard-part-was-picking-it--jzbmrxlbl","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/deepseek-v4-flash-found-the-answer-the-hard-part-was-picking-it--jzbmrxlbl"},"datePublished":"2026-08-24T09:17:39.498Z","dateModified":"2026-08-24T09:34:03.921Z","description":"An open-source framework called LLM-as-a-Verifier, developed by researchers at Stanford, UC Berkeley, and NVIDIA Research, tackles the problem of selecting the...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c20a5fb58aea6d8e6c0f7a8cd6a04d23?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c20a5fb58aea6d8e6c0f7a8cd6a04d23?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/deepseek-v4-flash-found-the-answer-the-hard-part-was-picking-it--jzbmrxlbl","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,reinforcement-learning,deepseek","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"DeepSeek V4 Flash Found the Answer. The Hard Part Was Picking It."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/deepseek-v4-flash-found-the-answer-the-hard-part-was-picking-it--jzbmrxlbl#faq","mainEntity":[{"@type":"Question","name":"What is LLM-as-a-Verifier and how does it improve Best-of-N selection for AI agents?","acceptedAnswer":{"@type":"Answer","text":"LLM-as-a-Verifier is an open-source framework from Stanford, UC Berkeley, and NVIDIA Research that uses a model to inspect and score its own multiple attempts rather than picking randomly or by a coarse discrete score. Instead of a single score token like 4 out of 5, it examines the probability distribution over neighboring score tokens to produce a continuous, finer-grained verification signal for ranking candidate outputs. Anyone building agent pipelines with self-verification can follow developments like this on daily.dev."}},{"@type":"Question","name":"How much does self-verification improve DeepSeek V4 Flash's performance on Terminal-Bench 2.1?","acceptedAnswer":{"@type":"Answer","text":"Using DeepSeek V4 Flash to generate multiple mini-swe-agent trajectories and verify them with the same model, Best-of-3 selection reached 86.5% accuracy compared to 79.4% for Pass@1, and Best-of-5 reached 88.0% versus 78.7% Pass@1. Oracle scores, representing the theoretical ceiling if the best trajectory were always chosen, reached 92.1% and 96.6% respectively. Developers comparing agent benchmarking approaches can track results like these on daily.dev."}},{"@type":"Question","name":"What is a Probabilistic Pivot Tournament used for in LLM agent evaluation?","acceptedAnswer":{"@type":"Answer","text":"A Probabilistic Pivot Tournament is a ranking method included in the LLM-as-a-Verifier repository that ranks multiple candidate agent trajectories without requiring every possible pairwise comparison, reducing evaluation cost. It pairs with prefix-cache optimization for long trajectories and progress-tracking tools that can score an agent's work as it unfolds, rather than only at completion. Those optimizing agent evaluation pipelines can keep up with methods like this via daily.dev."}}]} ```