--- title: State of Data — Sean Cai, Independent / State of Data description: A conference talk by an independent analyst exposing the inner workings of AI training data markets. Key insights include: the shift from bulk annotation to... canonical: https://daily.dev/posts/state-of-data-sean-cai-independent-state-of-data-0874tisa1 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: State of Data — Sean Cai, Independent / State of Data | daily.dev og:description: A conference talk by an independent analyst exposing the inner workings of AI training data markets. Key insights include: the shift from bulk annotation to... og:url: https://daily.dev/posts/state-of-data-sean-cai-independent-state-of-data-0874tisa1 og:image: https://api.daily.dev/og/posts/0874TiSa1.png og:image:alt: State of Data — Sean Cai, Independent / State of Data og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # State of Data — Sean Cai, Independent / State of Data **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 18 min read · 1 upvotes · 1 comments ## Summary A conference talk by an independent analyst exposing the inner workings of AI training data markets. Key insights include: the shift from bulk annotation to high-quality 'type one' (real-world process-based) data vs 'type two' (contrived expert-generated) data; why most industry benchmarks are quietly gamed via Goodhart's Law; a framework for predicting which AI application domains mature next using three verifiability axes (asymmetry, veracity, proliferation); why coding AI succeeded first due to GitHub's properties; and why successful data companies are pivoting to enterprise RL-as-a-service rather than staying pure data vendors. The speaker also announces a new venture building Antikythera-style mechanisms to help enterprises monetize data assets and implement RL pipelines. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is Jason Wei's Verifier's Law and why does it explain which AI application areas mature first? Verifier's Law states that the ease of training a model to do a task is proportional to how verifiable that task is, broken into three axes: asymmetry (how easily a task decomposes into checkable steps), veracity (consensus on what correctness means), and proliferation (how often real-world verified examples appear). Coding matured first because GitHub provided unit tests, community consensus, and infinite commit-message examples satisfying all three axes at once. _Developers weighing which AI application domains will mature next can track this reasoning on daily.dev._ ### What is the difference between type one and type two training data for AI models? Type one data is a pure capture of real workflows, such as GitHub commits or session replays, with minimal reward shaping from non-experts, while type two is contrived data manufactured by hired experts placed in artificial settings. Type two helps bootstrap weak models, but type one data is what pushes model performance from roughly 20% to 80% because its realism comes from the actual work rather than a manufactured exercise. _Teams sourcing or evaluating training data can follow this type one versus type two distinction on daily.dev._ ### Why do public AI benchmarks often fail to reflect real model performance differences? Many public benchmarks are built by hiring domain experts who generate plausible tasks with a chatbot, cherry-pick cases where a model diverges, package them as a benchmark, then sell data back to labs to hill-climb that same benchmark, creating a Goodhart's law dynamic. A single benchmark score under one scaffold is just one noisy sample; testing three real finance tasks showed Opus 4.8 scoring worse than 4.7 on rubric analysis while GPT-5.5 and Opus 4.8 diverged in opposite directions (arithmetic versus methodology) despite scoring within three points of each other. _Anyone comparing model releases before committing to one can track benchmark reliability debates on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@agustinbarrientos** · 0 upvotes > I'd report methodology and arithmetic failures separately, even with nearly identical leaderboard scores. ## Similar posts on daily.dev - [AI success starts with clean data, not just better models](https://daily.dev/posts/ai-success-starts-with-clean-data-not-just-better-models-iv2a1lnae) · databricks · 0 upvotes · 0 comments - [The New AI Problem Is a Lack of New Data](https://daily.dev/posts/the-new-ai-problem-is-a-lack-of-new-data-4uhgaxf7l) · Kilo Blog · 1 upvotes · 0 comments - [The next AI breakthrough won’t come from bigger models, but from better data](https://daily.dev/posts/the-next-ai-breakthrough-won-t-come-from-bigger-models-but-from-better-data-8ruccmjsi) · InfoWorld · 1 upvotes · 1 comments - [Predictions for how companies will manage data in 2026](https://daily.dev/posts/predictions-for-how-companies-will-manage-data-in-2026-xag0wvfpb) · SD Times · 0 upvotes · 0 comments - [A Turing Award winner says the industry’s fix for running out of data is ‘a big mistake’](https://daily.dev/posts/a-turing-award-winner-says-the-industry-s-fix-for-running-out-of-data-is-a-big-mistake--a5ss4ktbt) · The Next Web · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#llm](https://daily.dev/tags/llm), [#reinforcement-learning](https://daily.dev/tags/reinforcement-learning) [View this post on daily.dev](https://daily.dev/posts/state-of-data-sean-cai-independent-state-of-data-0874tisa1) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"State of Data — Sean Cai, Independent / State of Data","url":"https://daily.dev/posts/state-of-data-sean-cai-independent-state-of-data-0874tisa1","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/state-of-data-sean-cai-independent-state-of-data-0874tisa1"},"datePublished":"2026-07-26T17:08:26.039Z","dateModified":"2026-09-13T20:18:45.254Z","description":"A conference talk by an independent analyst exposing the inner workings of AI training data markets. Key insights include: the shift from bulk annotation to...","image":"https://i.ytimg.com/vi/ZyIoTOAbRfs/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/ZyIoTOAbRfs/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/state-of-data-sean-cai-independent-state-of-data-0874tisa1","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"machine-learning,llm,reinforcement-learning","timeRequired":"PT18M","video":{"@type":"VideoObject","name":"State of Data — Sean Cai, Independent / State of Data","description":"A conference talk by an independent analyst exposing the inner workings of AI training data markets. Key insights include: the shift from bulk annotation to...","thumbnailUrl":"https://i.ytimg.com/vi/ZyIoTOAbRfs/sddefault.jpg","uploadDate":"2026-07-26T17:08:26.039Z","duration":"PT18M","url":"https://api.daily.dev/r/0874TiSa1","embedUrl":"https://www.youtube.com/embed/ZyIoTOAbRfs"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"State of Data — Sean Cai, Independent / State of Data"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/state-of-data-sean-cai-independent-state-of-data-0874tisa1","comment":[{"@type":"Comment","text":"I’d report methodology and arithmetic failures separately, even with nearly identical leaderboard scores.","datePublished":"2026-08-02T21:20:50.236Z","url":"https://daily.dev/posts/0874TiSa1#c-IlDAt1I4x","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/state-of-data-sean-cai-independent-state-of-data-0874tisa1#faq","mainEntity":[{"@type":"Question","name":"What is Jason Wei's Verifier's Law and why does it explain which AI application areas mature first?","acceptedAnswer":{"@type":"Answer","text":"Verifier's Law states that the ease of training a model to do a task is proportional to how verifiable that task is, broken into three axes: asymmetry (how easily a task decomposes into checkable steps), veracity (consensus on what correctness means), and proliferation (how often real-world verified examples appear). Coding matured first because GitHub provided unit tests, community consensus, and infinite commit-message examples satisfying all three axes at once. Developers weighing which AI application domains will mature next can track this reasoning on daily.dev."}},{"@type":"Question","name":"What is the difference between type one and type two training data for AI models?","acceptedAnswer":{"@type":"Answer","text":"Type one data is a pure capture of real workflows, such as GitHub commits or session replays, with minimal reward shaping from non-experts, while type two is contrived data manufactured by hired experts placed in artificial settings. Type two helps bootstrap weak models, but type one data is what pushes model performance from roughly 20% to 80% because its realism comes from the actual work rather than a manufactured exercise. Teams sourcing or evaluating training data can follow this type one versus type two distinction on daily.dev."}},{"@type":"Question","name":"Why do public AI benchmarks often fail to reflect real model performance differences?","acceptedAnswer":{"@type":"Answer","text":"Many public benchmarks are built by hiring domain experts who generate plausible tasks with a chatbot, cherry-pick cases where a model diverges, package them as a benchmark, then sell data back to labs to hill-climb that same benchmark, creating a Goodhart's law dynamic. A single benchmark score under one scaffold is just one noisy sample; testing three real finance tasks showed Opus 4.8 scoring worse than 4.7 on rubric analysis while GPT-5.5 and Opus 4.8 diverged in opposite directions (arithmetic versus methodology) despite scoring within three points of each other. Anyone comparing model releases before committing to one can track benchmark reliability debates on daily.dev."}}]} ```