A Turing Award winner says the industry’s fix for running out of data is ‘a big mistake’
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Turing Award winner Richard Sutton calls the AI industry's shift to synthetic data 'a big mistake,' arguing on Sequoia's Training Data podcast that no simulation can capture other people's minds or the infinite complexity of the physical world. He favors 'experiential data' gathered by agents acting in their environment instead, building on his and David Silver's 2025 'Era of Experience' paper. The piece contrasts this with the industry's practical reality: Epoch AI projects public human text will be exhausted between 2026 and 2032, pushing labs like Microsoft (Phi-4, ~400B synthetic tokens) and Nvidia (Nemotron, ~10T synthetic tokens) toward synthetic data at scale. It also notes a 2024 Nature paper on model collapse and a rebuttal showing collapse only occurs when synthetic data replaces rather than accumulates alongside human data, plus Andrej Karpathy's counter-view that LLMs are legitimate 'ghosts' distilled from human writing.
Questions this post answers
Why does Richard Sutton think synthetic data is a mistake for training AI models?
Richard Sutton argues synthetic data cannot substitute for other people's minds or the physical world, since any simulation is a lossy, 'microscopic' compression of an infinitely complex reality (his 'big world hypothesis'). He also objects that humans must decide what synthetic data to generate, reintroducing the human judgment his 2019 essay The Bitter Lesson warned against. He instead advocates experiential data gathered by agents acting in their environment. Developers weighing synthetic versus real training data can follow debates like this one on daily.dev.
Does training AI models on synthetic data cause model collapse?
Not necessarily. A 2024 Nature paper by Ilia Shumailov and colleagues found models trained recursively on their own output degrade, a phenomenon called model collapse. However, work by Matthias Gerstgrasser, Rylan Schaeffer, and co-authors found collapse only occurs when synthetic data replaces human data rather than accumulating alongside it. Track evolving research on model collapse and data strategy on daily.dev before betting a training pipeline on it.
How much synthetic data have Microsoft and Nvidia used to train their AI models?
Microsoft's Phi-4 was trained on roughly 400 billion synthetic tokens spanning 50 dataset types, while Nvidia released a synthetic pre-training corpus of about 10 trillion tokens for its Nemotron models. These figures reflect how far labs have already moved past debating synthetic data in favor of using it at scale, particularly in domains like math and code with checkable answers. Compare how leading labs actually use synthetic data by following these releases on daily.dev.