--- title: Overtraining as the path to human-like AI | daily.dev description: Gwern's 13,000-word essay 'Human-like Neural Nets by Catapulting' proposes that LLMs fail to achieve human-like generalization because they haven't 'grokked'... canonical: https://daily.dev/posts/overtraining-as-the-path-to-human-like-ai-exvvztsf4 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Overtraining as the path to human-like AI | daily.dev og:description: Gwern's 13,000-word essay 'Human-like Neural Nets by Catapulting' proposes that LLMs fail to achieve human-like generalization because they haven't 'grokked'... og:url: https://daily.dev/posts/overtraining-as-the-path-to-human-like-ai-exvvztsf4 og:image: https://api.daily.dev/og/posts/eXVVztsf4.png og:image:alt: Overtraining as the path to human-like AI og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Overtraining as the path to human-like AI **[sean goedecke](https://daily.dev/sources/seangoedecke)** · 11 min read · 14 upvotes · 5 comments ## Summary Gwern's 13,000-word essay 'Human-like Neural Nets by Catapulting' proposes that LLMs fail to achieve human-like generalization because they haven't 'grokked' their training data. Grokking — a phenomenon where continued overtraining past apparent convergence causes a sudden leap in generalization — requires training a very large, over-parameterized model on a relatively small dataset. This is the opposite of what frontier AI labs currently do (training smaller models on massive datasets). Gwern argues that training a ~100-trillion-parameter model on constrained data, at a cost of $3–10B, could force the model to discover deeper generalizations rather than relying on memorization. The author summarizes and evaluates this argument, noting the political and engineering obstacles, and expresses cautious optimism that a major lab should attempt it. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is grokking in machine learning and why does it happen? Grokking is when a model trained far past the point where training loss appears to stall suddenly makes a massive jump in generalization ability, first documented by OpenAI in a 2022 paper on models trained on simple mathematical operations. Early training resembles rote memorization, but continued training under regularization pressure pushes the model toward simpler, more compressed representations, which eventually snap into actually understanding the underlying rule rather than memorizing examples. _daily.dev surfaces deep dives like this for developers tracking how model training techniques evolve._ ### Why do large language models generalize worse than humans despite being trained on massive datasets? Large language models are trained on oceans of data with relatively small parameter counts (frontier models are estimated at a couple trillion parameters, with open models like Kimi-K3 at just under three trillion), which favors memorization and simple pattern matching over deep generalization. One theory holds that true grokking requires the opposite approach: an enormous, over-parameterized model trained repeatedly on a much smaller dataset, forcing it to find deeper generalizations rather than simply memorizing more data. _developers weighing where AI capability gains come from can follow this debate on daily.dev._ ### Has anyone tried training a huge model on a small dataset to trigger grokking at scale? Frontier labs have not attempted this at the scale required, largely because the engineering challenges of training a hundred-trillion-parameter model remain unsolved and the approach is politically risky since training loss would appear to stall for weeks or months before any payoff. Smaller-scale attempts exist: the BabyLM challenge trains models on very small datasets and has run for four years without producing a generalization leap, and a separate experiment with a 9-billion-parameter model on constrained data also failed to show one, though these models may simply be too small to memorize enough to grok. _daily.dev helps engineers stay on top of experimental training approaches like this one before they hit the mainstream._ ## Community discussion Top comments from developers on daily.dev. **@vikyw** · 2 upvotes > I mean can they test the hypothesis first on a smaller model, with smaller dataset, smaller generalization ? **@geekluffy** · 2 upvotes > A 100T parameter model trained on a small dataset is the complete antithesis of the current AI playbook but if that's what it takes to transition from rote memorization to true generalization it’s a $10B gamble a major lab should take **@kartiknvj** · 1 upvotes > The framing that frontier labs do the opposite of grokking (big data, smaller models) is the tension I keep coming back to, because grokking as I understand it needs heavy over-parameterization on a constrained dataset. What I cannot square is how you would even detect the delayed generalization jump at a 100-trillion-parameter scale without an eval suite sensitive enough to separate memorization from real structure. The measurement problem feels like the actual blocker here, not the compute bill. ## Similar posts on daily.dev - [Are We Having the Wrong AI Dreams?](https://daily.dev/posts/are-we-having-the-wrong-ai-dreams--yepeegu0r) · JetBrains · 2 upvotes · 0 comments - [Why hasn't longer-horizon training slowed AI progress?](https://daily.dev/posts/why-hasn-t-longer-horizon-training-slowed-ai-progress--ppsb9bx2w) · sean goedecke · 1 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#llm](https://daily.dev/tags/llm), [#neural-networks](https://daily.dev/tags/neural-networks) [View this post on daily.dev](https://daily.dev/posts/overtraining-as-the-path-to-human-like-ai-exvvztsf4) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Overtraining as the path to human-like AI","url":"https://daily.dev/posts/overtraining-as-the-path-to-human-like-ai-exvvztsf4","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/overtraining-as-the-path-to-human-like-ai-exvvztsf4"},"datePublished":"2026-07-18T00:13:16.733Z","dateModified":"2026-09-14T09:28:20.578Z","description":"Gwern's 13,000-word essay 'Human-like Neural Nets by Catapulting' proposes that LLMs fail to achieve human-like generalization because they haven't 'grokked'...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/479e95ee2af112c350fa09a341f0c630?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/479e95ee2af112c350fa09a341f0c630?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"sean goedecke","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"sean goedecke","logo":"https://media.daily.dev/image/upload/s--WNgiZyDx--/f_auto,q_auto/v1765713419/logos/seangoedecke","url":"https://daily.dev/sources/seangoedecke"},"commentCount":5,"discussionUrl":"https://daily.dev/posts/overtraining-as-the-path-to-human-like-ai-exvvztsf4","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":14},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":5}],"keywords":"machine-learning,llm,neural-networks","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"sean goedecke","item":"https://daily.dev/sources/seangoedecke"},{"@type":"ListItem","position":3,"name":"Overtraining as the path to human-like AI"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/overtraining-as-the-path-to-human-like-ai-exvvztsf4","comment":[{"@type":"Comment","text":"I mean can they test the hypothesis first on a smaller model, with smaller dataset, smaller generalization ?","datePublished":"2026-07-18T17:42:33.123Z","url":"https://daily.dev/posts/eXVVztsf4#c-yZ60nA904","author":{"@type":"Person","name":"Viky","url":"https://daily.dev/vikyw","image":"https://media.daily.dev/image/upload/s--sjD6--px--/f_auto/v1790844228/avatars/avatar_tACgiGBrwLBEozSFs7dQu?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"A 100T parameter model trained on a small dataset is the complete antithesis of the current AI playbook but if that’s what it takes to transition from rote memorization to true generalization it’s a $10B gamble a major lab should take","datePublished":"2026-07-18T09:37:17.771Z","url":"https://daily.dev/posts/eXVVztsf4#c-IeH20BP7O","author":{"@type":"Person","name":"GeekLuffy","url":"https://daily.dev/geekluffy","image":"https://avatars.githubusercontent.com/u/71143694?v=4"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"The framing that frontier labs do the opposite of grokking (big data, smaller models) is the tension I keep coming back to, because grokking as I understand it needs heavy over-parameterization on a constrained dataset. What I cannot square is how you would even detect the delayed generalization jump at a 100-trillion-parameter scale without an eval suite sensitive enough to separate memorization from real structure. The measurement problem feels like the actual blocker here, not the compute bill.","datePublished":"2026-07-20T16:21:42.894Z","url":"https://daily.dev/posts/eXVVztsf4#c-w4vaLSza1","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/overtraining-as-the-path-to-human-like-ai-exvvztsf4#faq","mainEntity":[{"@type":"Question","name":"What is grokking in machine learning and why does it happen?","acceptedAnswer":{"@type":"Answer","text":"Grokking is when a model trained far past the point where training loss appears to stall suddenly makes a massive jump in generalization ability, first documented by OpenAI in a 2022 paper on models trained on simple mathematical operations. Early training resembles rote memorization, but continued training under regularization pressure pushes the model toward simpler, more compressed representations, which eventually snap into actually understanding the underlying rule rather than memorizing examples. daily.dev surfaces deep dives like this for developers tracking how model training techniques evolve."}},{"@type":"Question","name":"Why do large language models generalize worse than humans despite being trained on massive datasets?","acceptedAnswer":{"@type":"Answer","text":"Large language models are trained on oceans of data with relatively small parameter counts (frontier models are estimated at a couple trillion parameters, with open models like Kimi-K3 at just under three trillion), which favors memorization and simple pattern matching over deep generalization. One theory holds that true grokking requires the opposite approach: an enormous, over-parameterized model trained repeatedly on a much smaller dataset, forcing it to find deeper generalizations rather than simply memorizing more data. developers weighing where AI capability gains come from can follow this debate on daily.dev."}},{"@type":"Question","name":"Has anyone tried training a huge model on a small dataset to trigger grokking at scale?","acceptedAnswer":{"@type":"Answer","text":"Frontier labs have not attempted this at the scale required, largely because the engineering challenges of training a hundred-trillion-parameter model remain unsolved and the approach is politically risky since training loss would appear to stall for weeks or months before any payoff. Smaller-scale attempts exist: the BabyLM challenge trains models on very small datasets and has run for four years without producing a generalization leap, and a separate experiment with a 9-billion-parameter model on constrained data also failed to show one, though these models may simply be too small to memorize enough to grok. daily.dev helps engineers stay on top of experimental training approaches like this one before they hit the mainstream."}}]} ```