--- title: Chomsky and the Two Cultures of Statistical Learning (2011) description: The author argues that Chomsky's long-standing dismissal of probabilistic and statistical models of language is wrong, both as engineering and as science.... canonical: https://daily.dev/posts/chomsky-and-the-two-cultures-of-statistical-learning-2011--r4ekdzuzk twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Chomsky and the Two Cultures of Statistical Learning (2011) | daily.dev og:description: The author argues that Chomsky's long-standing dismissal of probabilistic and statistical models of language is wrong, both as engineering and as science.... og:url: https://daily.dev/posts/chomsky-and-the-two-cultures-of-statistical-learning-2011--r4ekdzuzk og:image: https://api.daily.dev/og/posts/R4ekDzUzK.png og:image:alt: Chomsky and the Two Cultures of Statistical Learning (2011) og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Chomsky and the Two Cultures of Statistical Learning (2011) **[Hacker News](https://daily.dev/sources/hn)** · 27 min read · 0 upvotes · 0 comments ## Summary The author argues that Chomsky's long-standing dismissal of probabilistic and statistical models of language is wrong, both as engineering and as science. Statistical methods, related to Shannon's 1948 work on probabilistic models, now dominate computational linguistics and generate several trillion dollars in revenue each year versus well under a billion from Chomsky's theories. The author shows that a model trained on 1800-1954 Google Book text rates Chomsky's 'colorless green ideas sleep furiously' as about 10,000 times more probable than its reverse, challenging the claim such models can't distinguish them. The author frames the dispute using Leo Breiman's 2001 'Two Cultures' distinction between data modeling and algorithmic modeling. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Does a statistical language model actually distinguish 'colorless green ideas sleep furiously' from its reversed word order? Yes. A finite-state model with word categories trained via expectation maximization on newspaper text computed the original sentence as 200,000 times more probable than the reversed version. A cruder Laplacian-smoothed model with no categories, trained on the Google Books corpus from 1800 to 1954, still found it about 10,000 times more probable, refuting the claim that such models cannot make this distinction. _Anyone debating whether statistical NLP can capture grammaticality can track this kind of evidence on daily.dev._ ### Why did Chomsky claim the notion of 'probability of a sentence' was useless? Chomsky argued in 1969 that under any known interpretation, a novel sentence must have zero probability, yet novel sentences occur constantly, creating a contradiction. This criticism is considered invalid because smoothing techniques in modern probabilistic models routinely assign non-zero probability to novel sentences never seen in training data, resolving the supposed contradiction. _Developers weighing probabilistic versus rule-based NLP approaches can follow this debate on daily.dev._ ### What are the two cultures of statistical modeling described by Leo Breiman? Data modeling culture, followed by an estimated 98% of statisticians, assumes nature follows a relatively simple underlying model whose parameters are estimated from data. Algorithmic modeling culture, used by the other 2% plus many in biology and AI, treats nature as a black box approximated by complex methods like support vector machines or boosted decision trees, with no claim that the resulting function mirrors nature's true form. _Those choosing between interpretable and black-box modeling approaches can revisit this framing on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 75 comments across hackernews (as of 2026-11-06). **TL;DR:** Discussion is split between those who think LLMs vindicate statistical approaches to language and those who argue LLMs' reliance on vastly more data than humans shows they aren't a good scientific model of human language acquisition, with a long tangent about Chomsky's personal ties to Epstein. **Sentiment:** 25% positive · 45% mixed · 30% skeptical **The case for** - Some argue LLMs' impressive output shows statistical/probabilistic approaches can capture real linguistic competence, challenging Chomsky's dismissal. - A few note that generative linguists already incorporate statistical inference, suggesting the dichotomy is overstated. - Some see LLMs as evidence that intelligence and language may be less mechanistically complex or mysterious than previously assumed. **The pushback** - Many argue LLMs require orders of magnitude more data and compute than children to learn language, undermining claims they model human acquisition. - Several compare LLMs to engineering analogies (airplanes vs. birds) to argue practical success says nothing about the underlying biological mechanism. - Some contend that neural network mechanisms (backprop, attention) are fundamentally unlike biological neurons, so similarity of output doesn't imply similarity of process. - A few argue Norvig misunderstands or oversimplifies generative linguistics, making his critique unserious. **By community** - hackernews (heated): A long, contentious thread arguing over whether LLMs validate statistical linguistics or merely reveal engineering success without scientific insight into human cognition, plus a sharp side debate about Chomsky's relationship with Epstein. **Hottest debate:** Whether LLMs' success at generating language constitutes evidence for statistical/probabilistic theories of human language acquisition, or is an engineering feat irrelevant to the science of how humans actually learn language. **Open questions** - Is there a meaningful mechanistic similarity between LLM architectures and human neural processes for language, or only superficial output similarity? - How should the vastly different data/compute requirements between humans and LLMs be fairly compared (text-only vs. all sensory input)? - Can innate linguistic structure and statistical inference coexist as complementary explanations, rather than competing ones? **Highlights** > Norvig (and Chomsky) are talking at cross purposes. Norvig confuses engineering for science. If an engineer wants something that works (and guarantees that it will work) then you use the most effective tool available to achieve that goal. No problem there. But there's absolutely no reason to think that tells you anything scientific about the phenomenon you're modeling. For example, aeronautical engineering says very little about how birds fly. And - currently relevant - LLMs learn to use and understand language, but they use millions of times more training data, and millions of times more computing power, than a child does. So an LLM may be a great technology, but it's probably not a good model for how humans learn and use language. Chomsky OTOH doesn't see any use for statistics in science, at least not in linguistics. Fortunately for the field of linguistics, Chomsky's influence in linguistics is waning and linguists are beginning to use statistical tools to do science. > — [NaiveEmpiricist on hackernews · 5 comments](https://news.ycombinator.com/item?id=50392038) > What reason to we have to believe that LLMs are the closest model we have for how humans learn language. We considered the hypothesis that humans are a general intelligence that figures language out by exposure. The evidence has consistently come back that that is not how humans actually work. A model where humans are hardwired with a lot of knowledge about language, then fill in the gaps based on exposure still appears to be far more accurate. The only thing LLMs show is that it is possible to learn language de-novo. > — [gizmo686 on hackernews · 2 comments](https://news.ycombinator.com/item?id=50392511) > > but from all models, it's the closest Care to back this up? What evidence do we have to suggest that back propagation, self attention, and byte pair encoding are mechanistically similar to human cognition? Similarity of the outputs says nothing about similarity of the machinery underneath. Turboprop engines tell us nothing about how birds fly - doesn't matter that they successfully transport millions of people around the world. I would say that when it comes to higher level cognition, logic engines are more human like even if they only have a tiny fraction of the practical utility of LLMs. > — [Calavar on hackernews · 2 comments](https://news.ycombinator.com/item?id=50392533) > I find it hard to see how Chomsky has a leg to stand on. > I understand how Chomsky might arrive at the conclusion that probabilistic models are unnecessary, from his study of the generation of language. I think the point turns here for me. We have probabilistic models now that generate unbelievably convincing language. Norvig's following argument on the value of interpreting language statistically becomes secondary. In order to stay moored in reality, our models must follow from evidence. I think it would take a peculiar obstinance to deny that the latest models show evidence of generating language. > — [stillpointlab on hackernews · 1 comments](https://news.ycombinator.com/item?id=50395164) > It's so clear that Norvig knows so little about linguistics as a subject, I mean the suggestion that English is "kinda" a pro-drop language like Spanish is just unserious. There's this kind of lazy criticism of generative work where people hear the vague idea of a theory and critique without even seriously reading anything about it. These are facts that generativists are perfectly aware of and have explanations for. It's sort of like criticising Galileo's putative experiment about dropping two balls from the leaning tower of Pisa by pointing out that a feather indeed falls slower than a iron ball. > — [suddenlybananas on hackernews](https://news.ycombinator.com/item?id=50393182) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50389570) · 83 points · 75 comments ## Similar posts on daily.dev - [The Universe of Discourse : Language models imply world models](https://daily.dev/posts/the-universe-of-discourse-language-models-imply-world-models-pixerm7h4) · Planet Haskell · 1 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#nlp](https://daily.dev/tags/nlp) [View this post on daily.dev](https://daily.dev/posts/chomsky-and-the-two-cultures-of-statistical-learning-2011--r4ekdzuzk) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Chomsky and the Two Cultures of Statistical Learning (2011)","url":"https://daily.dev/posts/chomsky-and-the-two-cultures-of-statistical-learning-2011--r4ekdzuzk","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/chomsky-and-the-two-cultures-of-statistical-learning-2011--r4ekdzuzk"},"datePublished":"2026-11-06T17:15:09.462Z","dateModified":"2026-11-06T22:46:21.045Z","description":"The author argues that Chomsky's long-standing dismissal of probabilistic and statistical models of language is wrong, both as engineering and as science....","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c8acab1fb6ca0fdcd5876d7060d21e90?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c8acab1fb6ca0fdcd5876d7060d21e90?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/chomsky-and-the-two-cultures-of-statistical-learning-2011--r4ekdzuzk","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,nlp","timeRequired":"PT27M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Chomsky and the Two Cultures of Statistical Learning (2011)"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/chomsky-and-the-two-cultures-of-statistical-learning-2011--r4ekdzuzk#faq","mainEntity":[{"@type":"Question","name":"Does a statistical language model actually distinguish 'colorless green ideas sleep furiously' from its reversed word order?","acceptedAnswer":{"@type":"Answer","text":"Yes. A finite-state model with word categories trained via expectation maximization on newspaper text computed the original sentence as 200,000 times more probable than the reversed version. A cruder Laplacian-smoothed model with no categories, trained on the Google Books corpus from 1800 to 1954, still found it about 10,000 times more probable, refuting the claim that such models cannot make this distinction. Anyone debating whether statistical NLP can capture grammaticality can track this kind of evidence on daily.dev."}},{"@type":"Question","name":"Why did Chomsky claim the notion of 'probability of a sentence' was useless?","acceptedAnswer":{"@type":"Answer","text":"Chomsky argued in 1969 that under any known interpretation, a novel sentence must have zero probability, yet novel sentences occur constantly, creating a contradiction. This criticism is considered invalid because smoothing techniques in modern probabilistic models routinely assign non-zero probability to novel sentences never seen in training data, resolving the supposed contradiction. Developers weighing probabilistic versus rule-based NLP approaches can follow this debate on daily.dev."}},{"@type":"Question","name":"What are the two cultures of statistical modeling described by Leo Breiman?","acceptedAnswer":{"@type":"Answer","text":"Data modeling culture, followed by an estimated 98% of statisticians, assumes nature follows a relatively simple underlying model whose parameters are estimated from data. Algorithmic modeling culture, used by the other 2% plus many in biology and AI, treats nature as a black box approximated by complex methods like support vector machines or boosted decision trees, with no claim that the resulting function mirrors nature's true form. Those choosing between interpretable and black-box modeling approaches can revisit this framing on daily.dev."}}]} ```