A podcast conversation with Joon Sung Park, creator of the viral 2023 Generative Agents ('Smallville') paper and now CEO of Simile AI, traces his path from AI research to building foundation models of human behavior. Park explains why frontier LLMs, trained mostly on self-reported web data, fail to capture real human behavior and require post-training on interviews, transaction data, and randomized controlled trials rather than just prompting. He details how digital twins of 1,000 people reproduced human behavior 85% as accurately as people reproduced their own responses, versus 20-60% for frontier models alone. The discussion covers population-level vs individual-level modeling, early scaling laws for simulation, data-center-scale ambitions to simulate all 8 billion people, and applications from market research to climate change and democratic stability, alongside Simile's recent $2B Series B backed by GreenOaks and Index Ventures.
Table of contents
We discuss:Joon Sung ParkTimestampsIntroduction: Joon Sung Park, Simile, and the Story So FarSmallville, Generative Agents, and the 2023 Breakout PaperFoundation Models and the Search for Killer ApplicationsThe Time Machine Game and Recreating the WorldPersonal Agents, User Models, and Why Simulation Came FirstMemory, Markdown, and the Limits of PromptingSocial Physics and Behavior Foundation ModelsThe Three Data Buckets: Interviews, Behavior, and CausalityPrediction vs. Simulation: Shaping the FutureHow Customers Use Simile: Populations, Queries, and ExperimentsConcept Testing, Gallup, and PoliticsCounterfactuals, Polling, and When Simulation Is UsefulSimulation as a Path, Not Just a PredictionGrounding and Evaluating Digital Twins85% Accuracy and Why Frontier Models Miss Human BehaviorPost-Training on RCTs and Replication StudiesPopulation-Level vs. Individual-Level ModelsHuman Biases, Mundane Choices, and What Models MissWhat Data Matters: Social Media, Transactions, and FacebookBillion Personas, Synthetic Demographics, and Bespoke DataScaling Simulation: From Thousands to SocietiesClimate Change, Democracy, and Societal SimulationSchelling, Agent-Based Models, and the Nobel PrizeCost, Reuse, and the Economics of SimulationMulti-Agent Simulation and Social InfluenceEfficiency, Enterprise Use, and Real-World Case StudiesProduct Testing, Websites, and Synthetic PanelsMarket Size, TAM, and Human Decision-MakingWhere Simulation Goes NextSimulation as Painting and Understanding Human EssenceUBI, Future Questions, and the Value of SimulationAre We Already in a Simulation?Simile as Research Lab and Product CompanyHiring and ClosingQuestions this post answers
How accurate are Simile's AI digital twins at predicting real human behavior compared to frontier LLMs like GPT or Claude?
Simile's digital twins reproduced human behavior and attitudes 85% as accurately as people reproduced their own responses in a study of 1,000 representatively-sampled US participants. By contrast, frontier models like ChatGPT and Claude alone scored between 20-30% on niche populations and 50-60% on general population tasks, because they're optimized for rational reasoning rather than modeling realistic, biased human decision-making. Developers evaluating simulation or synthetic-persona tools can track benchmarks like this on daily.dev.
Why do frontier LLMs struggle to simulate realistic human behavior even though they're trained on massive web data?
Frontier LLMs are trained largely on self-reported attitudinal data from the web (social media, Wikipedia) rather than actual behavioral data, and they're further optimized to be rational, objective reasoners using data from professional programmers and scientists. This makes them poor at reproducing irrational, biased, or mistake-prone human behavior, which requires post-training on interview data, transaction records, and randomized controlled trials instead of just prompting. Anyone weighing prompting versus fine-tuning for behavior modeling can follow this reasoning on daily.dev.
What kinds of data does Simile AI use to build behavioral foundation models of human populations?
Simile combines three data types: long-form interview data capturing life stories and stated reasons for decisions, observational behavioral data such as transaction records and web-scraped activity, and randomized controlled trial data that isolates causal mechanisms behind decisions. RCT data is considered most valuable because it reveals the causal 'why' behind behavior, which is rare since real-world events happen only once without controlled variables. Teams designing synthetic population or panel-testing pipelines can dig into approaches like this on daily.dev.