A thorough technical deep-dive into neural scaling laws for large language models, tracing their history from early ML loss predictability work (Amari 1992, Hestness 2017) through the landmark Kaplan et al. and Chinchilla papers. The post explains the core power-law relationships between model size, dataset size, and compute, then reconciles the disagreements between Kaplan and Chinchilla (embedding parameter counting, small-model extrapolation errors). It extends into data-limited regimes, covering how data repetition affects training and how to model effective token counts with exponential decay penalties. A dedicated section on the trickiness of fitting scaling laws in practice highlights how rounding precision, loss averaging, and fit-region selection can dramatically shift predictions. Includes a toy simulation demonstrating these failure modes.
