---
title: "Scaling Laws, Carefully"
url: https://daily.dev/posts/scaling-laws-carefully-wt9mkgtn8
source_url: https://lilianweng.github.io/posts/2026-06-24-scaling-laws
type: article
source: "Lil’Log"
published: 2026-06-25T20:37:52.698Z
updated: 2026-06-25T21:07:30.054Z
tags: ["llm", "deep-learning"]
reading_time: 15
upvotes: 1
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Scaling Laws, Carefully

**[Lil’Log](https://daily.dev/sources/lilianweng)** · 15 min read · 1 upvotes · 0 comments

## Summary

A thorough technical deep-dive into neural scaling laws for large language models, tracing their history from early ML loss predictability work (Amari 1992, Hestness 2017) through the landmark Kaplan et al. and Chinchilla papers. The post explains the core power-law relationships between model size, dataset size, and compute, then reconciles the disagreements between Kaplan and Chinchilla (embedding parameter counting, small-model extrapolation errors). It extends into data-limited regimes, covering how data repetition affects training and how to model effective token counts with exponential decay penalties. A dedicated section on the trickiness of fitting scaling laws in practice highlights how rounding precision, loss averaging, and fit-region selection can dramatically shift predictions. Includes a toy simulation demonstrating these failure modes.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://lilianweng.github.io/posts/2026-06-24-scaling-laws>

## Similar posts on daily.dev

- [How to build AI scaling laws for efficient LLM training and budget maximization](https://daily.dev/posts/how-to-build-ai-scaling-laws-for-efficient-llm-training-and-budget-maximization-mrur18unx) · MIT News · 0 upvotes · 0 comments

---

Tags: [#llm](https://daily.dev/tags/llm), [#deep-learning](https://daily.dev/tags/deep-learning)

[View this post on daily.dev](https://daily.dev/posts/scaling-laws-carefully-wt9mkgtn8)