Claude Shannon's approach to measuring the entropy of English text is explored through his historical experiments. Shannon needed to understand the predictability of English characters to determine its theoretical compressibility. His early statistical methods broke down for longer sequences, so he turned to human subjects — including his wife Betty — to guess letters in a passage, recording how many attempts were needed to arrive at the correct letter. By combining human guessing experiments with short-sequence statistics, Shannon estimated that English, given sufficient context, could be compressed to roughly 1 bit per character. The piece connects this foundational work to modern compression and AI, arguing that achieving compression near this theoretical limit requires engineering intelligence, not just data analysis.

2m watch time
125 Impressions