<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/eMAFEW4LM" -->

---
title: How the Math Used in 19th-Century Heat Physics Helps...
description: An explainer traces how sine and cosine functions, originally developed by Joseph Fourier for modeling heat diffusion, became the mathematical basis for...
canonical: https://daily.dev/posts/how-the-math-used-in-19th-century-heat-physics-helps-llms-understand-sentences-emafew4lm
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: How the Math Used in 19th-Century Heat Physics Helps LLMs Understand Sentences | daily.dev
og:description: An explainer traces how sine and cosine functions, originally developed by Joseph Fourier for modeling heat diffusion, became the mathematical basis for...
og:url: https://daily.dev/posts/how-the-math-used-in-19th-century-heat-physics-helps-llms-understand-sentences-emafew4lm
og:image: https://api.daily.dev/og/posts/eMAFEW4LM.png
og:image:alt: How the Math Used in 19th-Century Heat Physics Helps LLMs Understand Sentences
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# How the Math Used in 19th-Century Heat Physics Helps LLMs Understand Sentences

**[ByteByteGo](https://daily.dev/sources/bytebytego)** · 15 min read · 30 upvotes · 0 comments

## Summary

An explainer traces how sine and cosine functions, originally developed by Joseph Fourier for modeling heat diffusion, became the mathematical basis for positional encoding in transformer models. It walks through why basic attention is order-blind, how positional encodings inject sequence information into token embeddings, why multiple rotation frequencies improve positional distinguishability, how the original 2017 transformer adds position vectors to embeddings, and how Rotary Position Embedding (RoPE) instead rotates query/key pairs to encode relative position. It also touches on alternative approaches like learned positional embeddings, ALiBi, and causal masking.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://blog.bytebytego.com/p/how-the-math-used-in-19th-century>

## Questions this post answers

### Why do transformers need positional encoding if attention already sees all the tokens?

Basic attention has no inherent sense of order: it computes relationships between tokens based on their content, so swapping the positions of two words produces the same set of pairwise comparisons unless position is explicitly added. Positional encoding injects location information into each token's representation so word order can affect the model's calculations, since array indexes alone don't automatically become a usable input feature.

_Developers building intuition for transformer internals can find deeper breakdowns like this through daily.dev._

### How does RoPE (Rotary Position Embedding) encode token position differently from the original transformer's positional encoding?

RoPE rotates pairs of components within attention's query and key vectors according to token position, using sine and cosine functions, rather than adding a separate position vector to the token embedding as the original 2017 transformer did. When a rotated query is compared with a rotated key, the resulting signal depends only on their relative rotation, which directly encodes relative position, with different component pairs rotating at different speeds.

_Engineers comparing positional encoding schemes for model design can track explainers like this via daily.dev._

### Why does positional encoding use multiple sine and cosine frequencies instead of a single wave?

A single rotating wave repeats after one full cycle, so two positions separated by that cycle length become indistinguishable, for example positions 0 and 8 if the wave repeats every 8 tokens. Using several waves with different rotation speeds, such as periods of 8, 32, and 128, lets their combined readings distinguish far more positions, since faster components capture short-range differences and slower ones capture long-range differences.

_Anyone digging into why transformers encode position this way can follow similar deep dives on daily.dev._

## Similar posts on daily.dev

- [Understanding Positional Embeddings in Modern LLMs](https://daily.dev/posts/understanding-positional-embeddings-in-modern-llms-415mglzuo) · Sebastian Raschka · 18 upvotes · 1 comments
- [Why Transformers Need Positional Encoding For Time Series: A Visual Guide](https://daily.dev/posts/why-transformers-need-positional-encoding-for-time-series-a-visual-guide-frhbarwuv) · Towards Data Science · 0 upvotes · 0 comments
- [How LLMs Actually Work](https://daily.dev/posts/how-llms-actually-work-wvseu9bao) · Lobsters · 0 upvotes · 0 comments
- [Papers Explained 611: Position Encodings in Transformers](https://daily.dev/posts/papers-explained-611-position-encodings-in-transformers-zh0nssczz) · Medium · 0 upvotes · 0 comments
- [RoPE, Clearly Explained](https://daily.dev/posts/rope-clearly-explained-ar0nzwulr) · Towards Data Science · 1 upvotes · 0 comments

---

Tags: [#llm](https://daily.dev/tags/llm), [#deep-learning](https://daily.dev/tags/deep-learning)

[View this post on daily.dev](https://daily.dev/posts/how-the-math-used-in-19th-century-heat-physics-helps-llms-understand-sentences-emafew4lm)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"How the Math Used in 19th-Century Heat Physics Helps LLMs Understand Sentences","url":"https://daily.dev/posts/how-the-math-used-in-19th-century-heat-physics-helps-llms-understand-sentences-emafew4lm","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-the-math-used-in-19th-century-heat-physics-helps-llms-understand-sentences-emafew4lm"},"datePublished":"2026-10-13T15:33:22.239Z","dateModified":"2026-10-13T16:01:35.443Z","description":"An explainer traces how sine and cosine functions, originally developed by Joseph Fourier for modeling heat diffusion, became the mathematical basis for...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/42a9d74481cb57564689ec5bc805858d?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/42a9d74481cb57564689ec5bc805858d?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ByteByteGo","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ByteByteGo","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/35be29234ee14d01a9cd049c52e12753","url":"https://daily.dev/sources/bytebytego"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-the-math-used-in-19th-century-heat-physics-helps-llms-understand-sentences-emafew4lm","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":30},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,deep-learning","timeRequired":"PT15M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ByteByteGo","item":"https://daily.dev/sources/bytebytego"},{"@type":"ListItem","position":3,"name":"How the Math Used in 19th-Century Heat Physics Helps LLMs Understand Sentences"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-the-math-used-in-19th-century-heat-physics-helps-llms-understand-sentences-emafew4lm#faq","mainEntity":[{"@type":"Question","name":"Why do transformers need positional encoding if attention already sees all the tokens?","acceptedAnswer":{"@type":"Answer","text":"Basic attention has no inherent sense of order: it computes relationships between tokens based on their content, so swapping the positions of two words produces the same set of pairwise comparisons unless position is explicitly added. Positional encoding injects location information into each token's representation so word order can affect the model's calculations, since array indexes alone don't automatically become a usable input feature. Developers building intuition for transformer internals can find deeper breakdowns like this through daily.dev."}},{"@type":"Question","name":"How does RoPE (Rotary Position Embedding) encode token position differently from the original transformer's positional encoding?","acceptedAnswer":{"@type":"Answer","text":"RoPE rotates pairs of components within attention's query and key vectors according to token position, using sine and cosine functions, rather than adding a separate position vector to the token embedding as the original 2017 transformer did. When a rotated query is compared with a rotated key, the resulting signal depends only on their relative rotation, which directly encodes relative position, with different component pairs rotating at different speeds. Engineers comparing positional encoding schemes for model design can track explainers like this via daily.dev."}},{"@type":"Question","name":"Why does positional encoding use multiple sine and cosine frequencies instead of a single wave?","acceptedAnswer":{"@type":"Answer","text":"A single rotating wave repeats after one full cycle, so two positions separated by that cycle length become indistinguishable, for example positions 0 and 8 if the wave repeats every 8 tokens. Using several waves with different rotation speeds, such as periods of 8, 32, and 128, lets their combined readings distinguish far more positions, since faster components capture short-range differences and slower ones capture long-range differences. Anyone digging into why transformers encode position this way can follow similar deep dives on daily.dev."}}]}
```

