--- title: We Didn’t Invent Attention — We Just Rediscovered It description: Attention mechanisms in AI transformers aren't novel inventions but rediscoveries of fundamental optimization principles. The same mathematical... canonical: https://daily.dev/posts/we-didn-t-invent-attention-we-just-rediscovered-it-rds8cshsz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: We Didn’t Invent Attention — We Just Rediscovered It | daily.dev og:description: Attention mechanisms in AI transformers aren't novel inventions but rediscoveries of fundamental optimization principles. The same mathematical... og:url: https://daily.dev/posts/we-didn-t-invent-attention-we-just-rediscovered-it-rds8cshsz og:image: https://api.daily.dev/og/posts/Rds8cShsz.png og:image:alt: We Didn’t Invent Attention — We Just Rediscovered It og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # We Didn’t Invent Attention — We Just Rediscovered It **[Towards Data Science](https://daily.dev/sources/tds)** · 10 min read · 58 upvotes · 3 comments ## Summary Attention mechanisms in AI transformers aren't novel inventions but rediscoveries of fundamental optimization principles. The same mathematical pattern—selective amplification combined with normalization—emerges independently across evolution (500+ million years of neural systems), chemistry (autocatalytic reactions), and AI (gradient descent). This convergence suggests attention represents a universal solution to information processing under energy constraints. Reframing attention as amplification rather than selection offers practical insights for improving AI architectures: decoupling amplification from normalization, exploring non-content-based amplification, implementing local normalization pools, and designing systems that operate at critical dynamics for optimal information processing. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community discussion Top comments from developers on daily.dev. **@whlsk34** · 4 upvotes > The author goes searching for commonalities in a wide range of systems. He gives an example of very crude, common notions of attention--namely organisms being attentive in nature--a chemical reaction that requires output training with a tangent on energy efficiency. > > I have no doubt that there are commonalities across all sorts of varied phenomena. None of these yet seem to be interesting. These all can be describe by math, optimization occurs, rates of change are essential, they are all 'things'. The author comes up with another one: they are all emphasizing certain bits of information... **@plzstfunaren** · 1 upvotes > ==D ## Similar posts on daily.dev - [Self-Attention: The Innovation That Changed AI](https://daily.dev/posts/self-attention-the-innovation-that-changed-ai-x8tcpysf8) · Medium · 0 upvotes · 0 comments - [The Paper That Created Modern AI: The Story Behind the Transformer](https://daily.dev/posts/the-paper-that-created-modern-ai-the-story-behind-the-transformer-ottnd8r0s) · freeCodeCamp · 0 upvotes · 0 comments - [Attention Is Bayesian Inference](https://daily.dev/posts/attention-is-bayesian-inference-y68katwus) · Hacker News · 1 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#machine-learning](https://daily.dev/tags/machine-learning), [#deep-learning](https://daily.dev/tags/deep-learning), [#neural-networks](https://daily.dev/tags/neural-networks), [#transformers](https://daily.dev/tags/transformers) [View this post on daily.dev](https://daily.dev/posts/we-didn-t-invent-attention-we-just-rediscovered-it-rds8cshsz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"We Didn’t Invent Attention — We Just Rediscovered It","url":"https://daily.dev/posts/we-didn-t-invent-attention-we-just-rediscovered-it-rds8cshsz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/we-didn-t-invent-attention-we-just-rediscovered-it-rds8cshsz"},"datePublished":"2025-11-05T22:37:59.811Z","dateModified":"2025-11-05T22:38:21.732Z","description":"Attention mechanisms in AI transformers aren't novel inventions but rediscoveries of fundamental optimization principles. The same mathematical...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8213ea137af0bffeb27744544f5bc245?_a=AQAEulh","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8213ea137af0bffeb27744544f5bc245?_a=AQAEulh","isAccessibleForFree":true,"articleSection":"Towards Data Science","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Towards Data Science","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/tds","url":"https://daily.dev/sources/tds"},"commentCount":3,"discussionUrl":"https://daily.dev/posts/we-didn-t-invent-attention-we-just-rediscovered-it-rds8cshsz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":58},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":3}],"keywords":"ai,machine-learning,deep-learning,neural-networks,transformers","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Towards Data Science","item":"https://daily.dev/sources/tds"},{"@type":"ListItem","position":3,"name":"We Didn’t Invent Attention — We Just Rediscovered It"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/we-didn-t-invent-attention-we-just-rediscovered-it-rds8cshsz","comment":[{"@type":"Comment","text":"The author goes searching for commonalities in a wide range of systems. He gives an example of very crude, common notions of attention–namely organisms being attentive in nature–a chemical reaction that requires output training with a tangent on energy efficiency.\nI have no doubt that there are commonalities across all sorts of varied phenomena. None of these yet seem to be interesting. These all can be describe by math, optimization occurs, rates of change are essential, they are all ‘things’. The author comes up with another one: they are all emphasizing certain bits of information as part of the processing (he calls it amplification). I don’t find this all that interesting. Isn’t this inherent to almost any optimization process? LSTMs are also ‘amplifying’ certain neurons even though it’s not doing it in a global sense. This doesn’t even have to do with attention. It seems that the notion is so general and ill-defined that it doesn’t really seem to be all that interesting.","datePublished":"2025-11-14T07:04:12.575Z","url":"https://daily.dev/posts/Rds8cShsz#c-PjsPVOIYi","author":{"@type":"Person","name":"whlsk34","url":"https://daily.dev/whlsk34","image":"https://media.daily.dev/image/upload/s--O0TOmw4y--/f_auto/v1715772965/public/noProfile"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":4}},{"@type":"Comment","text":"==D","datePublished":"2025-12-16T17:39:51.664Z","url":"https://daily.dev/posts/Rds8cShsz#c-tDDcDGKlP","author":{"@type":"Person","name":"Naren Kumar","url":"https://daily.dev/plzstfunaren","image":"https://lh3.googleusercontent.com/a/ACg8ocKBhEc1pYQwik_PkNBJEuPg1AmkAORmjJ2A7DmJA2iFOEjTTA=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}]} ```