--- title: Attention Is Not What You Need (2025) | daily.dev description: Researchers propose an attention-free sequence model that replaces self-attention with Grassmann manifold geometry. The Causal Grassmann layer reduces token... canonical: https://daily.dev/posts/attention-is-not-what-you-need-2025--fee84nx9w twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Attention Is Not What You Need (2025) | daily.dev og:description: Researchers propose an attention-free sequence model that replaces self-attention with Grassmann manifold geometry. The Causal Grassmann layer reduces token... og:url: https://daily.dev/posts/attention-is-not-what-you-need-2025--fee84nx9w og:image: https://api.daily.dev/og/posts/feE84Nx9w.png og:image:alt: Attention Is Not What You Need (2025) og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Attention Is Not What You Need (2025) **[Hacker News](https://daily.dev/sources/hn)** · 3 min read · 1 upvotes · 0 comments ## Summary Researchers propose an attention-free sequence model that replaces self-attention with Grassmann manifold geometry. The Causal Grassmann layer reduces token states linearly, encodes local token pairs as two-dimensional subspaces using Plucker coordinates, and fuses them back via gated mixing. On Wikitext-2, Grassmann-based models with 13 to 18 million parameters land within about 10 to 15 percent of size-matched Transformers' validation perplexity. On SNLI, a Grassmann-Plucker head on DistilBERT slightly beats a Transformer head, scoring 0.8550/0.8538 validation/test accuracy versus 0.8545/0.8511. The authors show the method scales linearly with sequence length for fixed rank. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Can a model avoid self-attention entirely and still match Transformer performance on language modeling? Yes, within a margin. A Causal Grassmann layer that encodes token pairs as subspaces on a Grassmann manifold via Plucker coordinates, instead of forming an attention matrix, achieves validation perplexities on Wikitext-2 within about 10 to 15 percent of size-matched Transformers, using models with 13 to 18 million parameters. _Developers tracking alternatives to attention-based architectures can follow emerging research like this on daily.dev._ ### How does a Grassmann-Plucker head compare to a Transformer head on SNLI natural language inference accuracy? It slightly outperforms it. A Grassmann-Plucker head on top of DistilBERT reached 0.8550 validation and 0.8538 test accuracy, compared to 0.8545 and 0.8511 for a standard Transformer head on the same SNLI task. _Those evaluating new NLI architectures can track benchmark comparisons like this one on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 5 discussions and 61 comments across hackernews (as of 2026-10-26). **TL;DR:** Reaction is largely skeptical: the proposed Grassmann layer underperforms size-matched Transformers on a tiny toy benchmark, with pointed doubts about a possible causality bug in the pairing/gating math and suspicions the paper itself was AI-written; some see value in the broader direction of attention-free/linear architectures but find this specific paper unconvincing. **Sentiment:** 10% positive · 30% mixed · 60% skeptical **The case for** - Some see the general direction of geometric/linear alternatives to attention as promising, citing production models that already mix in linear attention layers. - A few note the authors are at least transparent that the design is deliberately minimal and meant as a starting point for future work. **The pushback** - Independent reproduction attempts disagree on whether the token pairing direction (past vs future token) matches the paper, with one person flagging a potential future-token leakage bug in the gating computation. - Results are only shown on a tiny 13-18M parameter model and an old dataset (Wikitext-2), which several argue proves little about scaling. - The model is reported as 10-15% worse perplexity than Transformers, which many feel undercuts the paper's own title/claim that attention isn't needed. - Multiple commenters suspect the writeup itself was largely LLM-generated and lacking human rigor, figures, or clear metric definitions. - The paper lacks a public code link, forcing the community to rely on third-party reproduction repos that give conflicting results. **By community** - hackernews (skeptical): Discussion skews critical, questioning the benchmark's scale, a possible causality bug in the method, and whether the paper was AI-generated, alongside broader tangents on scaling laws and peer review. **Hottest debate:** Whether the causal pairing/gating math actually leaks future tokens into the current step, which would undermine the reported results entirely. **Open questions** - Does the Grassmann layer add extra compute (MACs) that offsets its lower-rank representation benefits at inference time? - How much more training data would this architecture need compared to standard attention given its narrower information bottleneck? - Which token-pairing direction (past-relative vs future-relative) is actually implemented correctly in the authors' original code versus the paper's description? **Highlights** > The author describes the model's pairing operation as: (zt,zt+Delta) where Delta is a positive offset. This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating. I don't know if that's just the paper, or if the code was also b0rk3d. > — [mapontosevenths on hackernews · 1 comments](https://news.ycombinator.com/item?id=50239237) > > So z_t is a token in the input, and z_t+Δ is some later token. I think you get what I'm saying, but when they compute the gate they do: h_t_new = alpha_t * h_t + (1 - alpha_t) * g_t If I'm reading it right g_t = geometric_mix(z_t, z_(t+delta)) Which would mean that g_t includes t+Δ, which is a future token. It's leaking future tokens into the current math. So it could just be a sign error and still be right in the code, or I could just be getting it wrong. If I'm not though, and given that they got worse results than a standard transformer I really hope it's fixed in the code. If they do worse even when knowing the future it's bad news for the idea. > — [mapontosevenths on hackernews · 1 comments](https://news.ycombinator.com/item?id=50239875) > Fair point. If only the paper had a link to the code. I found this[1] repo which tries to reproduce it, and its accompanying blog post[2] uses (z_t, z_t - delta), without further mention. Another repo[3] trying to reproduce it also does the reverse from what I can see, so (current, past). Interestingly the first repo got underwhelming results, while the second repo matched the paper roughly. Not sure if it's related to the Claude-usage or not in the first, though it notes a lot of details were left out of the paper which is true. [1]: https://github.com/Infatoshi/grassmann-flows/ [2]: https://github.com/Infatoshi/grassmann-flows/blob/main/blog.... [3]: https://github.com/kanenorman/grassmann > — [magicalhippo on hackernews · 1 comments](https://news.ycombinator.com/item?id=50240260) > They advertised it as 10% to 15% worse than transformers Not sure what the point of the paper's title is if the resulting model does not match the transformer. > — [imtringued on hackernews · 1 comments](https://news.ycombinator.com/item?id=50241671) > Interesting idea, but it reads like an LLM write up. For me to engage properly with your written work, I need to see human effort. There are too many half-baked ideas out there, and unless you, as the author, thought the idea is worth putting proper effort into communicating, why should I think of this anything other than something someone said in passing? One way you could communicate effort to me, and likely others, rather than this work looking like the output of some LLM, is to use graphs, schematics and figures to support claims. Metrics are extremely misleading, only really useful when we've verified what is being measured is what one assumes is being measured. Metrics on their own are often deeply bullshit, so without any kind of graphs of trends or other important qualities, I assume it's all LLM output, since LLM's will default to not plotting anything while humans tend to go the opposite way. Also, more effort into completeness needs to go in. Perhaps it is in there, but you just report an "accuracy" without defining what that means. The more I look, the more it starts becoming apparent this likely hasn't received any real human over-sight at all. "What does Grassmann mixing actually buy us?" It buys nothing, because it is a concept, Claude. Neat idea, but in my opinion devoid of critical thinking / analysis or the level of rigour required to be suitable for broader dissemination. I'd put effort into making this a more credible body of work ASAP. Slop like this WILL cripple your standing in the field. > — [Grimblewald on hackernews · 2 comments](https://news.ycombinator.com/item?id=50237903) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50237224) · 161 points · 57 comments - [hackernews](https://news.ycombinator.com/item?id=46788310) · 4 points · 0 comments - [hackernews](https://news.ycombinator.com/item?id=46801602) · 3 points · 4 comments - [hackernews](https://news.ycombinator.com/item?id=46479280) · 3 points · 0 comments - [hackernews](https://news.ycombinator.com/item?id=46396103) · 3 points · 0 comments ## Similar posts on daily.dev - [Attention Is Bayesian Inference](https://daily.dev/posts/attention-is-bayesian-inference-y68katwus) · Hacker News · 1 upvotes · 0 comments - [What comes after attention? This startup says it already knows.](https://daily.dev/posts/what-comes-after-attention-this-startup-says-it-already-knows--lk6wmeah7) · The New Stack · 2 upvotes · 1 comments - [Self-Attention: The Innovation That Changed AI](https://daily.dev/posts/self-attention-the-innovation-that-changed-ai-x8tcpysf8) · Medium · 0 upvotes · 0 comments - [These startups are chasing the next big thing in LLMs](https://daily.dev/posts/these-startups-are-chasing-the-next-big-thing-in-llms-g8wfdvibc) · MIT Technology Review · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#nlp](https://daily.dev/tags/nlp) [View this post on daily.dev](https://daily.dev/posts/attention-is-not-what-you-need-2025--fee84nx9w) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Attention Is Not What You Need (2025)","url":"https://daily.dev/posts/attention-is-not-what-you-need-2025--fee84nx9w","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/attention-is-not-what-you-need-2025--fee84nx9w"},"datePublished":"2026-10-26T03:55:13.042Z","dateModified":"2026-10-26T15:22:13.725Z","description":"Researchers propose an attention-free sequence model that replaces self-attention with Grassmann manifold geometry. The Causal Grassmann layer reduces token...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1fed0de190485cbbb37a86fdf656d71b?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1fed0de190485cbbb37a86fdf656d71b?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/attention-is-not-what-you-need-2025--fee84nx9w","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,nlp","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Attention Is Not What You Need (2025)"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/attention-is-not-what-you-need-2025--fee84nx9w#faq","mainEntity":[{"@type":"Question","name":"Can a model avoid self-attention entirely and still match Transformer performance on language modeling?","acceptedAnswer":{"@type":"Answer","text":"Yes, within a margin. A Causal Grassmann layer that encodes token pairs as subspaces on a Grassmann manifold via Plucker coordinates, instead of forming an attention matrix, achieves validation perplexities on Wikitext-2 within about 10 to 15 percent of size-matched Transformers, using models with 13 to 18 million parameters. Developers tracking alternatives to attention-based architectures can follow emerging research like this on daily.dev."}},{"@type":"Question","name":"How does a Grassmann-Plucker head compare to a Transformer head on SNLI natural language inference accuracy?","acceptedAnswer":{"@type":"Answer","text":"It slightly outperforms it. A Grassmann-Plucker head on top of DistilBERT reached 0.8550 validation and 0.8538 test accuracy, compared to 0.8545 and 0.8511 for a standard Transformer head on the same SNLI task. Those evaluating new NLI architectures can track benchmark comparisons like this one on daily.dev."}}]} ```