--- title: These startups are chasing the next big thing in LLMs description: Transformers, the neural network architecture behind every major LLM, are increasingly showing limitations: quadratic compute costs, constrained context... canonical: https://daily.dev/posts/these-startups-are-chasing-the-next-big-thing-in-llms-g8wfdvibc twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: These startups are chasing the next big thing in LLMs | daily.dev og:description: Transformers, the neural network architecture behind every major LLM, are increasingly showing limitations: quadratic compute costs, constrained context... og:url: https://daily.dev/posts/these-startups-are-chasing-the-next-big-thing-in-llms-g8wfdvibc og:image: https://api.daily.dev/og/posts/g8wFdVibc.png og:image:alt: These startups are chasing the next big thing in LLMs og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # These startups are chasing the next big thing in LLMs **[MIT Technology Review](https://daily.dev/sources/technologyreview)** · 13 min read · 0 upvotes · 0 comments ## Summary Transformers, the neural network architecture behind every major LLM, are increasingly showing limitations: quadratic compute costs, constrained context windows, and poor fit for non-linguistic reasoning. Four startups are betting on alternatives. Subquadratic claims its sparse attention model SubQ rivals mainstream LLMs on search and coding. Manifest AI replaces attention with 'power retention,' a rolling-summary mechanism that scales to massive contexts, demonstrated via PowerCoder and Brumby. Liquid AI pairs transformers (20%) with liquid neural networks (80%) to build tiny, efficient models that run on a Raspberry Pi and have hit 34 million downloads. Inception applies image-diffusion techniques to text, generating whole blocks at once — its Mercury 2 model claims GPT-4-level quality at 10x the speed. Pathway goes furthest, replacing attention with state-space math so models can reason beyond language, scoring above 97% on 250,000 hard sudoku puzzles where leading LLMs failed entirely. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is sparse attention in LLMs and how does it differ from dense attention? Dense attention compares every token in a text with every other token via multiplication, making computation grow quadratically with text length — a 10,000-word document may require 50 million multiplications. Sparse attention runs calculations on only a subset of token pairings, radically reducing compute. The trade-off historically was lower accuracy, but Subquadratic claims its SubQ model is the first sparse attention mechanism to rival mainstream LLMs on tasks like search and coding. _Teams evaluating efficient LLM architectures for production workloads track developments like these on daily.dev._ ### How do diffusion models work for text generation and how fast are they compared to standard LLMs? Diffusion LLMs start with a random string of tokens and iteratively refine the entire block into coherent text simultaneously, rather than generating one token at a time. They still use transformers to encode meaning but predict many tokens at once, making them significantly more efficient. Inception's Mercury 2 claims performance comparable to GPT-4-class models while being 10 times faster, and Google has built a prototype called Diffusion Gemma using the same approach. _Developers choosing between LLM inference strategies for cost and latency find the latest comparisons on daily.dev._ ### What are liquid neural networks and how does Liquid AI use them in its LFMs? Liquid neural networks extend convolutional networks with a mechanism that lets a model adapt its behavior to new information at inference time — unlike transformers, whose behavior is fixed after training. Liquid AI's latest liquid foundation models (LFMs) are hybrid architectures: 20% transformers and 80% liquid neural networks, a ratio discovered by an internal AI-driven model designer. These models match rivals four times their size, run on a Raspberry Pi, and have reached nearly 34 million downloads. _Engineers building edge or on-device AI can follow liquid neural network progress on daily.dev._ ## Similar posts on daily.dev - [A startup says it cracked the bottleneck holding back AI](https://daily.dev/posts/a-startup-says-it-cracked-the-bottleneck-holding-back-ai-a0nvetkis) · The Next Web · 1 upvotes · 0 comments - [A startup claims it broke through a bottleneck that’s holding back LLMs](https://daily.dev/posts/a-startup-claims-it-broke-through-a-bottleneck-that-s-holding-back-llms-crc4dg053) · MIT Technology Review · 0 upvotes · 0 comments - [What comes after attention? This startup says it already knows.](https://daily.dev/posts/what-comes-after-attention-this-startup-says-it-already-knows--lk6wmeah7) · The New Stack · 2 upvotes · 1 comments - [Beyond Standard LLMs](https://daily.dev/posts/beyond-standard-llms-b1iaquuim) · Sebastian Raschka · 28 upvotes · 1 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#neural-networks](https://daily.dev/tags/neural-networks), [#diffusion-models](https://daily.dev/tags/diffusion-models) [View this post on daily.dev](https://daily.dev/posts/these-startups-are-chasing-the-next-big-thing-in-llms-g8wfdvibc) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"These startups are chasing the next big thing in LLMs","url":"https://daily.dev/posts/these-startups-are-chasing-the-next-big-thing-in-llms-g8wfdvibc","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/these-startups-are-chasing-the-next-big-thing-in-llms-g8wfdvibc"},"datePublished":"2026-08-10T09:29:44.866Z","dateModified":"2026-08-10T09:30:39.083Z","description":"Transformers, the neural network architecture behind every major LLM, are increasingly showing limitations: quadratic compute costs, constrained context...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2d2f697159d2c32dfdd55b0ca4149310?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2d2f697159d2c32dfdd55b0ca4149310?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"MIT Technology Review","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"MIT Technology Review","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/a8d03889722f4f22b3986e0b395ff076","url":"https://daily.dev/sources/technologyreview"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/these-startups-are-chasing-the-next-big-thing-in-llms-g8wfdvibc","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,neural-networks,diffusion-models","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"MIT Technology Review","item":"https://daily.dev/sources/technologyreview"},{"@type":"ListItem","position":3,"name":"These startups are chasing the next big thing in LLMs"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/these-startups-are-chasing-the-next-big-thing-in-llms-g8wfdvibc#faq","mainEntity":[{"@type":"Question","name":"What is sparse attention in LLMs and how does it differ from dense attention?","acceptedAnswer":{"@type":"Answer","text":"Dense attention compares every token in a text with every other token via multiplication, making computation grow quadratically with text length — a 10,000-word document may require 50 million multiplications. Sparse attention runs calculations on only a subset of token pairings, radically reducing compute. The trade-off historically was lower accuracy, but Subquadratic claims its SubQ model is the first sparse attention mechanism to rival mainstream LLMs on tasks like search and coding. Teams evaluating efficient LLM architectures for production workloads track developments like these on daily.dev."}},{"@type":"Question","name":"How do diffusion models work for text generation and how fast are they compared to standard LLMs?","acceptedAnswer":{"@type":"Answer","text":"Diffusion LLMs start with a random string of tokens and iteratively refine the entire block into coherent text simultaneously, rather than generating one token at a time. They still use transformers to encode meaning but predict many tokens at once, making them significantly more efficient. Inception's Mercury 2 claims performance comparable to GPT-4-class models while being 10 times faster, and Google has built a prototype called Diffusion Gemma using the same approach. Developers choosing between LLM inference strategies for cost and latency find the latest comparisons on daily.dev."}},{"@type":"Question","name":"What are liquid neural networks and how does Liquid AI use them in its LFMs?","acceptedAnswer":{"@type":"Answer","text":"Liquid neural networks extend convolutional networks with a mechanism that lets a model adapt its behavior to new information at inference time — unlike transformers, whose behavior is fixed after training. Liquid AI's latest liquid foundation models (LFMs) are hybrid architectures: 20% transformers and 80% liquid neural networks, a ratio discovered by an internal AI-driven model designer. These models match rivals four times their size, run on a Raspberry Pi, and have reached nearly 34 million downloads. Engineers building edge or on-device AI can follow liquid neural network progress on daily.dev."}}]} ```