--- title: Recent Developments in LLM Architectures: KV Sharing,... description: A deep technical analysis of recent architectural innovations in open-weight LLMs focused on long-context efficiency. Covers four key developments: (1) Gemma... canonical: https://daily.dev/posts/recent-developments-in-llm-architectures-kv-sharing-mhc-and-compressed-attention-qexkx5v1i twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention | daily.dev og:description: A deep technical analysis of recent architectural innovations in open-weight LLMs focused on long-context efficiency. Covers four key developments: (1) Gemma... og:url: https://daily.dev/posts/recent-developments-in-llm-architectures-kv-sharing-mhc-and-compressed-attention-qexkx5v1i og:image: https://api.daily.dev/og/posts/QeXkx5v1i.png og:image:alt: Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention **[Sebastian Raschka](https://daily.dev/sources/sebastianraschka)** · 25 min read · 55 upvotes · 1 comments ## Summary A deep technical analysis of recent architectural innovations in open-weight LLMs focused on long-context efficiency. Covers four key developments: (1) Gemma 4's cross-layer KV sharing and per-layer embeddings (PLE) that reduce KV cache size by ~50% at 128K context; (2) Laguna XS.2's per-layer query-head budgeting that allocates more heads to sliding-window layers and fewer to global attention; (3) ZAYA1-8B's Compressed Convolutional Attention (CCA) that performs attention directly in a compressed latent space with convolutional mixing on Q/K; and (4) DeepSeek V4's manifold-constrained hyper-connections (mHC) for wider residual streams plus CSA/HCA compressed attention that achieves 27% of FLOPs and 10% of KV cache size vs DeepSeek V3.2 at 1M-token context. The overarching trend is that transformer architectures are increasingly specialized for long-context inference efficiency rather than wholesale replacement. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community discussion Top comments from developers on daily.dev. **@whlsk34** · 0 upvotes > there are about 10 different variations on classical attention. Given that the top performing models are not using either of these, it's not totally clear how much an advancement these are compared to other more popular ones. ## Similar posts on daily.dev - [6 local LLMs I've used that prove they're not just smaller versions of cloud models](https://daily.dev/posts/6-local-llms-i-ve-used-that-prove-they-re-not-just-smaller-versions-of-cloud-models-y8cd7rwnc) · XDA Developers · 0 upvotes · 0 comments - [Attention Mechanisms in LLMs, clearly explained](https://daily.dev/posts/attention-mechanisms-in-llms-clearly-explained-srfzv2qkd) · Daily Dose of Data Science \| Avi Chawla \| Substack · 0 upvotes · 0 comments - [A Visual Guide to Attention Variants in Modern LLMs](https://daily.dev/posts/a-visual-guide-to-attention-variants-in-modern-llms-g7kuqvzmv) · Sebastian Raschka · 6 upvotes · 0 comments - [How LLMs Handle Infinite Context With Finite Memory](https://daily.dev/posts/how-llms-handle-infinite-context-with-finite-memory-4eiofc662) · Towards Data Science · 2 upvotes · 0 comments --- Tags: [#ai-inference](https://daily.dev/tags/ai-inference), [#deep-learning](https://daily.dev/tags/deep-learning), [#deepseek](https://daily.dev/tags/deepseek), [#llm](https://daily.dev/tags/llm) [View this post on daily.dev](https://daily.dev/posts/recent-developments-in-llm-architectures-kv-sharing-mhc-and-compressed-attention-qexkx5v1i) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention","url":"https://daily.dev/posts/recent-developments-in-llm-architectures-kv-sharing-mhc-and-compressed-attention-qexkx5v1i","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/recent-developments-in-llm-architectures-kv-sharing-mhc-and-compressed-attention-qexkx5v1i"},"datePublished":"2026-05-16T14:32:23.999Z","dateModified":"2026-10-15T02:24:53.818Z","description":"A deep technical analysis of recent architectural innovations in open-weight LLMs focused on long-context efficiency. Covers four key developments: (1) Gemma...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f8b0ca4d769740b3480b9b13cd136dc2?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f8b0ca4d769740b3480b9b13cd136dc2?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Sebastian Raschka","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Sebastian Raschka","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/eb22d2e73d074c5598d31588ad16a51c","url":"https://daily.dev/sources/sebastianraschka"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/recent-developments-in-llm-architectures-kv-sharing-mhc-and-compressed-attention-qexkx5v1i","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":55},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"ai-inference,deep-learning,deepseek,llm","timeRequired":"PT25M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Sebastian Raschka","item":"https://daily.dev/sources/sebastianraschka"},{"@type":"ListItem","position":3,"name":"Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/recent-developments-in-llm-architectures-kv-sharing-mhc-and-compressed-attention-qexkx5v1i","comment":[{"@type":"Comment","text":"there are about 10 different variations on classical attention. Given that the top performing models are not using either of these, it’s not totally clear how much an advancement these are compared to other more popular ones.","datePublished":"2026-05-17T00:38:59.513Z","url":"https://daily.dev/posts/QeXkx5v1i#c-8zw0KJJGN","author":{"@type":"Person","name":"whlsk34","url":"https://daily.dev/whlsk34","image":"https://media.daily.dev/image/upload/s--O0TOmw4y--/f_auto/v1715772965/public/noProfile"}}]} ```