--- title: DeepSeek's Engram: a memory lookup table that improves... description: DeepSeek's Engram paper proposes a third architectural component for transformers, sitting alongside attention and feed-forward networks. It functions as a... canonical: https://daily.dev/posts/deepseek-s-engram-a-memory-lookup-table-that-improves-transformer-accuracy-and-efficiency-lk5zjbxyb twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: DeepSeek's Engram: a memory lookup table that improves transformer accuracy and efficiency | daily.dev og:description: DeepSeek's Engram paper proposes a third architectural component for transformers, sitting alongside attention and feed-forward networks. It functions as a... og:url: https://daily.dev/posts/deepseek-s-engram-a-memory-lookup-table-that-improves-transformer-accuracy-and-efficiency-lk5zjbxyb og:image: https://api.daily.dev/og/posts/lk5ZjBxyb.png og:image:alt: DeepSeek's Engram: a memory lookup table that improves transformer accuracy and efficiency og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # DeepSeek's Engram: a memory lookup table that improves transformer accuracy and efficiency **[Collections](https://daily.dev/sources/collections)** · 3 min read · 3 upvotes · 0 comments ## Summary DeepSeek's Engram paper proposes a third architectural component for transformers, sitting alongside attention and feed-forward networks. It functions as a hashed lookup table that stores and retrieves multi-token pattern embeddings (e.g., named entities like 'Diana, Princess of Wales') rather than recomputing them. Unlike Mixture of Experts, which is conditional compute, Engram is conditional memory — scaling the hash table improves performance without increasing per-token compute. Ablation studies show factual knowledge tasks drop up to 56% and trivia accuracy falls 70% when Engram is removed, while reading comprehension is largely unaffected, confirming it handles factual retrieval specifically. Inference overhead is only 1.9–2.8%. Replacing 20–25% of MoE sparse capacity with Engram improves both efficiency and benchmark accuracy. DeepSeek V4 is expected to implement Engram at scale. ## Content DeepSeek's latest paper introduces Engram, a new architectural block that sits alongside attention and feed-forward networks inside a transformer. If that framing sounds significant, it's because it is — most transformer research tweaks existing components rather than proposing a genuinely new one. ## What Engram actually does The core idea is straightforward: instead of recomputing the meaning of a well-known phrase like "Diana, Princess of Wales" from scratch across multiple layers every single time it appears, Engram stores a representation of it and retrieves it on demand. Think of it as a pantry. Attention and feed-forward networks are the cooking — active, compute-intensive processes. Engram is just grabbing something off the shelf. Technically, it works as a hashed lookup table. When the model encounters a trigger token, it uses multi-head hashing to retrieve stored n-gram embeddings for matching multi-token patterns. A contextualized gating mechanism then filters out noisy or irrelevant retrievals before the result gets fused into the model's representation. The lookup is deterministic, which means it can be prefetched — inference overhead ends up being only about 1.9–2.8% slower, which is negligible. ## How it differs from Mixture of Experts It's worth being precise here because the distinction matters. Mixture of Experts (MoE) is conditional *compute* — the model activates different expert weights depending on the input. Engram is conditional *memory* — it fetches static stored patterns rather than running any additional computation. Scaling the hash table improves performance without increasing per-token compute at all. ## What the ablation studies show This is where things get interesting. The researchers ran ablation tests — removing Engram from a trained model — and found: - Factual knowledge and algorithmic task performance dropped by up to 56% - Trivia accuracy specifically fell by 70% - Reading comprehension stayed at 93% That last point is telling. Reading comprehension doesn't rely on stored facts — it works from context already present in the input. Trivia does rely on stored facts. The pattern suggests Engram is doing exactly what it's supposed to: handling factual retrieval specifically, rather than general language understanding. Ablations also found the optimal placement for Engram is at transformer layers 2 and 6, and the best budget split allocates 20–25% of sparse capacity to Engram rather than MoE layers. Replacing that slice of MoE with a simple lookup table not only improves efficiency — it improves accuracy across benchmarks, which is the genuinely surprising result here. ## Why this matters The practical implication is that models could become cheaper to run. If a meaningful portion of what large models do is just pattern-matching against known entities — and the ablation numbers suggest it is — then offloading that to a static lookup table rather than burning compute on it is a real efficiency gain. The authors suggest this could make capable models more viable to run locally. DeepSeek V4 is expected to implement Engram at scale, so we'll get a clearer picture of how it holds up outside controlled ablation conditions. For now, the paper makes a credible case that the transformer's two-component architecture has room for a third. --- Tags: [#deepseek](https://daily.dev/tags/deepseek), [#llm](https://daily.dev/tags/llm), [#mixture-of-experts](https://daily.dev/tags/mixture-of-experts), [#neural-networks](https://daily.dev/tags/neural-networks) [View this post on daily.dev](https://daily.dev/posts/deepseek-s-engram-a-memory-lookup-table-that-improves-transformer-accuracy-and-efficiency-lk5zjbxyb) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"DeepSeek's Engram: a memory lookup table that improves transformer accuracy and efficiency","url":"https://daily.dev/posts/deepseek-s-engram-a-memory-lookup-table-that-improves-transformer-accuracy-and-efficiency-lk5zjbxyb","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/deepseek-s-engram-a-memory-lookup-table-that-improves-transformer-accuracy-and-efficiency-lk5zjbxyb"},"datePublished":"2026-03-24T17:31:40.181Z","dateModified":"2026-04-02T02:14:28.138Z","description":"DeepSeek's Engram paper proposes a third architectural component for transformers, sitting alongside attention and feed-forward networks. It functions as a...","image":"https://i.ytimg.com/vi/xUlX6jvwVfM/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/xUlX6jvwVfM/sddefault.jpg","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/deepseek-s-engram-a-memory-lookup-table-that-improves-transformer-accuracy-and-efficiency-lk5zjbxyb","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"deepseek,llm,mixture-of-experts,neural-networks","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"DeepSeek's Engram: a memory lookup table that improves transformer accuracy and efficiency"}]} ```