--- title: Compression at the Edge — Chris Alexiuk, NVIDIA | daily.dev description: Quantize a single number in a model and it gets 20% dumber. That finding, from the super weights paper, is why Daniel Han's claim is less absurd than it... canonical: https://daily.dev/posts/compression-at-the-edge-chris-alexiuk-nvidia-mgdvcmrap twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Compression at the Edge — Chris Alexiuk, NVIDIA | daily.dev og:description: Quantize a single number in a model and it gets 20% dumber. That finding, from the super weights paper, is why Daniel Han's claim is less absurd than it... og:url: https://daily.dev/posts/compression-at-the-edge-chris-alexiuk-nvidia-mgdvcmrap og:image: https://api.daily.dev/og/posts/MgDVCmRAP.png og:image:alt: Compression at the Edge — Chris Alexiuk, NVIDIA og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Compression at the Edge — Chris Alexiuk, NVIDIA **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 46 min read · 0 upvotes · 0 comments ## Summary Quantize a single number in a model and it gets 20% dumber. That finding, from the super weights paper, is why Daniel Han's claim is less absurd than it sounds: GLM 5.2 goes from 1.5 terabytes to 250 GB, 86% smaller, without being 86% dumber. Layers are wildly unequal. The first and last carry enormous weight, the middle ones barely matter, and a model trained on 30 trillion tokens never saturates its parameters, so many sit near zero and can simply be set there. Choosing which layers stay in high precision is a combinatorial search, not a setting. NVFP4 is the format the NVIDIA side leans on, a 4 bit float where every group of 16 values shares one FP8 scale, targeting under 1% accuracy loss. New architectures keep breaking the old heuristics: quantize the linear attention layers and the model looks fine right up until a long context turns it to gibberish. Post training quantization works out of the box above roughly 20 billion parameters and needs quantization aware distillation below that. The panel is blunt that benchmarks only cover verifiable tasks, so the real test is running the model in an actual harness, and Han's preferred signal is KL divergence between the BF16 and quantized output logits rather than any accuracy score. Speaker info: Chris Alexiuk, moderator (NVIDIA): - https://x.com/llm_wizard - https://www.linkedin.com/in/csalexiuk Daniel Han (Unsloth): - https://x.com/danielhanchen - https://unsloth.ai Asma Beevi (NVIDIA): - https://www.linkedin.com/in/asma-beevi-k-t-433053a2 - https://realasma.github.io Merve Noyan (Hugging Face): - https://x.com/mervenoyann - https://hf.co/merve Parth Sareen (Ollama): - https://github.com/parthsareen - https://parthsareen.com Timestamps: 0:00 - Welcome and the panel 0:53 - What compression means to each of them 3:05 - GLM 5.2 from 1.5 terabytes to 250 GB 4:08 - When each of them got the compression bug 8:19 - QLoRA and finetuning on a T4 11:44 - 86% smaller without being 86% dumber 12:46 - Why layer importance is so uneven 14:26 - The super weight: one number, 20% dumber 14:51 - Evaluating the quantized checkpoints 16:37 - What NVFP4 actually is 17:55 - Does compression matter beyond the toaster 21:49 - Why compress a big model instead of using a small one 24:30 - Where Ollama fits 28:54 - How hard NVFP4 is to produce 32:46 - The cursed era of model architectures 35:17 - Why linear attention layers break quantization 37:22 - Where compression goes next 43:22 - How do you know a quant is any good ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: --- Tags: [#ai](https://daily.dev/tags/ai), [#ai-engineering](https://daily.dev/tags/ai-engineering) [View this post on daily.dev](https://daily.dev/posts/compression-at-the-edge-chris-alexiuk-nvidia-mgdvcmrap) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Compression at the Edge — Chris Alexiuk, NVIDIA","url":"https://daily.dev/posts/compression-at-the-edge-chris-alexiuk-nvidia-mgdvcmrap","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/compression-at-the-edge-chris-alexiuk-nvidia-mgdvcmrap"},"datePublished":"2026-08-07T01:28:57.592Z","dateModified":"2026-08-07T02:24:26.064Z","description":"Quantize a single number in a model and it gets 20% dumber. That finding, from the super weights paper, is why Daniel Han's claim is less absurd than it...","image":"https://i.ytimg.com/vi/J4_jCrTxMkk/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/J4_jCrTxMkk/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/compression-at-the-edge-chris-alexiuk-nvidia-mgdvcmrap","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,ai-engineering","timeRequired":"PT46M","video":{"@type":"VideoObject","name":"Compression at the Edge — Chris Alexiuk, NVIDIA","description":"Quantize a single number in a model and it gets 20% dumber. That finding, from the super weights paper, is why Daniel Han's claim is less absurd than it...","thumbnailUrl":"https://i.ytimg.com/vi/J4_jCrTxMkk/sddefault.jpg","uploadDate":"2026-08-07T01:28:57.592Z","duration":"PT46M","url":"https://api.daily.dev/r/MgDVCmRAP","embedUrl":"https://www.youtube.com/embed/J4_jCrTxMkk"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"Compression at the Edge — Chris Alexiuk, NVIDIA"}]} ```