--- title: The Hadamard Transform Bug That Quietly Made Our Inverse... description: A debugging story from the VeloxQuant-MLX project explains a subtle math bug in Hadamard transform-based preconditioning for KV-cache quantization. The team... canonical: https://daily.dev/posts/the-hadamard-transform-bug-that-quietly-made-our-inverse-worse-than-returning-zeros-8g2wg0sw5 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The Hadamard Transform Bug That Quietly Made Our Inverse Worse Than Returning Zeros | daily.dev og:description: A debugging story from the VeloxQuant-MLX project explains a subtle math bug in Hadamard transform-based preconditioning for KV-cache quantization. The team... og:url: https://daily.dev/posts/the-hadamard-transform-bug-that-quietly-made-our-inverse-worse-than-returning-zeros-8g2wg0sw5 og:image: https://api.daily.dev/og/posts/8g2wg0sW5.png og:image:alt: The Hadamard Transform Bug That Quietly Made Our Inverse Worse Than Returning Zeros og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The Hadamard Transform Bug That Quietly Made Our Inverse Worse Than Returning Zeros **[Medium](https://daily.dev/sources/medium_js)** · 9 min read · 0 upvotes · 0 comments ## Summary A debugging story from the VeloxQuant-MLX project explains a subtle math bug in Hadamard transform-based preconditioning for KV-cache quantization. The team assumed any Hadamard matrix accepted by MLX's hadamard_transform was self-inverse (applying it twice returns the original), which holds for symmetric Walsh-Hadamard constructions but not for Paley-constructed matrices used at factors m=20 and m=28. Because those matrices are orthogonal but not symmetric, the true inverse requires the transpose, not a repeated forward pass. The error silently propagated through dimensions like 40, 56, 80, 112, 160, and 224 - including real model configurations such as Phi-2 (head_dim=80), Qwen3-4B, and Qwen2.5-7B - producing a relative reconstruction error of about 1.7, worse than simply returning zeros (error of 1.0). No crash, NaN, or exception ever signaled the problem. The fix removed m=20 and m=28 from the fast Hadamard path, falling back to a more expensive but correct QR rotation, and added a round-trip regression test asserting the actual mathematical contract (self-inverse under double application) rather than just kernel execution. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Is every orthogonal Hadamard matrix its own inverse? No. An orthogonal matrix always satisfies H^T H equals the identity, but it only equals its own inverse when it is also symmetric, so that H^T equals H. Walsh-Hadamard matrices built from powers of two are symmetric and self-inverse, but Paley-construction Hadamard matrices, built from quadratic residues over finite fields, can be orthogonal without being symmetric, meaning the true inverse requires the transpose rather than reapplying the same transform. _Developers building quantization kernels can revisit these invertibility edge cases through related breakdowns on daily.dev._ ### Why did applying hadamard_transform twice fail to reconstruct the original tensor for certain dimensions in MLX? Because MLX's hadamard_transform supports dimensions built from base factors including 20 and 28, which correspond to Paley-constructed Hadamard matrices that are orthogonal but not symmetric. Code that assumed reapplying the same transform recovers the original (valid only for symmetric cases like m=1 and m=12) silently produced wrong results for m=20, m=28, and any dimension multiple of them, such as 40, 56, 80, 112, 160, and 224, with a measured relative reconstruction error around 1.7. _Teams relying on MLX's fast transforms for quantization pipelines can track such edge-case pitfalls via daily.dev._ ### How bad was the reconstruction error from the incorrect Hadamard inverse assumption, compared to just returning zeros? The incorrect inverse produced a relative reconstruction error of roughly 1.7, which is worse than the error of 1.0 obtained by simply returning an all-zeros tensor instead of attempting reconstruction. This occurred for dimensions derived from Paley-constructed Hadamard matrices such as 20 and 28 and their multiples, including realistic transformer widths like head_dim=80 used in Phi-2. _Anyone validating numerical correctness in ML kernels can compare similar failure cases on daily.dev._ ## Similar posts on daily.dev - [Before Q, K, and V: Reconstructing the Transformer](https://daily.dev/posts/before-q-k-and-v-reconstructing-the-transformer-g4e7qxjfq) · Towards Data Science · 1 upvotes · 0 comments - [Some Matrix Multiplication Engines Are Not As Accurate As We Thought – PyTorch](https://daily.dev/posts/some-matrix-multiplication-engines-are-not-as-accurate-as-we-thought-pytorch-bu0ytau0p) · PyTorch · 0 upvotes · 0 comments - [TurboQuant: A First-Principles Walkthrough](https://daily.dev/posts/turboquant-a-first-principles-walkthrough-wemarbhdg) · Hacker News · 0 upvotes · 0 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#testing](https://daily.dev/tags/testing) [View this post on daily.dev](https://daily.dev/posts/the-hadamard-transform-bug-that-quietly-made-our-inverse-worse-than-returning-zeros-8g2wg0sw5) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The Hadamard Transform Bug That Quietly Made Our Inverse Worse Than Returning Zeros","url":"https://daily.dev/posts/the-hadamard-transform-bug-that-quietly-made-our-inverse-worse-than-returning-zeros-8g2wg0sw5","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-hadamard-transform-bug-that-quietly-made-our-inverse-worse-than-returning-zeros-8g2wg0sw5"},"datePublished":"2026-10-03T19:01:46.779Z","dateModified":"2026-10-03T19:34:08.375Z","description":"A debugging story from the VeloxQuant-MLX project explains a subtle math bug in Hadamard transform-based preconditioning for KV-cache quantization. The team...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/377eb7903f95d8849f079840c98c36c0?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/377eb7903f95d8849f079840c98c36c0?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-hadamard-transform-bug-that-quietly-made-our-inverse-worse-than-returning-zeros-8g2wg0sw5","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-science,testing","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"The Hadamard Transform Bug That Quietly Made Our Inverse Worse Than Returning Zeros"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-hadamard-transform-bug-that-quietly-made-our-inverse-worse-than-returning-zeros-8g2wg0sw5#faq","mainEntity":[{"@type":"Question","name":"Is every orthogonal Hadamard matrix its own inverse?","acceptedAnswer":{"@type":"Answer","text":"No. An orthogonal matrix always satisfies H^T H equals the identity, but it only equals its own inverse when it is also symmetric, so that H^T equals H. Walsh-Hadamard matrices built from powers of two are symmetric and self-inverse, but Paley-construction Hadamard matrices, built from quadratic residues over finite fields, can be orthogonal without being symmetric, meaning the true inverse requires the transpose rather than reapplying the same transform. Developers building quantization kernels can revisit these invertibility edge cases through related breakdowns on daily.dev."}},{"@type":"Question","name":"Why did applying hadamard_transform twice fail to reconstruct the original tensor for certain dimensions in MLX?","acceptedAnswer":{"@type":"Answer","text":"Because MLX's hadamard_transform supports dimensions built from base factors including 20 and 28, which correspond to Paley-constructed Hadamard matrices that are orthogonal but not symmetric. Code that assumed reapplying the same transform recovers the original (valid only for symmetric cases like m=1 and m=12) silently produced wrong results for m=20, m=28, and any dimension multiple of them, such as 40, 56, 80, 112, 160, and 224, with a measured relative reconstruction error around 1.7. Teams relying on MLX's fast transforms for quantization pipelines can track such edge-case pitfalls via daily.dev."}},{"@type":"Question","name":"How bad was the reconstruction error from the incorrect Hadamard inverse assumption, compared to just returning zeros?","acceptedAnswer":{"@type":"Answer","text":"The incorrect inverse produced a relative reconstruction error of roughly 1.7, which is worse than the error of 1.0 obtained by simply returning an all-zeros tensor instead of attempting reconstruction. This occurred for dimensions derived from Paley-constructed Hadamard matrices such as 20 and 28 and their multiples, including realistic transformer widths like head_dim=80 used in Phi-2. Anyone validating numerical correctness in ML kernels can compare similar failure cases on daily.dev."}}]} ```