--- title: Your Softmax Is Broken. Here's the Fix. | daily.dev description: Standard cross entropy implementations in PyTorch and other frameworks silently zero out gradients once a prediction's confidence passes a 'precision cliff'... canonical: https://daily.dev/posts/your-softmax-is-broken-here-s-the-fix--asszb7wgb twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Your Softmax Is Broken. Here's the Fix. | daily.dev og:description: Standard cross entropy implementations in PyTorch and other frameworks silently zero out gradients once a prediction's confidence passes a 'precision cliff'... og:url: https://daily.dev/posts/your-softmax-is-broken-here-s-the-fix--asszb7wgb og:image: https://api.daily.dev/og/posts/asSZb7wGB.png og:image:alt: Your Softmax Is Broken. Here's the Fix. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Your Softmax Is Broken. Here's the Fix. **[Jia-Bin Huang](https://daily.dev/sources/jbhuang0604)** · 30 min read · 0 upvotes · 0 comments ## Summary Standard cross entropy implementations in PyTorch and other frameworks silently zero out gradients once a prediction's confidence passes a 'precision cliff' caused by floating point rounding. In FP32 this happens around a logit margin of 16.6, in BF16 at 5.5, and in FP64 at 36.7. The bug stems from computing softmax as 1 plus a tiny number, which rounds back to exactly 1. A fix that keeps the small quantity in the numerator (via softplus, log1p and expm1) avoids this: on a modular addition task, standard cross entropy hits 94% test accuracy while the fixed version reaches 100%, and on a harder task it prevents a crash from 61% to 23% accuracy. The same rounding issue causes spurious gradients in knowledge distillation, fixed the same way. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why does my cross entropy loss suddenly become exactly zero during training even though the prediction isn't perfect This happens because of a floating point rounding bug called the precision cliff. Once a prediction's confidence crosses a certain logit margin, the softmax denominator rounds to exactly 1, making the loss and gradient for the correct class compute as exactly zero. The cliff occurs at a margin of about 16.6 in FP32, 5.5 in BF16, and 36.7 in FP64. _daily.dev surfaces deep dives like this for engineers chasing down mysterious training instabilities._ ### how do I fix vanishing gradients in cross entropy loss caused by floating point precision Rewrite the loss using softplus and log1p so the tiny quantity near the cliff is never computed as 1 plus a small number that then gets subtracted. Mask out the target class, compute the log-sum-exp of the remaining logit differences as the negative margin, and use softplus of the negative margin as the loss, which keeps gradients accurate down to a margin near 88 with no meaningful slowdown. _developers patching numerical stability bugs can find implementation walkthroughs like this on daily.dev._ ### what causes the slingshot effect and loss spikes during grokking with the Adam optimizer The slingshot happens when cross entropy gradients hit the floating point precision cliff and collapse to zero, causing Adam's running average of squared gradients to decay toward zero while weight decay shrinks the margin back down; when a few examples fall below the cliff again, Adam divides the restored gradient by its tiny denominator, producing a huge step and a loss spike. With Adam's default epsilon of 1e-8, the predicted equilibrium margin is 18.4, just above the FP32 cliff of 16.6. _anyone debugging grokking or Adam-related training spikes can track root-cause explanations like this via daily.dev._ ## Similar posts on daily.dev - [Softmax, can you really derive the Jacobian? And should you care? — idlemachines](https://daily.dev/posts/softmax-can-you-really-derive-the-jacobian-and-should-you-care-idlemachines-nflk67kpf) · Hacker News · 0 upvotes · 0 comments - [The AI Model Confidence Trap](https://daily.dev/posts/the-ai-model-confidence-trap-93to0xhgy) · Towards Data Science · 5 upvotes · 1 comments - [Medium](https://daily.dev/posts/medium-wd9znaaop) · Medium · 0 upvotes · 0 comments - [Cutting LLM Memory by 84%: A Deep Dive into Fused Kernels](https://daily.dev/posts/cutting-llm-memory-by-84-a-deep-dive-into-fused-kernels-qa2yvt9ou) · Towards Data Science · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#pytorch](https://daily.dev/tags/pytorch) [View this post on daily.dev](https://daily.dev/posts/your-softmax-is-broken-here-s-the-fix--asszb7wgb) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Your Softmax Is Broken. Here's the Fix.","url":"https://daily.dev/posts/your-softmax-is-broken-here-s-the-fix--asszb7wgb","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/your-softmax-is-broken-here-s-the-fix--asszb7wgb"},"datePublished":"2026-10-25T03:22:08.051Z","dateModified":"2026-10-25T03:22:41.724Z","description":"Standard cross entropy implementations in PyTorch and other frameworks silently zero out gradients once a prediction's confidence passes a 'precision cliff'...","image":"https://i.ytimg.com/vi/dYxEmKbZfr4/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/dYxEmKbZfr4/sddefault.jpg","isAccessibleForFree":true,"articleSection":"Jia-Bin Huang","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Jia-Bin Huang","logo":"https://media.daily.dev/image/upload/s--8hwFrfWM--/f_auto,q_auto/v1776585907/logos/jbhuang0604","url":"https://daily.dev/sources/jbhuang0604"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/your-softmax-is-broken-here-s-the-fix--asszb7wgb","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,pytorch","timeRequired":"PT30M","video":{"@type":"VideoObject","name":"Your Softmax Is Broken. Here's the Fix.","description":"Standard cross entropy implementations in PyTorch and other frameworks silently zero out gradients once a prediction's confidence passes a 'precision cliff'...","thumbnailUrl":"https://i.ytimg.com/vi/dYxEmKbZfr4/sddefault.jpg","uploadDate":"2026-10-25T03:22:08.051Z","duration":"PT30M","url":"https://api.daily.dev/r/asSZb7wGB","embedUrl":"https://www.youtube.com/embed/dYxEmKbZfr4"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Jia-Bin Huang","item":"https://daily.dev/sources/jbhuang0604"},{"@type":"ListItem","position":3,"name":"Your Softmax Is Broken. Here's the Fix."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/your-softmax-is-broken-here-s-the-fix--asszb7wgb#faq","mainEntity":[{"@type":"Question","name":"why does my cross entropy loss suddenly become exactly zero during training even though the prediction isn't perfect","acceptedAnswer":{"@type":"Answer","text":"This happens because of a floating point rounding bug called the precision cliff. Once a prediction's confidence crosses a certain logit margin, the softmax denominator rounds to exactly 1, making the loss and gradient for the correct class compute as exactly zero. The cliff occurs at a margin of about 16.6 in FP32, 5.5 in BF16, and 36.7 in FP64. daily.dev surfaces deep dives like this for engineers chasing down mysterious training instabilities."}},{"@type":"Question","name":"how do I fix vanishing gradients in cross entropy loss caused by floating point precision","acceptedAnswer":{"@type":"Answer","text":"Rewrite the loss using softplus and log1p so the tiny quantity near the cliff is never computed as 1 plus a small number that then gets subtracted. Mask out the target class, compute the log-sum-exp of the remaining logit differences as the negative margin, and use softplus of the negative margin as the loss, which keeps gradients accurate down to a margin near 88 with no meaningful slowdown. developers patching numerical stability bugs can find implementation walkthroughs like this on daily.dev."}},{"@type":"Question","name":"what causes the slingshot effect and loss spikes during grokking with the Adam optimizer","acceptedAnswer":{"@type":"Answer","text":"The slingshot happens when cross entropy gradients hit the floating point precision cliff and collapse to zero, causing Adam's running average of squared gradients to decay toward zero while weight decay shrinks the margin back down; when a few examples fall below the cliff again, Adam divides the restored gradient by its tiny denominator, producing a huge step and a loss spike. With Adam's default epsilon of 1e-8, the predicted equilibrium margin is 18.4, just above the FP32 cliff of 16.6. anyone debugging grokking or Adam-related training spikes can track root-cause explanations like this via daily.dev."}}]} ```