--- title: How to Optimize Transformer-Based Models for... description: Training large transformer models is expensive, and low-precision formats like FP8 and NVFP4 on NVIDIA Hopper/Blackwell GPUs can accelerate GEMMs... canonical: https://daily.dev/posts/how-to-optimize-transformer-based-models-for-low-precision-training-iv9miobdi twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How to Optimize Transformer-Based Models for Low-Precision Training | daily.dev og:description: Training large transformer models is expensive, and low-precision formats like FP8 and NVFP4 on NVIDIA Hopper/Blackwell GPUs can accelerate GEMMs... og:url: https://daily.dev/posts/how-to-optimize-transformer-based-models-for-low-precision-training-iv9miobdi og:image: https://api.daily.dev/og/posts/iV9MiOBdi.png og:image:alt: How to Optimize Transformer-Based Models for Low-Precision Training og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How to Optimize Transformer-Based Models for Low-Precision Training **[NVIDIA Developer](https://daily.dev/sources/nvidiadev)** · 8 min read · 0 upvotes · 0 comments ## Summary Training large transformer models is expensive, and low-precision formats like FP8 and NVFP4 on NVIDIA Hopper/Blackwell GPUs can accelerate GEMMs significantly. However, speedups depend heavily on the actual M×K×N matrix shapes your model executes, not just the high-level config. This guide shows how to use NVIDIA Transformer Engine's benchmark tool to derive concrete GEMM shapes from a transformer config (using CodonFM 5B as an example), profile them across BF16, MXFP8, and NVFP4 precisions, and interpret the results. Key findings include: large GEMMs like MLP Down see 1.66x NVFP4 over MXFP8, while small GEMMs like attention output barely benefit (1.05x); autocast mode (including quantization overhead) vs. prequantized mode reveals whether quantization or kernel throughput is the bottleneck; and NVFP4 kernel-only speedup over BF16 can reach 3.48x but drops to 1.98x in realistic autocast mode. The post also covers how to detect silent FP4 fallbacks using NVTE_LOG_LEVEL or Nsight Systems. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Using NVFP4 Low-Precision Model Training for Higher Throughput Without Losing Accuracy](https://daily.dev/posts/using-nvfp4-low-precision-model-training-for-higher-throughput-without-losing-accuracy-mburtmoay) · NVIDIA Developer · 1 upvotes · 0 comments - [Creating the NVIDIA Nemotron 3 Ultra NVFP4 Checkpoint with NVIDIA Model Optimizer](https://daily.dev/posts/creating-the-nvidia-nemotron-3-ultra-nvfp4-checkpoint-with-nvidia-model-optimizer-o6e6ls0pg) · NVIDIA Developer · 1 upvotes · 0 comments - [Train Models Faster with JAX and MaxText Using NVFP4 on NVIDIA Blackwell](https://daily.dev/posts/train-models-faster-with-jax-and-maxtext-using-nvfp4-on-nvidia-blackwell-ouytd5oyn) · NVIDIA Developer · 0 upvotes · 0 comments --- [View this post on daily.dev](https://daily.dev/posts/how-to-optimize-transformer-based-models-for-low-precision-training-iv9miobdi) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How to Optimize Transformer-Based Models for Low-Precision Training","url":"https://daily.dev/posts/how-to-optimize-transformer-based-models-for-low-precision-training-iv9miobdi","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-to-optimize-transformer-based-models-for-low-precision-training-iv9miobdi"},"datePublished":"2026-06-16T16:01:27.110Z","dateModified":"2026-06-18T15:47:36.861Z","description":"Training large transformer models is expensive, and low-precision formats like FP8 and NVFP4 on NVIDIA Hopper/Blackwell GPUs can accelerate GEMMs...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f11ae772ea2a49b82738c25166a2b2fa?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f11ae772ea2a49b82738c25166a2b2fa?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"NVIDIA Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"NVIDIA Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/86e45aab42ba48ce83103d01b1119910","url":"https://daily.dev/sources/nvidiadev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-to-optimize-transformer-based-models-for-low-precision-training-iv9miobdi","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"NVIDIA Developer","item":"https://daily.dev/sources/nvidiadev"},{"@type":"ListItem","position":3,"name":"How to Optimize Transformer-Based Models for Low-Precision Training"}]} ```