--- title: Profiling in PyTorch (Part 3): Attention is all you profile description: Part 3 of a PyTorch profiling series focused on attention mechanisms. Starting from a naive hand-written causal attention implementation, the post walks... canonical: https://daily.dev/posts/profiling-in-pytorch-part-3-attention-is-all-you-profile-qhrrgp3h1 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Profiling in PyTorch (Part 3): Attention is all you profile | daily.dev og:description: Part 3 of a PyTorch profiling series focused on attention mechanisms. Starting from a naive hand-written causal attention implementation, the post walks... og:url: https://daily.dev/posts/profiling-in-pytorch-part-3-attention-is-all-you-profile-qhrrgp3h1 og:image: https://api.daily.dev/og/posts/qHRrGP3h1.png og:image:alt: Profiling in PyTorch (Part 3): Attention is all you profile og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Profiling in PyTorch (Part 3): Attention is all you profile **[Hugging Face](https://daily.dev/sources/huggingface)** · 20 min read · 1 upvotes · 0 comments ## Summary Part 3 of a PyTorch profiling series focused on attention mechanisms. Starting from a naive hand-written causal attention implementation, the post walks through profiler traces to reveal hidden costs: an unexpected Memcpy from out-of-place masked_fill (fixed with a one-line in-place change), and then a deep comparison of all four SDPA backends. The math backend launches 20 GPU kernels and is 3.7x slower due to FP32 upcasting, per-call mask materialization, and safe softmax overhead. The efficient (xformers/CUTLASS) and flash (FlashAttention-2) backends each collapse to a single fused kernel in bf16 on Tensor Cores. The cuDNN backend generates a per-problem kernel with no transpose overhead but pays a higher CPU cost for plan selection. Flash achieves the best GPU time despite low occupancy, which is explained by its deliberate heavy use of registers and shared memory to avoid HBM traffic. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why is PyTorch's scaled_dot_product_attention math backend so much slower than a naive hand-written attention implementation? The math backend is roughly 3.7x slower because it upcasts tensors to FP32 and runs on ordinary CUDA cores instead of Tensor Cores, rebuilds the causal mask from scratch on every call instead of reusing it, and uses a safe softmax that guards against NaNs from fully-masked rows. This adds up to about 20 GPU kernels per forward pass versus 5 for a naive implementation, making it a correctness-focused reference rather than a fast path. _Developers optimizing transformer inference can compare backend trade-offs like this through daily.dev._ ### Why does FlashAttention-2 show very low GPU occupancy in a profiler even though it's the fastest attention backend? FlashAttention-2 deliberately uses heavy per-thread register and shared memory usage per block to keep attention tiles on-chip and avoid materializing the full attention matrix in global memory, which limits how many blocks fit per streaming multiprocessor. On an Ampere GPU this can mean only two 128-thread blocks fit per SM, yielding roughly 13% occupancy, yet it remains the fastest backend because occupancy only hides latency rather than measuring efficiency. _Engineers debugging GPU kernel performance can track profiling nuances like this via daily.dev._ ### Why does the cuDNN attention backend in PyTorch show 0% achieved occupancy in the profiler? The 0% figure is a measurement gap, not a stalled GPU: cuDNN launches its generated kernel through the driver-level cuLaunchKernelEx API instead of the standard cudaLaunchKernel, and the CUPTI profiling backend cannot attribute occupancy to that launch path. The kernel's actual footprint shows about 240 registers times 256 threads per block, fitting one block per SM for roughly 12.5% occupancy, in line with the flash backend. _Teams choosing between SDPA backends for their models can dig into details like this on daily.dev._ ## Similar posts on daily.dev - [Tuning Flash Attention for Peak Performance in NVIDIA CUDA Tile](https://daily.dev/posts/tuning-flash-attention-for-peak-performance-in-nvidia-cuda-tile-ytpdwf7wh) · NVIDIA Developer · 0 upvotes · 0 comments - [Learning FlashAttention the Hard Way](https://daily.dev/posts/learning-flashattention-the-hard-way-os5untnuw) · ITNEXT · 0 upvotes · 0 comments - [FlexAttention \+ FlashAttention-4: Fast and Flexible – PyTorch](https://daily.dev/posts/flexattention-flashattention-4-fast-and-flexible-pytorch-2cancryfg) · PyTorch · 0 upvotes · 0 comments - [FlashAttention 4: Faster, Memory-Efficient Attention for LLMs](https://daily.dev/posts/flashattention-4-faster-memory-efficient-attention-for-llms-xzpyphq6n) · DigitalOcean Community · 1 upvotes · 0 comments - [Profiling in PyTorch \(Part 2\): From nn.Linear to a Fused MLP](https://daily.dev/posts/profiling-in-pytorch-part-2-from-nn-linear-to-a-fused-mlp-zxpmkydkp) · Hugging Face · 1 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#pytorch](https://daily.dev/tags/pytorch) [View this post on daily.dev](https://daily.dev/posts/profiling-in-pytorch-part-3-attention-is-all-you-profile-qhrrgp3h1) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Profiling in PyTorch (Part 3): Attention is all you profile","url":"https://daily.dev/posts/profiling-in-pytorch-part-3-attention-is-all-you-profile-qhrrgp3h1","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/profiling-in-pytorch-part-3-attention-is-all-you-profile-qhrrgp3h1"},"datePublished":"2026-07-10T07:57:19.844Z","dateModified":"2026-09-13T19:27:02.361Z","description":"Part 3 of a PyTorch profiling series focused on attention mechanisms. Starting from a naive hand-written causal attention implementation, the post walks...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2ba6b97faaf3b8d75aea62e8c2e0af00?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2ba6b97faaf3b8d75aea62e8c2e0af00?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hugging Face","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hugging Face","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/f1f55c67d81a4330acf5b90b26b0c8e1","url":"https://daily.dev/sources/huggingface"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/profiling-in-pytorch-part-3-attention-is-all-you-profile-qhrrgp3h1","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,pytorch","timeRequired":"PT20M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hugging Face","item":"https://daily.dev/sources/huggingface"},{"@type":"ListItem","position":3,"name":"Profiling in PyTorch (Part 3): Attention is all you profile"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/profiling-in-pytorch-part-3-attention-is-all-you-profile-qhrrgp3h1#faq","mainEntity":[{"@type":"Question","name":"Why is PyTorch's scaled_dot_product_attention math backend so much slower than a naive hand-written attention implementation?","acceptedAnswer":{"@type":"Answer","text":"The math backend is roughly 3.7x slower because it upcasts tensors to FP32 and runs on ordinary CUDA cores instead of Tensor Cores, rebuilds the causal mask from scratch on every call instead of reusing it, and uses a safe softmax that guards against NaNs from fully-masked rows. This adds up to about 20 GPU kernels per forward pass versus 5 for a naive implementation, making it a correctness-focused reference rather than a fast path. Developers optimizing transformer inference can compare backend trade-offs like this through daily.dev."}},{"@type":"Question","name":"Why does FlashAttention-2 show very low GPU occupancy in a profiler even though it's the fastest attention backend?","acceptedAnswer":{"@type":"Answer","text":"FlashAttention-2 deliberately uses heavy per-thread register and shared memory usage per block to keep attention tiles on-chip and avoid materializing the full attention matrix in global memory, which limits how many blocks fit per streaming multiprocessor. On an Ampere GPU this can mean only two 128-thread blocks fit per SM, yielding roughly 13% occupancy, yet it remains the fastest backend because occupancy only hides latency rather than measuring efficiency. Engineers debugging GPU kernel performance can track profiling nuances like this via daily.dev."}},{"@type":"Question","name":"Why does the cuDNN attention backend in PyTorch show 0% achieved occupancy in the profiler?","acceptedAnswer":{"@type":"Answer","text":"The 0% figure is a measurement gap, not a stalled GPU: cuDNN launches its generated kernel through the driver-level cuLaunchKernelEx API instead of the standard cudaLaunchKernel, and the CUPTI profiling backend cannot attribute occupancy to that launch path. The kernel's actual footprint shows about 240 registers times 256 threads per block, fitting one block per SM for roughly 12.5% occupancy, in line with the flash backend. Teams choosing between SDPA backends for their models can dig into details like this on daily.dev."}}]} ```