---
title: "FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling"
url: https://daily.dev/posts/flashattention-4-algorithm-and-kernel-pipelining-co-design-for-asymmetric-hardware-scaling-v9fwqmklk
source_url: https://www.together.ai/blog/flashattention-4
type: article
source: "Together AI"
published: 2026-05-31T07:40:43.678Z
updated: 2026-05-31T08:23:32.685Z
tags: ["cuda"]
reading_time: 9
upvotes: 0
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling

**[Together AI](https://daily.dev/sources/togetherai)** · 9 min read · 0 upvotes · 0 comments

## Summary

FlashAttention-4 (FA4) is a new attention kernel co-designed for NVIDIA Blackwell GPUs, addressing the asymmetric hardware scaling where tensor core throughput grows much faster than SFU units and shared memory bandwidth. On B200 with BF16, FA4 reaches 1605 TFLOPs/s (71% utilization), up to 1.3× faster than cuDNN 9.13 and 2.7× faster than Triton. Key innovations include: a ping-pong pipeline that overlaps MMA with softmax; software emulation of the exponential function via polynomial approximation (Cody-Waite range reduction + FMA units) to bypass the MUFU bottleneck; use of Blackwell's tensor memory (TMEM) and 2-CTA MMA mode to reduce shared memory traffic in the backward pass; conditional online softmax rescaling to remove rescaling from the critical path; and a longest-processing-time-first tile scheduler for causal masking and variable sequence lengths. FA4 is implemented in CuTe-DSL (CUTLASS' Python kernel DSL), reducing compile times by 20–30× versus C++ templates.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://www.together.ai/blog/flashattention-4>

## Similar posts on daily.dev

- [Overcoming Compute and Memory Bottlenecks with FlashAttention-4 on NVIDIA Blackwell](https://daily.dev/posts/overcoming-compute-and-memory-bottlenecks-with-flashattention-4-on-nvidia-blackwell-cb1uqp4vo) · NVIDIA Developer · 0 upvotes · 0 comments
- [FlashAttention 4: Faster, Memory-Efficient Attention for LLMs](https://daily.dev/posts/flashattention-4-faster-memory-efficient-attention-for-llms-xzpyphq6n) · DigitalOcean Community · 1 upvotes · 0 comments
- [We reverse-engineered Flash Attention 4](https://daily.dev/posts/we-reverse-engineered-flash-attention-4-vuuzhgl36) · Modal · 0 upvotes · 0 comments

---

Tags: [#cuda](https://daily.dev/tags/cuda)

[View this post on daily.dev](https://daily.dev/posts/flashattention-4-algorithm-and-kernel-pipelining-co-design-for-asymmetric-hardware-scaling-v9fwqmklk)