FlashAttention-4 (FA4) is a new attention kernel co-designed for NVIDIA Blackwell GPUs, addressing the asymmetric hardware scaling where tensor core throughput grows much faster than SFU units and shared memory bandwidth. On B200 with BF16, FA4 reaches 1605 TFLOPs/s (71% utilization), up to 1.3× faster than cuDNN 9.13 and 2.7× faster than Triton. Key innovations include: a ping-pong pipeline that overlaps MMA with softmax; software emulation of the exponential function via polynomial approximation (Cody-Waite range reduction + FMA units) to bypass the MUFU bottleneck; use of Blackwell's tensor memory (TMEM) and 2-CTA MMA mode to reduce shared memory traffic in the backward pass; conditional online softmax rescaling to remove rescaling from the critical path; and a longest-processing-time-first tile scheduler for causal masking and variable sequence lengths. FA4 is implemented in CuTe-DSL (CUTLASS' Python kernel DSL), reducing compile times by 20–30× versus C++ templates.
