Meta's GEM (Generative Ads Recommendation Model), the foundation model powering ads on Instagram and Facebook, now trains at LLM scale on thousands of GPUs with doubled end-to-end training efficiency (20–25% MFU) and 4x more training FLOPs over 12 months. The gains came from co-designing two axes: compute efficiency via a custom recommendation kernel library (Jagged Flash Attention, Generalized Dot-Product Attention, BlockAttention) and mixed ultra-low-precision training (MXFP8); and scaling efficiency via topology-aware 5D parallelism (2D FSDP + Expert Parallelism for dense, Fully Sharded 2D Model Parallelism for sparse parameters), SM-free networking with NCCLX, compiler-based automatic activation checkpointing with quantization, and a zero-communication Base Batch Shuffling technique for load balancing. Key challenges included jagged variable-length inputs wasting up to 50% compute if padded, asymmetric attention patterns breaking standard FlashAttention assumptions, numerical sensitivity of CTR/CVR objectives to low-precision training, and trillion-parameter sparse embedding tables creating memory and communication bottlenecks at scale.
