---
title: "GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model"
url: https://daily.dev/posts/gem-training-how-meta-doubled-the-efficiency-of-its-llm-scale-ads-foundation-model-yuawarr3a
source_url: https://engineering.fb.com/2026/08/03/ml-applications/training-gem-at-llm-scale-meta-ads-recommendation-foundation-model
type: article
source: "Facebook Engineering\n"
published: 2026-08-03T18:03:53.962Z
updated: 2026-08-03T20:18:13.449Z
tags: ["machine-learning", "llm", "gpu", "pytorch", "recommendation-systems"]
reading_time: 24
upvotes: 2
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

**[Facebook Engineering
](https://daily.dev/sources/facebook_code)** · 24 min read · 2 upvotes · 0 comments

## Summary

Meta's GEM (Generative Ads Recommendation Model), the foundation model powering ads on Instagram and Facebook, now trains at LLM scale on thousands of GPUs with doubled end-to-end training efficiency (20–25% MFU) and 4x more training FLOPs over 12 months. The gains came from co-designing two axes: compute efficiency via a custom recommendation kernel library (Jagged Flash Attention, Generalized Dot-Product Attention, BlockAttention) and mixed ultra-low-precision training (MXFP8); and scaling efficiency via topology-aware 5D parallelism (2D FSDP + Expert Parallelism for dense, Fully Sharded 2D Model Parallelism for sparse parameters), SM-free networking with NCCLX, compiler-based automatic activation checkpointing with quantization, and a zero-communication Base Batch Shuffling technique for load balancing. Key challenges included jagged variable-length inputs wasting up to 50% compute if padded, asymmetric attention patterns breaking standard FlashAttention assumptions, numerical sensitivity of CTR/CVR objectives to low-precision training, and trillion-parameter sparse embedding tables creating memory and communication bottlenecks at scale.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://engineering.fb.com/2026/08/03/ml-applications/training-gem-at-llm-scale-meta-ads-recommendation-foundation-model>

## Similar posts on daily.dev

- [Meta Details GEM Ads Model Using LLM-Scale Training, Hybrid Parallelism, and Knowledge Transfer](https://daily.dev/posts/meta-details-gem-ads-model-using-llm-scale-training-hybrid-parallelism-and-knowledge-transfer-21ryza9tt) · InfoQ · 0 upvotes · 0 comments
- [Meta’s Generative Ads Model \(GEM\): The Central Brain Accelerating Ads Recommendation AI Innovation](https://daily.dev/posts/meta-s-generative-ads-model-gem-the-central-brain-accelerating-ads-recommendation-ai-innovation-jduz74tbf) · Facebook Engineering
 · 1 upvotes · 0 comments
- [How Meta Built a New AI-Powered Ads Model for 5% Better Conversions](https://daily.dev/posts/how-meta-built-a-new-ai-powered-ads-model-for-5-better-conversions-plmz0bqth) · ByteByteGo · 0 upvotes · 0 comments

---

Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#llm](https://daily.dev/tags/llm), [#gpu](https://daily.dev/tags/gpu), [#pytorch](https://daily.dev/tags/pytorch), [#recommendation-systems](https://daily.dev/tags/recommendation-systems)

[View this post on daily.dev](https://daily.dev/posts/gem-training-how-meta-doubled-the-efficiency-of-its-llm-scale-ads-foundation-model-yuawarr3a)