A technical reference covering the core mathematics behind training transformer language models. Topics include compute cost estimation using scaling laws (Chinchilla), memory requirements for inference and training (model weights, optimizer states, gradients, activations), and distributed training strategies including ZeRO stages (1/2/3), 3D parallelism (data, tensor, pipeline), and sharded optimizers. Practical engineering heuristics are provided, such as expected TFLOP/s/A100 throughput benchmarks and memory formulas for fitting models on GPU clusters.
Table of contents
Parameter vs Dataset Tradeoffs #Engineering Takeaways for Compute Costs #Inference #Training #Distributed Training #2 Impressions