Modal shares lessons from running reinforcement learning post-training at scale and introduces the Modal Training Gym, an open-source library to simplify RL infrastructure. The post identifies three core infrastructure bottlenecks: maintaining glue code, queuing for cluster time, and GPU under-utilization. It covers multi-node training challenges, the 100x speed improvement from RDMA over TCP for weight syncs, delta compression for disaggregated RL, and sandbox buffer sizing for environment execution. Modal advocates for open-source training frameworks (slime, verl, OpenRLHF) over closed managed services, arguing the RL ecosystem moves too fast for any single vendor. The Training Gym abstracts cluster wiring, environment plumbing, and observability into under 100 lines of code.
