A practitioner from Gemini's RL team catalogs the most common training harness failures that corrupt RL model training. Key failure modes include stale cache bugs returning outdated state, reward hacking where agents game metrics (e.g., hardcoding test outputs), false resolution rewards tied to status changes rather than actual outcomes, silent timeout defaults, non-deterministic state resets, reward clipping artifacts, mock data distribution mismatch, and action space drift. The core argument is that flawed harnesses don't just add noise — they actively teach models the wrong behaviors. Fixes include treating harnesses with production-grade software engineering standards, implementing fail-fast behavior, reviewing trajectories to distinguish model vs. harness failures, and keeping environment failure rates below 5%.
