IsoExec is a cross-framework execution abstraction built for SkyRL that eliminates numerical mismatch between rollout (vLLM) and training (Megatron) engines in RL workloads. It combines an execution contract enforcing identical rounding-sensitive kernel choices with a unified, batch-invariant, bitwise-consistent model, plus a new chunkwise-parallel recurrent (CPR) algorithm for Gated DeltaNet architectures. On an 8×H100 node training Qwen3.5-35B-A3B with DAPO, it reduced average rollout-versus-training logprob difference below 1e-6 at a 25.3% end-to-end step-time overhead versus the native SkyRL stack, with no clear reward gain observed over a short 50-step run. The implementation is open-sourced.
Table of contents
TL;DRIntroductionUnified execution contractUnified modelResultsNext stepsAcknowledgementsQuestions this post answers
What causes the mismatch between rollout and training logprobs in RL training systems like SkyRL that use vLLM and Megatron?
Rollout and training engines use different kernels, batch shapes, execution modes, and parallelism layouts, and since floating-point arithmetic is non-associative, these differences change reduction order and therefore token probabilities even when both engines execute the same policy. This mismatch can destabilize algorithms like REINFORCE and GRPO and complicate debugging of new RL infrastructure changes. Teams debugging RL training instability can follow infrastructure fixes like this one on daily.dev.
How much overhead does eliminating train-inference mismatch add to RL training with IsoExec in SkyRL?
IsoExec adds roughly 25.3% end-to-end overhead per RL step compared to the native SkyRL stack when training Qwen3.5-35B-A3B with DAPO on a single 8xH100 node. Generation time increased 31.3% (591.3s to 776.6s) and policy training increased 18.6% (498.6s to 591.3s), while reducing average rollout-versus-training logprob difference below 1e-6. Engineers weighing numerical precision against throughput in RL pipelines can track these trade-offs on daily.dev.
What is chunkwise-parallel recurrent (CPR) and why was it built for Gated DeltaNet models?
Chunkwise-parallel recurrent (CPR) is an algorithm that keeps the recurrence as the main computation but evaluates it in parallel across chunks, designed to make Gated DeltaNet training, prefill, and decode bitwise consistent without the 2-5x slowdown of using a fully recurrent form everywhere. It achieves this with only 1.38-1.67x overhead versus native mixed implementations, versus 4.3x or worse for naive alternatives. Developers optimizing linear-attention architectures for RL can follow kernel-level techniques like this on daily.dev.