A practical guide to aligning open LLMs using Direct Preference Optimization (DPO) and on-policy synthetic data. Covers the full pipeline: generating preference pairs from a supervised fine-tuned (SFT) model using a rule-based reward model, training with Hugging Face TRL's DPOTrainer using Q-LoRA on a 24GB GPU, and evaluating on GSM8K. The approach achieved a 5% accuracy improvement over the SFT baseline using only ~2k preference pairs over 3 epochs, with best results reaching 62% on multi-GPU training with DeepSpeed.
