Together AI now supports Direct Preference Optimization (DPO) fine-tuning on its platform. DPO is a method for aligning language models with human preferences without reinforcement learning — it trains directly on preference data (prompt + chosen response + rejected response) to increase the probability of preferred outputs and decrease rejected ones. Compared to RLHF, DPO skips the intermediate reward model and PPO training loop, making it simpler and more computationally efficient. The post covers the β hyperparameter (controlling deviation from the reference model), when to use DPO vs. prompting, ideal use cases (nuanced quality judgments like tone, helpfulness, truthfulness), and the recommended SFT+DPO stacked training pipeline. Key monitoring metrics include reward accuracy and KL divergence. DPO is not suited for tasks with objectively correct answers like classification or math.
