5 useful things you'll learn in my post-training textbook
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Nathan Lambert announces the publication of his textbook 'Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs' by Manning. The book covers five key areas: intuitions for RL algorithms (PPO, GSPO, CISPO), systems design for modern RL training, the history of post-training from 2018 to present, demystifying distillation techniques, and practical pitfalls like over-optimization and character training. It targets CS-degree-level readers and is freely available online alongside a 12-hour YouTube course and code exercises. A 50% discount is available until August 19th with code PBLambert.
Table of contents
1. Intuitions for how RL algorithms change the outputs of models2. An understanding of the crucial factors facing new RL systems and algorithms3. The histories that lead to modern post-training4. Dispelling the magic of “distillation”5. A survey of all the other little headaches you encounter when trying to do post-training rightQuestions this post answers
What topics does the RLHF post-training textbook by Nathan Lambert cover?
The book covers RL algorithm intuitions (policy-gradient theorem, PPO, GSPO, CISPO), systems design for asynchronous RL training, the history of post-training from 2018 onward, knowledge distillation techniques including multi-teacher on-policy distillation, and practical challenges like over-optimization, regularization, evaluation, and character training. It targets readers with a CS bachelor's degree or equivalent background. Engineers building on top of LLMs track post-training research like this on daily.dev.
Is there a free version of the RLHF book by Nathan Lambert available online?
Yes, the full book is freely available at rlhfbook.com. It also comes with a 12-hour course including slides and YouTube videos, a code repository with exercises, and model completion comparisons. A print copy is available from Manning with 50% off using code PBLambert until August 19th, and from Amazon US now and Amazon UK in October. Researchers keeping up with LLM alignment resources find them faster on daily.dev.