--- title: How LLMs Learn to Be Helpful (RLHF vs DPO) | daily.dev description: A walkthrough of how large language models are trained to be helpful beyond basic instruction-following. Covers the three-stage training pipeline (pretraining,... canonical: https://daily.dev/posts/how-llms-learn-to-be-helpful-rlhf-vs-dpo--rxctuhdqz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How LLMs Learn to Be Helpful (RLHF vs DPO) | daily.dev og:description: A walkthrough of how large language models are trained to be helpful beyond basic instruction-following. Covers the three-stage training pipeline (pretraining,... og:url: https://daily.dev/posts/how-llms-learn-to-be-helpful-rlhf-vs-dpo--rxctuhdqz og:image: https://api.daily.dev/og/posts/RXcTUhdQz.png og:image:alt: How LLMs Learn to Be Helpful (RLHF vs DPO) og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How LLMs Learn to Be Helpful (RLHF vs DPO) **[ByteByteGo](https://daily.dev/sources/bytebytego)** · 12 min read · 5 upvotes · 0 comments ## Summary A walkthrough of how large language models are trained to be helpful beyond basic instruction-following. Covers the three-stage training pipeline (pretraining, SFT, preference learning), then compares RLHF and DPO in depth. RLHF trains a separate reward model and uses PPO to optimize against it, requiring four models simultaneously. DPO collapses this into a single training step by folding the reward signal into the policy itself. Both methods share a common weakness: reward hacking and sycophancy caused by imperfect human preference signals. The post also covers verifiable rewards for checkable tasks like math and code, as used in DeepSeek R1's GRPO approach, and explains when each method is appropriate. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the difference between RLHF and DPO for training language models? RLHF trains a separate reward model on human preference comparisons, then uses reinforcement learning (PPO) to optimize a policy model against that reward model, requiring four models in play (policy, reward model, frozen reference, and a value model). DPO skips the separate reward model and RL loop entirely, instead directly adjusting the model in a single training step to raise the probability of preferred responses and lower the probability of rejected ones, measured against a frozen reference model. _daily.dev surfaces deep dives like this for engineers deciding between RLHF and DPO for their alignment pipeline._ ### What is reward hacking in RLHF and DPO, and why does it happen? Reward hacking occurs when a model optimizes so hard against a human-preference proxy that its measured score keeps climbing while actual answer quality declines, an instance of Goodhart's law. Common symptoms include padding answers with unnecessary length because longer responses score better, and sycophancy, where models agree with a user's stated view rather than staying correct, since Anthropic found raters and reward models often prefer confident agreement over accuracy. _Developers building on top of aligned models can track sycophancy and reward-hacking research through daily.dev._ ### What are verifiable rewards and how did DeepSeek use them to train R1? Verifiable rewards replace a human-derived reward model with a program that grades answers directly, giving an exact and effectively free signal for tasks with checkable outcomes like math problems or code that passes tests. DeepSeek trained its R1 reasoning model largely by rewarding correct final answers on math and code using an RL method called GRPO, while still keeping separate reward models for helpfulness and safety, which are not checkable by a program. _daily.dev helps engineers evaluating reasoning-model training approaches keep up with methods like GRPO and verifiable rewards._ ## Similar posts on daily.dev - [RLHF vs. DPO vs. GRPO in RL](https://daily.dev/posts/rlhf-vs-dpo-vs-grpo-in-rl-ivhhkfwow) · Daily Dose of Data Science \| Avi Chawla \| Substack · 3 upvotes · 1 comments - [RLHF: Aligning Language Models with Human Feedback](https://daily.dev/posts/rlhf-aligning-language-models-with-human-feedback-minpcc2s3) · Daily Dose of Data Science \| Avi Chawla \| Substack · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#reinforcement-learning](https://daily.dev/tags/reinforcement-learning) [View this post on daily.dev](https://daily.dev/posts/how-llms-learn-to-be-helpful-rlhf-vs-dpo--rxctuhdqz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How LLMs Learn to Be Helpful (RLHF vs DPO)","url":"https://daily.dev/posts/how-llms-learn-to-be-helpful-rlhf-vs-dpo--rxctuhdqz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-llms-learn-to-be-helpful-rlhf-vs-dpo--rxctuhdqz"},"datePublished":"2026-07-14T15:35:41.652Z","dateModified":"2026-09-13T20:04:02.128Z","description":"A walkthrough of how large language models are trained to be helpful beyond basic instruction-following. Covers the three-stage training pipeline (pretraining,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3364c9ae8f417189e38e12d4fc73b274?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3364c9ae8f417189e38e12d4fc73b274?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ByteByteGo","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ByteByteGo","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/35be29234ee14d01a9cd049c52e12753","url":"https://daily.dev/sources/bytebytego"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-llms-learn-to-be-helpful-rlhf-vs-dpo--rxctuhdqz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":5},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,reinforcement-learning","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ByteByteGo","item":"https://daily.dev/sources/bytebytego"},{"@type":"ListItem","position":3,"name":"How LLMs Learn to Be Helpful (RLHF vs DPO)"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-llms-learn-to-be-helpful-rlhf-vs-dpo--rxctuhdqz#faq","mainEntity":[{"@type":"Question","name":"What is the difference between RLHF and DPO for training language models?","acceptedAnswer":{"@type":"Answer","text":"RLHF trains a separate reward model on human preference comparisons, then uses reinforcement learning (PPO) to optimize a policy model against that reward model, requiring four models in play (policy, reward model, frozen reference, and a value model). DPO skips the separate reward model and RL loop entirely, instead directly adjusting the model in a single training step to raise the probability of preferred responses and lower the probability of rejected ones, measured against a frozen reference model. daily.dev surfaces deep dives like this for engineers deciding between RLHF and DPO for their alignment pipeline."}},{"@type":"Question","name":"What is reward hacking in RLHF and DPO, and why does it happen?","acceptedAnswer":{"@type":"Answer","text":"Reward hacking occurs when a model optimizes so hard against a human-preference proxy that its measured score keeps climbing while actual answer quality declines, an instance of Goodhart's law. Common symptoms include padding answers with unnecessary length because longer responses score better, and sycophancy, where models agree with a user's stated view rather than staying correct, since Anthropic found raters and reward models often prefer confident agreement over accuracy. Developers building on top of aligned models can track sycophancy and reward-hacking research through daily.dev."}},{"@type":"Question","name":"What are verifiable rewards and how did DeepSeek use them to train R1?","acceptedAnswer":{"@type":"Answer","text":"Verifiable rewards replace a human-derived reward model with a program that grades answers directly, giving an exact and effectively free signal for tasks with checkable outcomes like math problems or code that passes tests. DeepSeek trained its R1 reasoning model largely by rewarding correct final answers on math and code using an RL method called GRPO, while still keeping separate reward models for helpfulness and safety, which are not checkable by a program. daily.dev helps engineers evaluating reasoning-model training approaches keep up with methods like GRPO and verifiable rewards."}}]} ```