--- title: Proximal Policy Optimization in RL | daily.dev description: Part 8 of a Reinforcement Learning course covering Proximal Policy Optimization (PPO) in depth. Topics include why large policy updates cause collapse, trust... canonical: https://daily.dev/posts/proximal-policy-optimization-in-rl-fkyhqeybv twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Proximal Policy Optimization in RL | daily.dev og:description: Part 8 of a Reinforcement Learning course covering Proximal Policy Optimization (PPO) in depth. Topics include why large policy updates cause collapse, trust... og:url: https://daily.dev/posts/proximal-policy-optimization-in-rl-fkyhqeybv og:image: https://api.daily.dev/og/posts/fKYHQeYBV.png og:image:alt: Proximal Policy Optimization in RL og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Proximal Policy Optimization in RL **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 3 min read · 1 upvotes · 0 comments ## Summary Part 8 of a Reinforcement Learning course covering Proximal Policy Optimization (PPO) in depth. Topics include why large policy updates cause collapse, trust regions, the clipped surrogate objective, the KL-penalty variant used in LLM alignment, practical diagnostics for unhealthy training runs, and a from-scratch PyTorch implementation trained on LunarLander. The post also explains how PPO underpins RLHF for ChatGPT and how later methods like DPO, GRPO, and Constitutional AI were all designed in response to PPO. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Proximal Policy Optimization Explained](https://daily.dev/posts/proximal-policy-optimization-explained-onoxdgjfd) · DigitalOcean Community · 0 upvotes · 0 comments - [Distributed Reinforcement Learning for Scalable High-Performance Policy Optimization](https://daily.dev/posts/distributed-reinforcement-learning-for-scalable-high-performance-policy-optimization-4w4kmbsq8) · Towards Data Science · 1 upvotes · 0 comments - [Verifiable Rewards and GRPO in RL](https://daily.dev/posts/verifiable-rewards-and-grpo-in-rl-trts1bpii) · Daily Dose of Data Science \| Avi Chawla \| Substack · 0 upvotes · 0 comments --- Tags: [#pytorch](https://daily.dev/tags/pytorch), [#reinforcement-learning](https://daily.dev/tags/reinforcement-learning) [View this post on daily.dev](https://daily.dev/posts/proximal-policy-optimization-in-rl-fkyhqeybv) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Proximal Policy Optimization in RL","url":"https://daily.dev/posts/proximal-policy-optimization-in-rl-fkyhqeybv","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/proximal-policy-optimization-in-rl-fkyhqeybv"},"datePublished":"2026-06-14T21:07:44.732Z","dateModified":"2026-06-14T21:08:03.859Z","description":"Part 8 of a Reinforcement Learning course covering Proximal Policy Optimization (PPO) in depth. Topics include why large policy updates cause collapse, trust...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/85d87068ab26a6aea80f80bd81a214a8?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/85d87068ab26a6aea80f80bd81a214a8?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Daily Dose of Data Science | Avi Chawla | Substack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Daily Dose of Data Science | Avi Chawla | Substack","logo":"https://media.daily.dev/image/upload/s--4IHQgTOw--/f_auto/v1710503712/logos/dailydoseofds","url":"https://daily.dev/sources/dailydoseofds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/proximal-policy-optimization-in-rl-fkyhqeybv","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"pytorch,reinforcement-learning","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Daily Dose of Data Science | Avi Chawla | Substack","item":"https://daily.dev/sources/dailydoseofds"},{"@type":"ListItem","position":3,"name":"Proximal Policy Optimization in RL"}]} ```