--- title: Scaling up Continual Learning — Ronak Malde, Trajectory description: A founder of Trajectory (formerly at WinSurf, where his team trained the model behind a $2B DeepMind acquisition) presents a conference talk on scaling... canonical: https://daily.dev/posts/scaling-up-continual-learning-ronak-malde-trajectory-ugp4xw2r0 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Scaling up Continual Learning — Ronak Malde, Trajectory | daily.dev og:description: A founder of Trajectory (formerly at WinSurf, where his team trained the model behind a $2B DeepMind acquisition) presents a conference talk on scaling... og:url: https://daily.dev/posts/scaling-up-continual-learning-ronak-malde-trajectory-ugp4xw2r0 og:image: https://api.daily.dev/og/posts/Ugp4xW2r0.png og:image:alt: Scaling up Continual Learning — Ronak Malde, Trajectory og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Scaling up Continual Learning — Ronak Malde, Trajectory **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 23 min read · 0 upvotes · 0 comments ## Summary A founder of Trajectory (formerly at WinSurf, where his team trained the model behind a $2B DeepMind acquisition) presents a conference talk on scaling continual learning for AI models. He argues current RL methods like GRPO suffer from off-policy task distributions, heavy rollout infrastructure, and sequence-level rewards. He introduces on-policy self-distillation (OPSD), where a student model is trained to match a teacher's log-probs on its own rollouts, with the teacher given a 'hint' to be smarter. He details problems that emerge at scale (120B+ parameters, long tool-calling trajectories), including a 'buttweight problem' where models collapse into hedging words, solved via step-level KL divergence weighting, and 'hint leakage', solved via LLM-filtered hints or residual guidance. The talk closes with a pitch for Trajectory's platform for continual learning from production agent traces. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is on-policy self-distillation (OPSD) and how does it differ from GRPO for training language models? On-policy self-distillation trains a student model to match a teacher model's token-level log probabilities on the student's own rollouts, where the teacher is given a privileged 'hint' about the correct answer. Unlike GRPO, it needs no parallel rollouts or group sampling, uses per-token dense reward instead of sequence-level scalar reward, and can train on genuinely online task distributions rather than fixed benchmarks. _daily.dev surfaces deep dives like this for teams weighing RL against distillation-based training approaches._ ### What is the 'buttweight problem' that occurs when scaling on-policy self-distillation to long tool-calling tasks? It is a failure mode where, on long trajectories, the student model diverges from the teacher and the teacher repeatedly tries to course-correct at every step, pushing the student's outputs toward hedging tokens like 'wait', 'maybe', and 'but' instead of confident answers. The fix involves computing step-level KL divergence between student and teacher and using it as a per-step token weight rather than a blanket penalty, so only genuinely divergent steps get corrected. _engineers debugging long-horizon agent training instabilities can track approaches like this via daily.dev._ ### What is hint leakage in self-distillation training and how can it be prevented? Hint leakage happens when a teacher model is given a hint containing the final answer or solution steps, causing the student's rollout to implausibly reference information it could not have known, such as stating the answer before deriving it. Mitigations include using an LLM to filter hints down to only reasonable clues a real system would have (like log details without the exact fix), or using residual guidance, which blends partial-hint and full-hint teacher log-probs to avoid shifting the student into out-of-distribution territory. _anyone designing reward or hint signals for agent training can find practical patterns like this on daily.dev._ --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#reinforcement-learning](https://daily.dev/tags/reinforcement-learning) [View this post on daily.dev](https://daily.dev/posts/scaling-up-continual-learning-ronak-malde-trajectory-ugp4xw2r0) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Scaling up Continual Learning — Ronak Malde, Trajectory","url":"https://daily.dev/posts/scaling-up-continual-learning-ronak-malde-trajectory-ugp4xw2r0","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/scaling-up-continual-learning-ronak-malde-trajectory-ugp4xw2r0"},"datePublished":"2026-08-12T14:54:05.912Z","dateModified":"2026-08-12T14:54:33.107Z","description":"A founder of Trajectory (formerly at WinSurf, where his team trained the model behind a $2B DeepMind acquisition) presents a conference talk on scaling...","image":"https://i.ytimg.com/vi/zL1kLftVTlo/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/zL1kLftVTlo/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/scaling-up-continual-learning-ronak-malde-trajectory-ugp4xw2r0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,reinforcement-learning","timeRequired":"PT23M","video":{"@type":"VideoObject","name":"Scaling up Continual Learning — Ronak Malde, Trajectory","description":"A founder of Trajectory (formerly at WinSurf, where his team trained the model behind a $2B DeepMind acquisition) presents a conference talk on scaling...","thumbnailUrl":"https://i.ytimg.com/vi/zL1kLftVTlo/sddefault.jpg","uploadDate":"2026-08-12T14:54:05.912Z","duration":"PT23M","url":"https://api.daily.dev/r/Ugp4xW2r0","embedUrl":"https://www.youtube.com/embed/zL1kLftVTlo"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"Scaling up Continual Learning — Ronak Malde, Trajectory"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/scaling-up-continual-learning-ronak-malde-trajectory-ugp4xw2r0#faq","mainEntity":[{"@type":"Question","name":"What is on-policy self-distillation (OPSD) and how does it differ from GRPO for training language models?","acceptedAnswer":{"@type":"Answer","text":"On-policy self-distillation trains a student model to match a teacher model's token-level log probabilities on the student's own rollouts, where the teacher is given a privileged 'hint' about the correct answer. Unlike GRPO, it needs no parallel rollouts or group sampling, uses per-token dense reward instead of sequence-level scalar reward, and can train on genuinely online task distributions rather than fixed benchmarks. daily.dev surfaces deep dives like this for teams weighing RL against distillation-based training approaches."}},{"@type":"Question","name":"What is the 'buttweight problem' that occurs when scaling on-policy self-distillation to long tool-calling tasks?","acceptedAnswer":{"@type":"Answer","text":"It is a failure mode where, on long trajectories, the student model diverges from the teacher and the teacher repeatedly tries to course-correct at every step, pushing the student's outputs toward hedging tokens like 'wait', 'maybe', and 'but' instead of confident answers. The fix involves computing step-level KL divergence between student and teacher and using it as a per-step token weight rather than a blanket penalty, so only genuinely divergent steps get corrected. engineers debugging long-horizon agent training instabilities can track approaches like this via daily.dev."}},{"@type":"Question","name":"What is hint leakage in self-distillation training and how can it be prevented?","acceptedAnswer":{"@type":"Answer","text":"Hint leakage happens when a teacher model is given a hint containing the final answer or solution steps, causing the student's rollout to implausibly reference information it could not have known, such as stating the answer before deriving it. Mitigations include using an LLM to filter hints down to only reasonable clues a real system would have (like log details without the exact fix), or using residual guidance, which blends partial-hint and full-hint teacher log-probs to avoid shifting the student into out-of-distribution territory. anyone designing reward or hint signals for agent training can find practical patterns like this on daily.dev."}}]} ```