--- title: Training a coding model to paint watercolours with TRL... description: An engineer reproduces a viral experiment where a language model is trained via reinforcement learning to paint watercolour-style images by writing JavaScript... canonical: https://daily.dev/posts/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv-qmt24z3za twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Training a coding model to paint watercolours with TRL and OpenEnv | daily.dev og:description: An engineer reproduces a viral experiment where a language model is trained via reinforcement learning to paint watercolour-style images by writing JavaScript... og:url: https://daily.dev/posts/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv-qmt24z3za og:image: https://api.daily.dev/og/posts/QMt24Z3ZA.png og:image:alt: Training a coding model to paint watercolours with TRL and OpenEnv og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Training a coding model to paint watercolours with TRL and OpenEnv **[Hugging Face](https://daily.dev/sources/huggingface)** · 22 min read · 0 upvotes · 0 comments ## Summary An engineer reproduces a viral experiment where a language model is trained via reinforcement learning to paint watercolour-style images by writing JavaScript code through the p5.brush library. Using TRL and OpenEnv on Hugging Face infrastructure, the author builds a full open pipeline: a hand-rated reference pool of 178 AI-generated hibiscus paintings, an RL environment that renders and scores sketches with a headless browser, a pairwise judge (Qwen3-VL-30B) and an aesthetic preference model (HPSv3) as reward signals, and GRPO training on Qwen3.5-35B-A3B via LoRA. Three reward mixes are compared, debugging notes cover learning rate, LoRA target modules, and an OpenEnv websocket bug fixed upstream, and every dataset, environment, script, and model is published openly. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I fix a LoRA adapter not training properly on a mixture-of-experts model like Qwen3.5-35B-A3B during GRPO training? Set target_modules to all-linear instead of a hand-picked layer list. A dense-model target_modules list assumes uniform projection names, but MoE architectures like Qwen3.5-35B-A3B name most projections differently, causing only about 10 of 40 layers to actually train. Switching to all-linear reaches every linear layer, though fused routed-expert tensors remain frozen even then. _Anyone tuning LoRA on MoE models can compare configuration gotchas like this one on daily.dev._ ### Why does a GRPO training reward curve stay flat even when the model should be learning? A linear learning rate scheduler can spend most of its learning rate by mid-run, causing the reward to never take off, while a constant_with_warmup scheduler with a higher learning rate (5e-5 instead of 2e-5) fixes this. Another cause is scale_rewards set to group, where a single gate rejection shrinks every other advantage in that rollout group; setting it to none resolves it. _Developers debugging stalled RL training curves can track fixes like these on daily.dev._ ### What reward weighting works best when training a model with RL on subjective aesthetic preference instead of a verifiable reward? Splitting reward weight between a pairwise judge model and a preference scorer like HPSv3 both allow learning, but the balance changes what the model optimizes for: a judge-led mix (0.60 judge, 0.30 HPSv3) produced the most diverse and artistically interesting outputs, while an HPSv3-only mix converged fastest but produced the least varied paintings, since HPSv3 alone stops rewarding improvement once basic subject features appear. _Teams designing aesthetic reward functions for generative RL can compare tradeoffs like these on daily.dev._ ## Similar posts on daily.dev - [GitHub - Danau5tin/ai-trains-ai: RL-training an AI agent to RL-train AI agents.](https://daily.dev/posts/github---danau5tin-ai-trains-ai-rl-training-an-ai-agent-to-rl-train-ai-agents--gqshn7as2) · Hacker News · 0 upvotes · 0 comments - [Reinforcement Learning Environments](https://daily.dev/posts/reinforcement-learning-environments-yi19edoxs) · DigitalOcean Community · 1 upvotes · 0 comments - [How to Build an RL Environment](https://daily.dev/posts/how-to-build-an-rl-environment-u09fzsb0r) · Daily Dose of Data Science \| Avi Chawla \| Substack · 2 upvotes · 0 comments --- Tags: [#reinforcement-learning](https://daily.dev/tags/reinforcement-learning), [#huggingface](https://daily.dev/tags/huggingface), [#generative-art](https://daily.dev/tags/generative-art) [View this post on daily.dev](https://daily.dev/posts/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv-qmt24z3za) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Training a coding model to paint watercolours with TRL and OpenEnv","url":"https://daily.dev/posts/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv-qmt24z3za","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv-qmt24z3za"},"datePublished":"2026-09-03T08:00:34.946Z","dateModified":"2026-09-03T08:25:11.830Z","description":"An engineer reproduces a viral experiment where a language model is trained via reinforcement learning to paint watercolour-style images by writing JavaScript...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/47d56c51b14a8dd9e13a9579fc4323c9?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/47d56c51b14a8dd9e13a9579fc4323c9?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hugging Face","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hugging Face","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/f1f55c67d81a4330acf5b90b26b0c8e1","url":"https://daily.dev/sources/huggingface"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv-qmt24z3za","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"reinforcement-learning,huggingface,generative-art","timeRequired":"PT22M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hugging Face","item":"https://daily.dev/sources/huggingface"},{"@type":"ListItem","position":3,"name":"Training a coding model to paint watercolours with TRL and OpenEnv"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv-qmt24z3za#faq","mainEntity":[{"@type":"Question","name":"How can I fix a LoRA adapter not training properly on a mixture-of-experts model like Qwen3.5-35B-A3B during GRPO training?","acceptedAnswer":{"@type":"Answer","text":"Set target_modules to all-linear instead of a hand-picked layer list. A dense-model target_modules list assumes uniform projection names, but MoE architectures like Qwen3.5-35B-A3B name most projections differently, causing only about 10 of 40 layers to actually train. Switching to all-linear reaches every linear layer, though fused routed-expert tensors remain frozen even then. Anyone tuning LoRA on MoE models can compare configuration gotchas like this one on daily.dev."}},{"@type":"Question","name":"Why does a GRPO training reward curve stay flat even when the model should be learning?","acceptedAnswer":{"@type":"Answer","text":"A linear learning rate scheduler can spend most of its learning rate by mid-run, causing the reward to never take off, while a constant_with_warmup scheduler with a higher learning rate (5e-5 instead of 2e-5) fixes this. Another cause is scale_rewards set to group, where a single gate rejection shrinks every other advantage in that rollout group; setting it to none resolves it. Developers debugging stalled RL training curves can track fixes like these on daily.dev."}},{"@type":"Question","name":"What reward weighting works best when training a model with RL on subjective aesthetic preference instead of a verifiable reward?","acceptedAnswer":{"@type":"Answer","text":"Splitting reward weight between a pairwise judge model and a preference scorer like HPSv3 both allow learning, but the balance changes what the model optimizes for: a judge-led mix (0.60 judge, 0.30 HPSv3) produced the most diverse and artistically interesting outputs, while an HPSv3-only mix converged fastest but produced the least varied paintings, since HPSv3 alone stops rewarding improvement once basic subject features appear. Teams designing aesthetic reward functions for generative RL can compare tradeoffs like these on daily.dev."}}]} ```