--- title: "Harness Engineering for Self-Improvement" url: https://daily.dev/posts/harness-engineering-for-self-improvement-myalinbsq source_url: https://lilianweng.github.io/posts/2026-07-04-harness type: article source: "Lil’Log" published: 2026-07-07T03:02:58.254Z updated: 2026-08-04T17:36:43.703Z tags: ["llm", "ai-agents", "context-engineering"] reading_time: 28 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Harness Engineering for Self-Improvement **[Lil’Log](https://daily.dev/sources/lilianweng)** · 28 min read · 0 upvotes · 0 comments ## Summary A deep technical survey of harness engineering as a path toward recursive self-improvement (RSI) in AI systems. A 'harness' is the orchestration layer surrounding a base model that controls how it plans, uses tools, manages context, and evaluates results. The post organizes recent research into four main areas: context engineering (ACE, MCE, Meta-Harness), workflow design automation (ADAS, AFlow, AI Scientist), self-improving harnesses (Self-Harness, STOP), and evolutionary program search (AlphaEvolve, Darwin Gödel Machine, ShinkaEvolve). Key insight: code is a universal language for harness optimization, enabling LLM-based coding agents to search the same design space human engineers use. The post also covers joint harness+weight optimization (SIA) and closes with seven open challenges: weak evaluators, memory lifecycle, negative results, diversity collapse, reward hacking, long-term success metrics, and the role of human oversight. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community take How the wider developer community reacted, aggregated from 5 discussions and 53 comments across hackernews (as of 2026-08-04). **TL;DR:** The community finds the technical survey genuinely valuable and sparks real practitioner discussion about harness optimization, evals, and tool design — but a vocal thread runs dark humor about recursive self-improvement as an existential risk ('Torment Nexus'), and a few commenters question whether the field is as rigorous as the 'engineering' label implies. **Sentiment:** 45% positive · 35% mixed · 20% skeptical **The case for** - Practitioners share concrete wins from harness tuning, such as reducing tool count and using codebase-index MCPs to cut cost and improve output quality. - The article is praised as a valuable, well-organized resource that maps a fragmented research space. - The 'fail-closed on coverage' pattern is highlighted as a practical mitigation for the weak-evaluator problem the post identifies. - Rolling your own harness is seen as giving developers meaningful control that hosted solutions strip away. **The pushback** - The 'Torment Nexus' thread signals widespread unease that RSI research is being pursued despite known risks, with race-to-the-bottom dynamics normalizing safety trade-offs. - Several commenters question whether 'harness engineering' is really engineering or more of a soft/empirical science driven by vibes. - The SIA joint harness+weight optimization paper is called out as lacking rigorous evidence that the approach actually works. - Public benchmarks are seen as either contaminated, saturated, or non-representative of real-world codebases, making evaluation a largely unsolved problem. - One commenter dismisses the entire effort as reinventing ontologies and expert systems solved decades ago. **By community** - hackernews (mixed): Engaged and substantive — practitioners share real harness-tuning learnings alongside genuine appreciation for the survey, but a prominent satirical thread frames RSI research as reckless, and several technical critiques question rigor and novelty. **Hottest debate:** Whether building toward recursive self-improvement is responsible given known risks, or whether opting out just cedes ground to less careful actors. **Open questions** - How do you build a reliable, repo-specific fitness function (eval) that isn't contaminated by public benchmarks? - At what point does joint harness+weight optimization become genuinely autonomous rather than human-managed iteration? - Will frontier model companies successfully build moats around their harnesses, or will open-source tooling commoditize them? - Is there a principled way to determine the right level of harness complexity, given evidence that stripping it down sometimes improves results? **Highlights** > I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a generic, reliable, and accurate *fitness function* for codebases - turning PRs into gradeable tasks that an agent can solve, and improve on. I’m pretty curious how others are optimizing the coding agent harness now, as this has been a huge pain point for my company (we’re pretty much relying solely on vibes). > — [bisonbear on hackernews · 4 comments](https://news.ycombinator.com/item?id=49170298) > Thanks for the comment - I’ve actually been working on something to generate evals from private repos, as I 100% agree that public benchmarks are either contaminated (as OpenAI found with SWE-bench Verified), not high quality (as OpenAI found with SWE-bench Pro), or in the case of “good” benchmarks (Cognition’s Frontier Code comes to mind), non-representative of a specific repo’s work. Sharing some challenges / learnings from the journey so far: * Tests alone are non-representative: an agent can pass tests but write the code in a way that’s subjectively worse than another test-passing result * However, tests are still important and the best form of deterministic evaluation we have * Creating executable environments to run the tests is challenging, especially when considering that this should be doable for arbitrary repos * Selecting tasks that discriminate is an art - they need to be challenging, but not too challenging, represent the variety of work that’s done in the repo, and contain tests that failed prior to the change * LLMs / agents are very powerful at judging code - but doing so in a way that is calibrated, consistent, and representative of codebase standards requires careful rubric creation and grading * I’ve shifted towards using agents (from “static” LLM calls) to generate the rubrics, and to later grade the rubrics. This is powerful but even more non-deterministic > — [bisonbear on hackernews](https://news.ycombinator.com/item?id=49171367) > > Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts? They're trying. I see a few main avenues: 1. Fitting the models to their specific harness, so that if you want peak model performance, you're stuck with their harness. But this only works if there aren't alternative models that are similar in capability or good enough that don't have that restriction. 2. Locking you out of the harness so that you eventually are just letting it do "stuff" with your data and system, although you don't get to see what the stuff is or why it's doing it. This is the pattern discussed in that Earendil blog post, "The session you cannot take with you." Encrypted reasoning tokens, secret agent prompts, perhaps eventually not even seeing what files are being read or what data is being sent back to their server. This way, you are also shackled to their harness because nothing is portable. But it only works if you trust them implicitly and don't have alternative models and harnesses that don't treat you this way. 3. Tying subscription pricing to the use of their harness, so that it's financially punitive to use another one. This is what Anthropic does. But again, it only works if there aren't alternative models and harnesses that work similarly well for you and don't have that restriction. 4. Marketing. Anthropic is leaning into this one heavy, from what I can tell, based on the constant ads I see for Claude Code. Can it work? People do overpay for things like vodka in fancy bottles that are chemically identical to the cheap stuff. Is Veblen-good AI a trillion dollar business, though? I think the pressure from open source innovation in models and model tooling is going to make it a tough row to hoe. But I'm biased, as I'm actively rooting for Openthropic's demise. > — [anon373839 on hackernews](https://news.ycombinator.com/item?id=49169044) > If we don't create the Torment Nexus first, somebody else who is much less responsible is gonna create the Torment Nexus before us. It's outright irresponsible to not take the lead, we might have to even give up on all safety concerns to make sure we make it to Torment Nexus IPO first. > — [Drakim on hackernews · 3 comments](https://news.ycombinator.com/item?id=49166512) > Really good points Lillian. Agreed on keeping the evaluator outside the loop that evolves the harness. One practical failure mode I’ve have experienced in my agentic harness tasks similar to the “weak evaluators” point: an incomplete check suite that still reports full success. That’s was worse than a weak evaluator, because it made it look correct and decisive. After several trial and error, what helped was fail-closed on coverage i.e if the fixed checks for each operation aren’t all there, nothing ships / nothing gets sent. Otherwise you can “pass” while never running the cases that would have failed. > — [gopalraja on hackernews](https://news.ycombinator.com/item?id=49167251) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49164896) · 226 points · 51 comments - [hackernews](https://news.ycombinator.com/item?id=48814583) · 4 points · 2 comments - [hackernews](https://news.ycombinator.com/item?id=48835030) · 4 points · 0 comments - [hackernews](https://news.ycombinator.com/item?id=48910272) · 2 points · 0 comments - [hackernews](https://news.ycombinator.com/item?id=48898944) · 2 points · 0 comments ## Similar posts on daily.dev - [Harness engineering for coding agent users](https://daily.dev/posts/harness-engineering-for-coding-agent-users-ujzvxz4ny) · Martin Fowler · 1 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#context-engineering](https://daily.dev/tags/context-engineering) [View this post on daily.dev](https://daily.dev/posts/harness-engineering-for-self-improvement-myalinbsq)