<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/when-ai-builds-itself-who-owns-the-loop--fioa6ljey" -->

---
title: When AI Builds Itself, Who Owns the Loop? | daily.dev
description: An essay argues that recursive AI self-improvement is fundamentally a systems and governance problem rather than a pure intelligence problem. It contends that...
canonical: https://daily.dev/posts/when-ai-builds-itself-who-owns-the-loop--fioa6ljey
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: When AI Builds Itself, Who Owns the Loop? | daily.dev
og:description: An essay argues that recursive AI self-improvement is fundamentally a systems and governance problem rather than a pure intelligence problem. It contends that...
og:url: https://daily.dev/posts/when-ai-builds-itself-who-owns-the-loop--fioa6ljey
og:image: https://api.daily.dev/og/posts/FiOA6Ljey.png
og:image:alt: When AI Builds Itself, Who Owns the Loop?
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# When AI Builds Itself, Who Owns the Loop?

**[Alexandra Lapunova](https://daily.dev/sources/0f31q6bm6i7mjn7b7o744)** · [@alexandralapunova](https://daily.dev/alexandralapunova) · 10 min read · 1 upvotes · 0 comments

## Summary

An essay argues that recursive AI self-improvement is fundamentally a systems and governance problem rather than a pure intelligence problem. It contends that the first practical forms of self-improving AI will look like agent loops proposing changes, running experiments, and evaluating results, rather than a single model rewriting its own weights. Key architectural needs identified include a mutation boundary separating what agents can change from what must stay human-governed, executable success contracts instead of subjective evaluation, protection of the evaluator from being gamed by the same system it judges, durable append-only memory and provenance for lineage tracking, and reliable rollback mechanisms. It closes by tying these ideas to brnrd, a persistent infrastructure product for coding agents, framing today's session-persistence features as precursors to tomorrow's self-improving system safeguards.

## Content

**Recursive self-improvement is not only a model problem. It is a runtime, authority, and provenance problem.**

The phrase “when AI builds itself” sounds almost inevitably like science fiction.

A model becomes intelligent enough to inspect itself.

It writes a better version.

That version becomes smarter.

Then it writes an even better version.

Repeat.

Intelligence curves upward while humans watch from increasingly far behind.

It is dramatic.

It is also probably the wrong abstraction.

The first useful forms of recursive self-improvement are unlikely to look like one giant model opening its own brain and rewriting its weights.

They will look much more familiar.

An agent notices a weakness.

Another agent proposes an experiment.

Code gets modified.

A benchmark runs.

A verifier evaluates the result.

A successful artifact becomes the new baseline.

Then the system tries again.

And suddenly the most important object is not the model.

It is the loop.

**We are already assembling the first version**

AI systems are already writing software used to build other AI systems.

They generate training code.

They optimize kernels.

They design experiments.

They search architectural choices.

They analyze failures.

They produce synthetic data.

They operate research tools.

They write evaluation suites.

None of this means we already have fully autonomous recursive self-improvement.

We do not.

But that distinction can distract from something more interesting.

We do not need to wait for some magical moment when an AI system suddenly becomes “self-improving.”

The feedback loop is already being assembled piece by piece.

And loops have architecture.

**Self-improvement is a systems primitive**

Suppose an AI agent improves a CUDA kernel by 7%.

Great.

What happens next?

Who decides the result is actually better?

Which benchmark matters?

Was latency improved at the expense of numerical stability?

Did performance improve only on one hardware architecture?

Which version becomes canonical?

Can the system roll back?

Can another researcher reconstruct exactly what changed?

Now make the problem slightly harder.

Let the system modify the evaluator too.

The builder can now modify the thing that decides whether the builder succeeded.

At that point, “AI improves AI” stops being a benchmark story.

It becomes a systems problem.

More specifically, it becomes a problem of authority, verification, continuity, and control.

**The model should not own the constitution**

A sufficiently capable agent can reason.

It can plan.

It can propose.

It can execute.

It can verify.

But some things should exist outside the thing being optimized.

Authority.

Boundaries.

Audit.

Recovery.

Escalation.

A system becoming more intelligent should not automatically mean it gains more authority over the rules governing its own behavior.

That is why self-improving systems may eventually look less like chatbots and more like systems with an explicit control plane.

The AI system can become more capable.

The control plane decides what that capability is allowed to touch.

This distinction matters enormously.

Capability can scale.

Authority should remain explicit.

**Improvement needs a mutation boundary**

“Improve yourself” sounds straightforward until you ask what exactly the system is allowed to modify.

A coding agent might reasonably be allowed to change prompts, routing, tool strategy, plans, code patches, temporary state, or workflow structure.

It might be allowed to propose changes to its harness or its evaluators.

But should it be allowed to rewrite its own root policy?

Its audit history?

Its rollback mechanism?

Its credentials?

Its acceptance criteria?

Probably not.

The interesting boundary is therefore not simply:

AI / human

It is:

mutable / governed

A self-improving system needs an explicit mutation surface.

Some things are safe to explore.

Some things may be changed only through review.

Some things should remain outside autonomous authority entirely.

The more capable the system becomes, the more important that distinction gets.

**“Better” should be an executable contract**

An agent cannot simply report:

I improved myself.

Improved according to what?

Before a self-improvement run begins, the system needs a definition of success.

Maybe the target benchmark must improve.

Maybe latency cannot regress.

Maybe cost must remain inside a fixed budget.

Maybe security evaluations must stay green.

Maybe a held-out test suite must not regress.

Maybe the system cannot modify files outside an approved scope.

The important part is that these criteria exist before execution.

Then verification becomes:

Did this candidate satisfy the contract?

Not:

Does another AI think this looks better?

That distinction becomes extremely important when a system is running hundreds or thousands of experiments without a human watching each one.

Self-improvement needs executable definitions of “better.”

**The evaluator is part of the machine**

At first, the optimization problem can be simple.

The system modifies the model.

The evaluator remains fixed.

But eventually the optimization surface expands.

The model changes.

The harness changes.

The prompts change.

The tools change.

The routing changes.

The memory strategy changes.

The runtime changes.

Perhaps even the evaluation system changes.

Then the obvious recursive question appears:

Who evaluates the evaluator?

If the same optimizer can modify both the target and the measuring stick, “improvement” can eventually become whatever the system learns to make the metric say.

This does not require an evil AI.

It only requires optimization.

Any optimizer will exploit degrees of freedom we accidentally give it.

So one architectural principle follows naturally:

The component proposing a change should not have unrestricted authority over the system deciding whether that change is acceptable.

Verification is not a second opinion.

It is a contract boundary.

**A self-improving system needs memory**

Every experiment produces knowledge.

This optimization failed.

That architecture worked only under these conditions.

This benchmark improvement turned out to be caused by caching.

That tool configuration reduced retries.

This branch looked promising until experiment 381.

Throwing those observations away after every run would be absurd.

A recursive system therefore needs more than model context.

It needs durable research state.

Not merely:

What happened in the previous conversation?

But:

What have we learned about ourselves?

That means persistent artifacts, experiments, decisions, failures, baselines, hypotheses, rejected branches, human interventions, and evaluation evidence.

The longer the system improves itself, the more valuable this accumulated history becomes.

The intelligence may change.

The learning process needs continuity.

**Continuity creates another problem**

Imagine version 41 deciding that version 40’s memory strategy was inefficient.

It rewrites it.

But that memory contains evidence explaining why version 38 failed.

Or a constraint imposed by a human three weeks ago.

Or the provenance of an evaluation dataset.

Should the system be able to optimize those things away?

Probably not.

Some state is working memory.

Some state is historical evidence.

They should not necessarily have the same mutability.

A serious self-improving system may need append-only regions of its own history.

Because otherwise it can improve not only its behavior.

It can improve the story of how it got there.

And that is a very different capability.

**Recursive improvement needs lineage**

Suppose version 173 is dramatically better than version 172.

Why?

Which experiment produced the improvement?

Which agent proposed it?

What context did it see?

Which artifacts entered the branch?

What code changed?

Which evaluator accepted it?

Which human intervention mattered?

Which previous discovery was reused?

At sufficient scale, nobody will reconstruct this from memory.

The runtime has to preserve it.

Provenance is not paperwork around the system.

It becomes part of the system.

Without lineage, we could eventually have AI systems that are clearly getting better while being increasingly unable to explain why.

That is a dangerous kind of technical debt.

**Rollback is part of intelligence**

Software engineers already understand something that AI discourse sometimes forgets.

A system that can deploy but cannot roll back is not sophisticated.

It is reckless.

Every accepted mutation should preserve a trail:

previous state

candidate state

evaluation evidence

promotion decision

rollback path

If version 214 develops a catastrophic behavior that version 213 did not have, reverting should not require archaeology.

A self-improvement mechanism without a reliable rollback mechanism is not really self-improvement.

It is self-mutation.

There is a large difference.

**Humans should not approve every edit**

The obvious answer to all these problems is:

Put a human in the loop.

But that does not scale.

If an AI system runs thousands of experiments per hour, requiring a human to approve every mutation defeats the entire point.

Human agency cannot mean human micromanagement.

A better model is:

Humans define the constitution. Agents operate inside it.

Humans decide what may change.

Humans define what counts as success.

Humans establish boundaries.

Humans decide which events require escalation.

Humans retain accountability for the system.

The AI searches, proposes, implements, executes, monitors, verifies, and reports inside those constraints.

The goal is not to keep humans busy.

It is to keep humans sovereign.

**The strange future of AI engineering**

For decades, research roughly looked like this:

human -> idea -> experiment -> result -> human

We are gradually inserting machines into every edge:

human -> AI proposes -> AI implements -> AI experiments -> AI evaluates -> AI synthesizes -> repeat

Eventually the human may interact mostly with the outer loop.

Choosing objectives.

Setting policy.

Reviewing anomalies.

Changing constraints.

Deciding when the system’s definition of “better” is no longer ours.

That is a fundamentally different relationship with software.

The human is no longer specifying every operation.

The human is designing the space of legitimate operations.

**The model is not the recursive system**

This is perhaps the distinction I care about most.

People often imagine recursive self-improvement as a property of intelligence.

A sufficiently smart model becomes capable of improving itself.

Maybe.

But useful recursive improvement also requires infrastructure.

It requires persistent state.

Tools.

Experiments.

Evaluation.

Provenance.

Artifacts.

Authority.

Promotion.

Rollback.

Continuity.

Without those things, intelligence can generate modifications.

It cannot reliably build upon them.

The decisive system probably will not look like:

SMART MODEL -> SMARTER MODEL

It will look like a persistent research loop in which intelligence is one component among many.

The model supplies reasoning.

The runtime supplies continuity.

The evaluator supplies evidence.

The control plane supplies legitimacy.

The artifact graph supplies history.

The rollback path supplies reversibility.

The loop is the machine.

**Where brnrd fits**

brnrd is not a recursively self-improving AI system.

We are building persistent infrastructure for coding agents.

But the same primitives keep appearing.

Persistent identity.

Durable state.

Execution boundaries.

Verification.

Artifacts.

Recovery.

Human authority.

Provenance.

Agent replacement.

Runtime continuity.

Today those primitives let an agent survive a session and continue coding tomorrow.

Tomorrow, systems trying to improve their own tools, workflows, and eventually parts of themselves will need the same primitives much more urgently.

Because once the worker can modify the worker, the environment around it becomes the safety boundary.

**When AI builds itself**

So when I hear the phrase:

When AI builds itself

the question that interests me is not only:

How intelligent will it become?

It is:

What architecture decides what it is allowed to become?

Who controls the evaluator?

Who preserves the history?

Who defines better?

Who can stop an experiment?

Who can promote a candidate?

Who can roll it back?

Which parts can the system rewrite?

Which parts remain outside its authority?

Those questions sound much less cinematic than recursive superintelligence.

But if recursive self-improvement becomes real, they may determine almost everything that follows.

Because once AI can build AI, intelligence is no longer the only thing being engineered.

The improvement loop itself becomes infrastructure.

And whoever owns that loop owns much more than the model.

HugiMuni / brnrd

Persistent infrastructure for coding agents.

[https://brnrd.dev](https://brnrd.dev)

## Similar posts on daily.dev

- [Recursive Self-Improvement Edges Closer In AI Labs](https://daily.dev/posts/recursive-self-improvement-edges-closer-in-ai-labs-vhlqfhjap) · IEEE Spectrum · 0 upvotes · 0 comments

---

Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#ai-safety](https://daily.dev/tags/ai-safety), [#mlops](https://daily.dev/tags/mlops), [#system-design](https://daily.dev/tags/system-design)

[View this post on daily.dev](https://daily.dev/posts/when-ai-builds-itself-who-owns-the-loop--fioa6ljey)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/when-ai-builds-itself-who-owns-the-loop--fioa6ljey","headline":"When AI Builds Itself, Who Owns the Loop?","text":"An essay argues that recursive AI self-improvement is fundamentally a systems and governance problem rather than a pure intelligence problem. It contends that the first practical forms of self-improving AI will look like agent loops proposing changes, running experiments, and evaluating results, rather than a single model rewriting its own weights. Key architectural needs identified include a mutation boundary separating what agents can change from what must stay human-governed, executable success contracts instead of subjective evaluation, protection of the evaluator from being gamed by the same system it judges, durable append-only memory and provenance for lineage tracking, and reliable rollback mechanisms. It closes by tying these ideas to brnrd, a persistent infrastructure product for coding agents, framing today's session-persistence features as precursors to tomorrow's self-improving system safeguards.","url":"https://daily.dev/posts/when-ai-builds-itself-who-owns-the-loop--fioa6ljey","datePublished":"2026-09-19T18:25:16.315Z","dateModified":"2026-11-01T18:47:22.891Z","author":{"@type":"Person","name":"Alexandra Lapunova","url":"https://daily.dev/alexandralapunova","image":"https://media.daily.dev/image/upload/s--4HRzRmOE--/f_auto/v1789679122/avatars/avatar_0f31q6bM6i7MjN7B7o744?_a=BAMAMicg0","description":"🦄 Co-founder \u0026 CEO at HugiMuni · Building brnrd, a persistent resident for coding agents","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":20}},"image":"https://media.daily.dev/image/upload/s--9ZYBrl2u--/f_auto/v1789842317/posts/FiOA6Ljey?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/0f31q6bm6i7mjn7b7o744","name":"Alexandra Lapunova"}}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Alexandra Lapunova","item":"https://daily.dev/sources/0f31q6bm6i7mjn7b7o744"},{"@type":"ListItem","position":3,"name":"When AI Builds Itself, Who Owns the Loop?"}]}
```

