Addy Osmani

AI Engineer

<h2>The shift nobody fully planned for</h2>
Something changed in the last year or two. AI coding agents stopped being demos and started being infrastructure. Thirty percent of deployments on Vercel are now agent-initiated — up from near zero six months ago. At Anthropic, 65% of product PRs are landed by Claude Tag, their Slack-integrated coding agent. Amazon’s Kiro caused a 13-hour AWS Cost Explorer outage after being handed full operator credentials and no guardrails.
The tools got good enough to use in production before anyone figured out how to govern them. That gap is what most of the interesting work right now is trying to close.
<hr>
<h2>The five-layer model</h2>
The clearest framework I’ve seen for thinking about AI engineering breaks the problem into five layers, each one zooming further out from the model itself:
Prompt engineering controls what the model sees in a single call — role, instructions, examples, output format. Chain-of-thought, few-shot examples, JSON schema constraints. This is where most people started, and where most people still spend too much time.
Context engineering controls what the model knows before it starts. The context window fills fast, so the real work is deciding what deserves to be in it: retrieving only relevant chunks, reranking before insertion, summarizing older turns, keeping important instructions away from the middle of long contexts.
Harness engineering is the code surrounding the model — tool calling, retries, structured output parsing, permissions, model routing, human approval gates, observability. One agent retrieves information. Another writes code. Another runs tests. The harness coordinates all of it. This is where the real engineering lives now.
Loop engineering moves the human out of the inner loop. Instead of writing a prompt, inspecting the result, and deciding what to do next, you engineer the system to plan, act, observe, verify, repair, and stop on its own. The hard part isn’t making the loop run — it’s making it stop correctly. An agent claiming it’s done isn’t evidence. Passing tests are evidence.
Graph engineering extends this to entire organizations of agents: planners, researchers, builders, reviewers, parallel branches, conditional routing, shared memory, feedback channels. A loop makes one agent’s behavior programmable. A graph makes a collection of agents operate like an organization.
The model is becoming a commodity. The system around it is where the differentiation is.
<hr>
<h2>What “harness engineering” actually means in practice</h2>
The term is getting used a lot, so it’s worth being specific about what it involves.
A harness is the environment you build around an agent to make its output trustworthy. That means:
<ul>
<li>Encoding your nonfunctional requirements — reliability, security, maintainability — into the agent’s context so it can recover intent and respect authority without being told every time</li>
<li>Automatic gates that run regardless of what the agent thinks it did: type checking, branch coverage, end-to-end tests, schema validation</li>
<li>Scoped credentials so the agent can only do what it needs to do for this specific task</li>
<li>Audit trails that don’t rely on the agent’s own account of events</li>
<li>Stop conditions based on real signals, not the agent’s self-assessment</li>
</ul>
One concrete example: a team at Form3 built Patch Pilot for automated CVE remediation across thousands of repositories. The key architectural decision was keeping dangerous credentials — GitHub write access, CI triggers — in the deterministic orchestration layer rather than handing them to the agent itself. The agent reasons; the harness acts. That boundary is what limits the blast radius when something goes wrong.
Another: Microsoft’s Ace voice tutor uses a state-machine harness that defines each lesson step and feeds the model only the input needed for that specific step. The model never decides where it is in the flow — the harness does. This let them replace Claude Opus with Claude Haiku without sacrificing performance, because the harness was doing the structural work the larger model had been compensating for.
<hr>
<h2>The security problem nobody has fully solved</h2>
Natural language instructions are suggestions to a language model, not enforceable constraints. This is the core problem, and it doesn’t get fixed by better prompting.
Agents inherit the full credentials of whoever runs them. There’s no structural safeguard preventing destructive actions — only the model’s judgment, which is probabilistic and manipulable. The PocketOS incident, where a Cursor agent deleted an entire production database, is the canonical example. The Amazon Kiro incident is a larger-scale version of the same failure: an agent with full operator credentials, no identity boundary, no confirmation gate.
The fixes are architectural:
<ul>
<li>Least-privilege RBAC scoped to narrow credentials per task, not per user</li>
<li>Mandatory approval checkpoints enforced by policy-as-code (Open Policy Agent, not prompts)</li>
<li>Deletion protection independent of agent context — the database shouldn’t be deletable by an agent regardless of what it’s been told</li>
<li>Ephemeral, task-scoped tokens via OAuth 2.0 Token Exchange (RFC 8693), so credentials expire when the task ends</li>
<li>Immutable audit logs that don’t run through the agent’s own account</li>
<li>Kernel-isolated sandboxes — standard containers share a kernel and aren’t sufficient boundaries for agents with broad tool access</li>
</ul>
Google DeepMind’s AI Control Roadmap treats agents as potential insider threats and layers AI-powered supervisors that monitor agent reasoning in real time, scaling from asynchronous transcript review for low-risk actions to synchronous blocking for high-risk ones. They’ve analyzed one million coding agent trajectories to refine behavioral detection. That’s the direction serious security work is heading.
The key reframe from security researcher Ben Hanson: stop asking “what control failed?” and start asking “what about the system’s structure allowed this behavior?”
<hr>
<h2>The context problem</h2>
A study titled “AI Agents Do Not Fail Alone: The Context Fails First” ran 300 tests across 7,500 turns and found that agents fail most often not because of model limitations but because of bad context: vague instructions, poor tool descriptions, missing factual support, inconsistent rules. The same fixed models performed significantly better when context was structured rather than vague. More factual support correlated with fewer hallucinations. Clearer tool descriptions correlated with better tool use.
One counterintuitive finding: adding more safety rules didn’t always improve outcomes. Hardened agents sometimes became too cautious, which created its own failure mode.
This connects to the <a href="http://MEMORY.md" target="_blank" rel="noopener nofollow">MEMORY.md</a> debate. Claude Code’s auto-memory feature accumulates context across sessions, which sounds useful but causes subtle behavioral drift over time. The argument for turning it off: stateless agents are more predictable. If you want the agent to know something, put it in <a href="http://CLAUDE.md" target="_blank" rel="noopener nofollow">CLAUDE.md</a> explicitly, where you control it. Accumulated memory that the agent manages itself is context you can’t audit.
<hr>
<h2>The verification bottleneck</h2>
Code generation is no longer the bottleneck. Verification is.
AI tools increase PR volume and size. Code review load is going up everywhere, and nobody has a clean solution. The widely held belief that AI has shifted the bottleneck from coding to code review misses something: over 90% of teams ship in batches, so changes accumulate after code review in deployment queues. Speeding up reviews just pushes pressure to the next constraint — manual verification steps, change approval processes, deployment friction.
The actual fix is making validation as cheap and parallelizable as deployment itself. Per-PR isolated validation against live dependencies, using a shared Kubernetes cluster with header-based traffic routing, gives each change a lightweight ephemeral environment in seconds. This restores meaningful confidence signals regardless of whether code was human- or agent-authored.
For platform teams, the implication is significant: environments need to be rethought as a serving system, judged on latency, concurrency, and marginal cost. A 100-developer org where each engineer runs a few agent sessions generates hundreds of environment requests daily. Ticket-based or shared staging models can’t handle that.
<hr>
<h2>What humans are for now</h2>
Addy Osmani’s framing of “light” versus “dark” software factories is useful here. Light factories have humans reviewing before shipping. Dark factories are fully automated — no human reads the code. The problem with dark factories isn’t that they fail immediately; it’s that they accumulate comprehension debt silently while tests stay green, until eventually nobody understands the system.
The bottleneck in agentic development is verification, not generation. Keeping humans in the outer loop means owning architecture, design decisions, and review gates — not every line of code.
This creates a real tension for junior developers. The repetitive coding tasks that traditionally built judgment and taste are being automated away. CS graduates face 6-7.5% unemployment; junior tech job postings have dropped 34% since 2020. The skills that remain durable are the ones AI can’t replicate: knowing what to build, knowing whether it’s good, finishing the last mile, solving genuinely hard problems.
Some concrete practices for building those skills deliberately:
<ul>
<li>Read more code than you generate</li>
<li>Keep a log of agent mistakes and what caused them</li>
<li>Do some things the hard way on purpose</li>
<li>Go deep on one system end-to-end</li>
<li>Learn to specify and verify separately — these are different skills</li>
<li>Build an eval framework for AI-generated PRs</li>
</ul>
The engineers who are thriving right now are the ones who spent years automating their own work — better lint rules, CI pipelines, test suites, environment setup. Those instincts transfer directly. Building the infrastructure that steers agents toward correct behavior multiplies output across entire teams, not just individuals. Encoding domain knowledge as automation rather than keeping it in people’s heads is the new path to senior-level impact.
<hr>
<h2>The engineering progression</h2>
The trajectory is clear even if the destination isn’t:
Prompt engineering → context engineering → harness engineering → loop engineering → graph engineering
Most teams are somewhere in the middle of this progression. The ones moving fastest aren’t chasing the best model — they’re building better evaluation discipline, tighter delegation frameworks, and more trustworthy infrastructure around whatever model they’re using.
The organizations that succeed won’t have the best demos. They’ll have the best systems for knowing when their agents are wrong.

Collections

A survey of ~1,000 engineers and observations from the AI Engineer World's Fair 2026 reveal that 95% of teams now use agents, with write-enabled agents jumping from 52% to 89% adoption in a year. Yet full autonomy hasn't followed — human-in-the-loop governance remains dominant. The real engineering challenge has shifted from building agents to building the outer loops, checkpoints, and recovery systems around them. Coding agents like Claude Code, Cursor, and Codex are replacing the IDE as the primary development interface, raising concerns about long-term liability for AI-generated codebases. Enterprise adoption is emerging through 'forward deployed engineers' embedded in business units, and 'skills' — portable, declarative agent capabilities — are gaining traction as a reuse pattern. Model selection debates have faded; reliability, state management, and governance are now the hard problems.

AI engineering in 2026: agents are everywhere, but humans still run the loops