unknown

An AWS developer advocate, Mike Chambers, explains why AI agents that impress in demos often fail in production, arguing the real problems lie in the 'harness' around the model rather than the model itself. Key points include treating observability and evaluations as top priorities from day one, separating memory extraction from the agent itself, using multi-agent systems mainly to isolate context rather than just to parallelize work, and building deployment infrastructure properly instead of cramming everything into one container. He closes by urging engineers to develop strong intuition for what LLMs can and can't do rather than chasing the latest model or SDK.

We Usually Blame the AI Model, But the Harness Is What Breaks Production

(Un)popular opinion written by engineers, first-hand advice on career or challenging projects, and thoughts from prominent industry voices. This is a place to share and comment the best of ShiftMag.

ShiftMag - Insightful Engineering Content

No, the harness cannot delete stuff. It’s the LLM deleting stuff.

I’d move memory extraction outside the request path before adding a second agent.

Separating memory extraction from the agent is a strong design choice; it makes behavior easier to inspect and evolve. I also like the point that multi-agent systems are mainly about context isolation, not just parallel work. Good evals and traces are usually more valuable than another orchestration layer.

There is no constraints to llms there is no simple yes/no , correct or not correct, it’s probability. Unlike a schema validator or compiler flag, there is no real hard constraints with llms. You can constrain the print/decode with a grammar to prevent what it produces but in the end everything is probability and your most rigid attempts are suggestions which may or may not be ignored.
The hype tries to wash this away with endless tools, abstractions, revisions to quantization. As the context window grows , and ambiguity slowly creeps in, each deviation from the expected expands to those tokens in the proximity, it is a cascading failure subtle at first but potentially fatal the longer the context and/or number of steps etc.
Think about it , a token gets encoded, assigned a position, determines relationships to all other words, may change its meaning during attention, moves through a ffn expanding or refining context, gets further normalized a cleanup step whatever this is, the target model takes this and predicts the next token(s) or the logit(s), then on to softmax that fits all the logits neatly into a 0…1 range (more math transform) then onto further transforms based on topk/topp/minp/temp
So many layers, transforms happening for each token where the initial stages can differ wildly based on how token relates to every other ,
Even if just one transform for one token was off, one encoding is off I am not talking totally off I mean in a precision shift, that has the potential to change things dramatically.
This is why it is disturbing to hear of companies buying up original copies of old books , ingesting and disposing of them, the ai is likely rewriting history and with each new model trained by its predecessor after enough iteration our history probably won’t resemble anything close to what it really was.

Agreed on the no-hard-constraints part. The line we drew was to keep the model out of the path wherever the output has to be trustworthy. Our SEC filing rows render from deterministic templates with no model call at ingest, and the model only suggests a category, which a resolver can overrule. What it does extract has to survive a check against the source text. A proposed ticker is dropped unless the company name or the symbol shows up in the body. About two dozen symbols that collide with a foreign listing or a currency need the full name or an exchange notation, because a bare IAG is Iamgold in our symbol table and International Airlines Group in half the articles that say it.