With all respect to the creativity and tinkering of AI thought leaders like Peter Steinberger, there are limits with the transformer technology in LLMs that you cannot bypass with skills, loops, or reasoning. The main limit is the lack of logical and computational intelligence. In other words, an LLM never understands what it’s doing. Their responses are all about patterns, probabilities, and interpolation based on existing solutions in the training data. You might argue that this doesn’t matter because there are working and stunning examples. However, probability doesn’t mean that it can’t produce working results; it just means that it did not understand what it did, and that this isn’t just a nasty trade-off you can deal with on novel or business-critical problems. Even if you add detailed instructions and guardrails, the AI will never understand the intent and logic of a problem; it will just try to mirror your expectations through probabilistic math and interpolation. Which means it mimics understanding.
<h2>Transformer-based LLMs will always hallucinate</h2>
This lack of understanding is why AI coding in proprietary business applications doesn’t work reliably, and it’s also the reason why AI in production is a serious problem. The job of a software engineer is about logic and understanding, even in the implementation phase where code is written, because some issues are only visible when you code and require iterative design adjustments. A developer knows about the constraints and trade-offs of different approaches and about the environmental constraints of the software. The dev can make predictions and understand their consequences. LLMs can’t. When an LLM reasons, it always adds its own assumptions to the context based on probability and known patterns. It doesn’t understand your codebase, it doesn’t understand your problem, and it doesn’t understand your intent. It finds a similar problem and solution based on probabilistic math, and if the training data doesn’t cover your case, it interpolates a close-enough solution that might be right, which is the cause of hallucinated libraries, functions, etc. LLMs basically guess a solution with trial and error; especially in loops and agentic harnesses it feels like a brute-force approach. This also means that loops actually multiply the negative effects of assumptions and hallucinations and cost more money. Unfortunately, you can’t fix this with more compute and better harnesses.
<h2>A real-world example and why frontier LLMs failed</h2>
Despite knowing about the limits, as a curious engineer, I still test AI coding, and I had the perfect real-world challenge. Our managed mail server recently changed its behavior for non-existent virtual mailboxes. In the past, it accepted SMTP messages to non-existent internal email addresses and bounced them back via the mailer daemon. Now, I get a 550 SMTP error. This behavior is OK because I can generate a human-readable and understandable error message. However, the email client is proprietary, and it has a function that doesn’t exist in most other products, which allows me to send one email individually to each recipient. The system basically generates n copies of the same email with one recipient. The first solution seems to be to just wrap the sending method and catch the exception. But what if the first three emails passed, and the last one didn’t? The user would get an error message, but three of four messages were actually sent.
So, I used GPT 5.5 and Claude Opus 4.8 to try to solve this problem. First, I only told them that the server returns a 550 SMTP error when an internal email address doesn’t exist. And their solution? They added a try-catch. Problem solved? Well, technically, yes. There is no uncaught exception, all tests pass, and the user gets an error message. However, the sending behavior was not atomic because emails could still be partially delivered and the error messages lacked necessary details. I then rolled back, adjusted the prompt, and told them that they need to make sure they catch the errors before sending because the app allows sending n copies of the same email with the “send separately to each recipient” function. This time, they added the try-catch to the internal sending loop. Again, all tests passed, and there were no uncaught exceptions. However, the core issue remained. I tried another prompt, told them which mail library is used, and specified that they are allowed and need to adjust it to test the addresses before sending. Despite telling the AI agent about the mail library, it hallucinated the core functionality by assuming that this was an open-source library. After several attempts and new prompts, I was able to force the AI to use the actual logic and interfaces of my library and application; however, it failed to solve the problem and eventually fell back to weird assumptions and the initial try-catch solution. They just did not understand why and how they had to refactor the mail library to change its behavior to address the problem properly. After two hours, I gave up and coded the solution and tests by hand in less than an hour.
This means that the current best frontier LLMs were not able to solve my real-world problem in a production business application. They wasted my time and money. This is just one example that proves that agentic coding, skills, loops, etc., are not a magical solution for software development. Actually, they address something that was never the problem for experienced software engineers in real products. Especially not in the era of autocomplete, static analysis, intelligent IDEs, and CLI commands to address boilerplate. All those LLM-based approaches have the same thing in common: They ignore the technical limitations of LLMs and oversimplify real-world applications. I agree that a harness makes LLMs more usable for coding. They avoid broken code and tool calls. They also have routines to validate the response and let the AI automatically correct it. They can evaluate the output and try to ensure that the LLM doesn’t break out of the sandboxed workspace. All of this is true. However, the model behind the harness still doesn’t understand the logic of the problem and its own solution, and it doesn’t understand your intent. Even if the syntax is correct, all tests pass, and the code compiles, it doesn’t mean that the solution is correct and maintainable. And I am not alone here. Mitchell Hashimoto explained the same issue here:
<a href="https://x.com/mitchellh/status/2060088112257372610" target="_blank" rel="noopener nofollow">https://x.com/mitchellh/status/2060088112257372610</a>
<h2>LLMs in harnesses are useful assistants</h2>
No one knows everything, and sometimes we spend hours, days, weeks, or even months researching a technical solution. Books, docs, and search engines are sometimes no help because you don’t know the relevant keywords to search. Most documentation only helps if you already know that a solution exists and you want to make sure you use it correctly. LLMs address this issue for me. Because they are probabilistic and have a massive database, they can help me finding existing solutions or at least keywords based on my description. In other words, the probabilistic weakness in coding tasks is a strength in research. For almost every problem, there is an unknown solution on the internet that LLMs can find in their data and modify. This also explains why AI can produce functional apps and websites and why this works better from scratch, which is my second use case.
Everyone knows that “getting started” docs often fail to address new users. And maybe it’s just me, but I understand best if I read the code and try to run it. This means that a working demo application helps me understanding the app better than a 10-page “getting started” guide, written from the perspective of someone who already knows how it works, ever could. This lowers the barrier to starting something new that might help you solve problems you have not addressed before because your existing approaches were too time-consuming. It also helps you to test different versions and pick the best starting point for your next project. However, it’s still a starting point, not a final solution. And this might explain the hype, because many people scale working demos to actual products and miss the complexity and real-world problems of business apps in production, where coding is only a minor part of the actual job.
My third and final use case is for discussions and code review. I discuss my ideas and ask about flaws and alternative solutions. I let AI review parts of my code to find issues I might have missed. Since an LLM mirrors my expectations and knowledge level, it can simulate talking to an expert who knows a lot but misses important details. However, since I am talking to someone on my level, I can see these issues, and I can also see what’s actually useful. I said that no one knows everything, but LLMs are damn close. However, knowing and understanding are fundamentally different, and that’s the reason why I don’t use AI to generate production code, because that’s where understanding matters, not the speed of code generation.

Erik Hazington

A software engineer argues that transformer-based LLMs fundamentally lack logical and computational understanding, making them unreliable for production business applications. Using a real-world SMTP error-handling problem, the author demonstrates how GPT and Claude failed to grasp the core issue despite multiple prompt iterations, ultimately wasting two hours before solving it manually in under one hour. The post distinguishes between LLMs' probabilistic strengths (research, finding existing solutions, generating demo apps, code review discussions) and their fundamental weakness in novel production coding tasks requiring genuine understanding of intent, constraints, and trade-offs. Agentic harnesses and loops are criticized for multiplying hallucinations rather than fixing the underlying limitation.

AI thought leaders seem not to understand LLM limitations

I agree wholeheartedly. I used GitHub Copilot to migrate a data access library from using mongo DB to Postgres. Started with co-creating a migration plan with tests, schema design, and so forth. 90% of the migration went well, but then the LLM tried to implement the main fetching function.
It fetched all rows from the main table and then filtered it in Python.
All tests were green, even in the test deployment nothing would have looked odd. If I had not read the code, it would have crashed in production, when it tried to fetch several thousand rows. And let’s not talk about the amount of N+1 query patterns I found.
LLMs are useful, but the mistakes they make are subtle yet devastating.

I appreciate including a real world example in this post, I feel like many times people just say “AI can’t solve all real world problems” and leave it at that… actually including your example gives a great picture on the actual type of problems that LLMs still struggle with.

Ditto. LLM mimics understanding, but they are pattern machine. They can be great for narrow scope research or rapid testing, but they are never designed for for high level abstract logic, no matter how big tech hyped their capabilities.

I like hearing about these measured ways to use AI. Right now, I’m using it as a teacher to walk me through implementation so I can learn some patterns and webdev conventions. I agree that ‘getting started’ guides are usually hard to decipher and LLMs are a great use case for this.