--- title: AI Software Development – What Does The Data Say? description: A curated roundup of recent peer-reviewed and industry research on LLMs in software development argues that fully autonomous, reliable long-horizon agentic... canonical: https://daily.dev/posts/ai-software-development-what-does-the-data-say--y7gdakum5 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: AI Software Development – What Does The Data Say? | daily.dev og:description: A curated roundup of recent peer-reviewed and industry research on LLMs in software development argues that fully autonomous, reliable long-horizon agentic... og:url: https://daily.dev/posts/ai-software-development-what-does-the-data-say--y7gdakum5 og:image: https://api.daily.dev/og/posts/y7GdaKUM5.png og:image:alt: AI Software Development – What Does The Data Say? og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # AI Software Development – What Does The Data Say? **[Lobsters](https://daily.dev/sources/lobsters)** · 7 min read · 0 upvotes · 0 comments ## Summary A curated roundup of recent peer-reviewed and industry research on LLMs in software development argues that fully autonomous, reliable long-horizon agentic coding remains implausible. Key findings cited: effective context limits are far smaller than advertised due to lossy 'compression'; models can't reliably prioritize recent context over training-time 'dominant priors'; repo-level .md context files often hurt performance; negation instructions are unreliable; demonstrations outperform descriptions; large industry studies (CircleCI, Faros, DORA) show more code output but worse, slower outcomes overall except for teams with strong existing engineering practices; psychological research links AI reliance to reduced critical thinking and burnout; and achieving an order-of-magnitude reliability gain would require ~10^20 times current compute, meaning models will likely grow more powerful but not more reliable soon. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why do AI coding agents perform worse when given repo-level markdown files with coding standards and architecture summaries? Repository-level markdown files, especially ones generated by AI models themselves, tend to add noise rather than useful signal for many specific coding tasks, degrading model performance. This suggests that including a team's full coding standards and architecture summary in every task prompt is often counterproductive rather than helpful for agentic coding tools. _Anyone tuning prompts and repo context for coding agents can track findings like this on daily.dev._ ### How much more compute would it take to make LLMs significantly more reliable at coding tasks? Making a large language model an order of magnitude more reliable, for example reducing errors from 30% to 3%, would require roughly 10^20 times the energy and compute that current frontier models need to train. This means major reliability gains are unlikely soon, and improvements will more likely come from better context engineering and output quality gates rather than bigger models. _Developers weighing how far to trust AI coding tools can follow this kind of reliability research on daily.dev._ ### Do industry studies show AI coding tools actually improving software development outcomes? Large-scale industry studies including DORA's 2025 report, Faros AI's Acceleration Whiplash report, and CircleCI's analysis of 28 million workflows show that while code output metrics like commits and diff size are rising, overall outcomes are not improving and average teams may be taking longer to ship worse software. A small percentage of teams see modest gains, correlated with already-strong existing development capability, suggesting AI amplifies rather than fixes team strengths and weaknesses. _Teams deciding how to measure real AI coding impact can follow this outcomes data on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 3 discussions and 9 comments across hackernews, lobsters (as of 2026-09-14). **TL;DR:** Reaction is dominated by pushback on the post's framing and rigor rather than its underlying conclusions, with several commenters agreeing LLMs have real limits but faulting the piece for unsourced claims and an anti-AI tone. **Sentiment:** 5% positive · 35% mixed · 60% skeptical **The case for** - One commenter agreed LLMs are useful for boilerplate and narrow, well-scoped tasks but not as a full replacement for human design/architecture work. - Another agreed models are getting better at solving problems even if error rates aren't clearly improving. **The pushback** - The opening framing reads as strongly anti-LLM, which undermines trust in the rest of the argument for some readers. - Several claims (e.g. about .md context files or benchmark saturation) are seen as shallow, missing nuance about how these techniques are actually used in practice. - Repeated criticism that psychological/cognitive research claims are cited vaguely ('multiple studies') without links to the actual papers or publication dates, making them hard to verify. - One reply noted the huge energy cost of current, already-unreliable models as an additional concern beyond the reliability scaling argument. **By community** - hackernews (mixed): Very light engagement, with the one substantive reply flagging the enormous energy cost implied by the reliability-scaling claim. - lobsters (skeptical): Discussion leans critical of the post itself, questioning its tone, rigor, and lack of citations, even as some commenters share its skepticism about LLM coding limits. **Hottest debate:** Whether the post is a genuine, well-supported analysis or an emotionally slanted rant dressed up as data-driven research. **Open questions** - Which specific studies support the claimed link between AI reliance and reduced critical thinking/cognition? - How exactly were the poorly-performing repo-level .md context files generated, and would better practices change the outcome? **Highlights** > There's something about the tone of this post (and others like it) that immediately discredits it to me. From the first point: > Truly autonomous and reliable long-horizon agentic software development is so highly improbable using LLMs that it’s essentially science fiction. The way this sentence is written makes me feel immediately that the author is already very anti-LLM for software. The rest of the points seem to corroborate that, e.g. "In case you were wondering why some of your guardrails are about as reliable as a coin-toss.". Starting with such a strong emotional stance means I'm unlikely to trust what you're about to tell me. I point this out because I assume the author is making a genuine attempt to change people's minds, and they've gone to all this work to read blog posts and papers and collate actual data, only to lose their intended audience (people who *are* overexcited about AI) in the first sentence. Or, maybe it's just intended as a rant. In which case, rant on! > — [Brids on lobsters · 10 points, 1 comments](https://lobste.rs/s/joyawz/ai_software_development_what_does_data#c_k9ocdj) > I don't think it's even a genuine attempt. The points are at a level of "I heard about this idea" without understanding the actual limits and existing/popular ways to deal with them. It would take too much effort to address them all, but for example: > Repo-level .md files tend to make model performance worse, probably because they add noise instead of signal in many specific tasks. Model-generated .md files are especially problematic in this respect, it seems. I know which article this references, but that article didn't dig into how the files are generated. They talked mostly about a simple unguided generation, which is going to perform way worse than closing a long session for an often repeated task in the repo with "Which information in this session took many tool calls to find? What lookups started in the wrong place? Note the correct information very succinctly in agents.md". For more specific tasks you can reference both a task and more specific file to read so you don't affect the context too much with the noise they mention. But badly using a feature gives you bad results - that's not specific to LLMs. The author also doesn't follow the development of benchmarks where models do get overly optimised for, but that's why we keep getting new benchmarks. One of the biggest changes recently was Deepswe aligning much more with the real dev experience. And we will keep going through that cycle many more times. > — [viraptor on lobsters · 3 points](https://lobste.rs/s/joyawz/ai_software_development_what_does_data#c_mc3gpf) > This article would be a lot more useful if it directly linked the summarized claims to the underlying papers - and included dates of publication for those papers at the same time. > The psychological and cognitive factors in LLM use are a growing field of serious research. One study found a significant correlation between confidence in AI output and belief in the paranormal. Multiple studies found a negative impact on learning, cognition and critical thinking with greater LLM reliance. *Which* "multiple studies"? Are they recent and credible? If you're going to summarize their findings you still need to link to them. > — [simonw on lobsters · 2 points, 1 comments](https://lobste.rs/s/joyawz/ai_software_development_what_does_data#c_xvjg4t) > That matches my experience pretty closely. LLMs are good at filling in the gaps, and doing boiler plate, but you still have to do design and architecture the same way you always did. These things are applicable in fairly narrow contexts. It takes a lot of trial and error to develop intuition for how to actually use them effectively, and where it makes sense to apply them in a project. I'd argue that having a clean architecture with low coupling becomes extremely important when using LLMs. As the article points out, they do best on smaller tasks that don't have dependencies. So, if you can break up your project into small pieces that can be worked on in isolation, then you can give the agent a task with clear boundaries. And that also makes it much easier to review the output as well. I find functional style actually maps to this extremely well because there's so much focus on context isolation and passing state around explicitly. The same tricks that make large code bases manageable by humans also help LLMs for the same reasons. This tech is not a replacement for human developers, it's just another tool in programmer's toolbox. > — [Yogthos on lobsters · 4 points](https://lobste.rs/s/joyawz/ai_software_development_what_does_data#c_xpz0j3) > > The energy and compute needed to train an LLM to be an order of magnitude more reliable – e.g., wrong 3% of the time instead of 30% – is 10^20 times what the current frontier models require. And the energy they require to be 30% wrong is already astronomical. > — [goodthink on hackernews](https://news.ycombinator.com/item?id=49310675) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49310392) · 5 points · 2 comments - [hackernews](https://news.ycombinator.com/item?id=49328248) · 2 points · 0 comments - [lobsters](https://lobste.rs/s/joyawz/ai_software_development_what_does_data) · 22 points · 7 comments ## Similar posts on daily.dev - [Why Some Companies Are Pulling Back on AI Coding](https://daily.dev/posts/why-some-companies-are-pulling-back-on-ai-coding-pfrs6ovwx) · SD Times · 2 upvotes · 1 comments - [Understanding The Limits of AI in Software Development](https://daily.dev/posts/understanding-the-limits-of-ai-in-software-development-2ituxdrgp) · Organizing Automation · 1 upvotes · 0 comments - [The Future of Software Development is Software Developers](https://daily.dev/posts/the-future-of-software-development-is-software-developers-clpbexose) · Hacker News · 8 upvotes · 2 comments - [Why I remain a skeptic](https://daily.dev/posts/why-i-remain-a-skeptic-bpv99cftw) · Lobsters · 0 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#context-engineering](https://daily.dev/tags/context-engineering) [View this post on daily.dev](https://daily.dev/posts/ai-software-development-what-does-the-data-say--y7gdakum5) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"AI Software Development – What Does The Data Say?","url":"https://daily.dev/posts/ai-software-development-what-does-the-data-say--y7gdakum5","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/ai-software-development-what-does-the-data-say--y7gdakum5"},"datePublished":"2026-08-17T18:17:18.525Z","dateModified":"2026-09-14T08:21:01.719Z","description":"A curated roundup of recent peer-reviewed and industry research on LLMs in software development argues that fully autonomous, reliable long-horizon agentic...","image":"https://i0.wp.com/codemanship.wordpress.com/wp-content/uploads/2026/08/evidence.png?fit=1200%2C675&ssl=1","thumbnailUrl":"https://i0.wp.com/codemanship.wordpress.com/wp-content/uploads/2026/08/evidence.png?fit=1200%2C675&ssl=1","isAccessibleForFree":true,"articleSection":"Lobsters","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Lobsters","logo":"https://media.daily.dev/image/upload/s--tl8v_Fku--/f_auto,t_logo/v1698841318/logos/lobste.jpg","url":"https://daily.dev/sources/lobsters"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/ai-software-development-what-does-the-data-say--y7gdakum5","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,llm,ai-agents,context-engineering","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Lobsters","item":"https://daily.dev/sources/lobsters"},{"@type":"ListItem","position":3,"name":"AI Software Development – What Does The Data Say?"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/ai-software-development-what-does-the-data-say--y7gdakum5#faq","mainEntity":[{"@type":"Question","name":"Why do AI coding agents perform worse when given repo-level markdown files with coding standards and architecture summaries?","acceptedAnswer":{"@type":"Answer","text":"Repository-level markdown files, especially ones generated by AI models themselves, tend to add noise rather than useful signal for many specific coding tasks, degrading model performance. This suggests that including a team's full coding standards and architecture summary in every task prompt is often counterproductive rather than helpful for agentic coding tools. Anyone tuning prompts and repo context for coding agents can track findings like this on daily.dev."}},{"@type":"Question","name":"How much more compute would it take to make LLMs significantly more reliable at coding tasks?","acceptedAnswer":{"@type":"Answer","text":"Making a large language model an order of magnitude more reliable, for example reducing errors from 30% to 3%, would require roughly 10^20 times the energy and compute that current frontier models need to train. This means major reliability gains are unlikely soon, and improvements will more likely come from better context engineering and output quality gates rather than bigger models. Developers weighing how far to trust AI coding tools can follow this kind of reliability research on daily.dev."}},{"@type":"Question","name":"Do industry studies show AI coding tools actually improving software development outcomes?","acceptedAnswer":{"@type":"Answer","text":"Large-scale industry studies including DORA's 2025 report, Faros AI's Acceleration Whiplash report, and CircleCI's analysis of 28 million workflows show that while code output metrics like commits and diff size are rising, overall outcomes are not improving and average teams may be taking longer to ship worse software. A small percentage of teams see modest gains, correlated with already-strong existing development capability, suggesting AI amplifies rather than fixes team strengths and weaknesses. Teams deciding how to measure real AI coding impact can follow this outcomes data on daily.dev."}}]} ```