--- title: AI agents still struggle badly at long-horizon tasks,... description: A Microsoft research paper tested eight leading AI models, including GPT-5.6 Sol and Claude Opus 4.8, on a simulated year-long task with interconnected... canonical: https://daily.dev/posts/ai-agents-still-struggle-badly-at-long-horizon-tasks-new-research-shows-pxleh28n9 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: AI agents still struggle badly at long-horizon tasks, new research shows | daily.dev og:description: A Microsoft research paper tested eight leading AI models, including GPT-5.6 Sol and Claude Opus 4.8, on a simulated year-long task with interconnected... og:url: https://daily.dev/posts/ai-agents-still-struggle-badly-at-long-horizon-tasks-new-research-shows-pxleh28n9 og:image: https://api.daily.dev/og/posts/PxLEH28N9.png og:image:alt: AI agents still struggle badly at long-horizon tasks, new research shows og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # AI agents still struggle badly at long-horizon tasks, new research shows **[Collections](https://daily.dev/sources/collections)** · 5 min read · 1 upvotes · 0 comments ## Summary A Microsoft research paper tested eight leading AI models, including GPT-5.6 Sol and Claude Opus 4.8, on a simulated year-long task with interconnected decisions and delayed feedback. The best performer, Qwen3.7-Max with Hermes, ended with only 27.3% of the money an average human participant accumulated. The findings suggest current agents struggle specifically with long-horizon reliability: handling delayed feedback, recovering from earlier mistakes, and staying coherent across hundreds of steps — problems that short benchmarks and one-off successful runs fail to expose. ## Content ## The core problem There's a pattern worth paying attention to across a bunch of recent papers on AI agents: models that look capable in short bursts fall apart when you actually run them for a while. The Qwen team's FM-Bench is a good example. They had 15 frontier models manage a football club for 20 simulated years, across 340-400 decision points involving transfers, contracts, cash, and rival behavior. The year-5 rankings correlated just 0.19 with the final standings. DeepSeek-V4-Pro led at years 5 and 10 and finished 12th. The lesson: short-task success is a weak proxy for sustained decision quality, and benchmarks that end too early may be stopping before the real failures begin. The Qwen paper on long-horizon financial simulation makes this even starker. Eight leading models, including GPT-5.6 Sol and Claude Opus 4.8, were given a year of interconnected decisions with delayed feedback and consequences from their own past actions. The best-performing setup, Qwen3.7-Max with Hermes, ended with 27.3% as much money as the average human participant. A system finishing a year-long task at barely a quarter of human performance isn't close to dependable. ## The controller problem One thing these failures have in common: it's not always the model doing the work that breaks down. Often it's whatever is managing the work. LoopArena tested this directly. They fixed Qwen3.7-Plus as the coding worker and varied only the controller deciding what the worker should do next. On full 27-task runs, even the best controller (GPT-5.5) hit just 24.69% strict success rate. Simply restating the original goal every round scored 18.52%, the same as running the worker with no controller at all. Useful control has to react to what's actually happening, shifting the worker between implementation, verification, recovery, and stopping rather than just saying "keep going." A strong coding model isn't enough if the model managing it doesn't know when to redirect or stop. The Amazon/Microsoft SPACE paper makes a related point from the other direction: agents shouldn't need an LLM decision after every tiny action, but they also shouldn't blindly batch actions together. SPACE learns from successful trajectories which actions can safely run together, converting them into programmatic skills. On ScienceWorld, success rose from 35.9% to 67.2% while average LLM rounds dropped from 10.2 to 5.2. ## What practitioners are actually doing Matt Shumer shared a practical approach he calls the Manager Loop, developed while working with Astra on long-horizon coding tasks. The basic structure: a manager agent builds a checklist, breaks it into phases, then spawns a separate implementer agent. The manager runs each phase on the implementer sequentially, and they message each other until everything is done. The reason it helps is that Astra (and probably most models) tends to plateau over long tasks, getting stuck in small details while overall progress stalls. Forcing phase-by-phase work keeps momentum. A few details that seemed to matter in practice: - Asking for each phase to be done "extremely well" rather than "perfectly" worked better. "Perfectly" sent the model back into minutiae; "extremely well" implied it could move on once things were good enough. - Having the implementer maintain a simple HTML checklist with a progress chart helped the model notice when it hadn't made progress and move on. - Running 96 sub-agents in parallel was part of what made the approach scale. This is essentially how a human would steer a model, except the model is doing the steering. ## Hallucination and reliability in research agents A Google paper on autonomous AI research found that even when a final paper looks convincing, the underlying process can be badly wrong. Severe result hallucinations appeared in 90% of Agent Laboratory papers and 46% of Co-Scientist papers when reliability modules were removed. With Co-Scientist checking manuscript claims against actual execution logs, that rate dropped to 4%, and complete data fabrication fell to 0%. Separately, a team that gave AI agents 6 days and $3,000 to produce research papers had both papers rejected. The researchers who had spent months on similar work were not impressed. Producing something that looks like a paper and producing something that holds up are different things. ## Memory and context Long-running agents don't just need bigger context windows. They need to learn what deserves to stay in context at all. A few papers have looked at whether graph memory beats flat retrieval for long-term agents, and the results suggest it can help, though the gains depend heavily on the task structure. ## How to evaluate agent systems The READY framework from Scale AI and UC argues that enterprise teams should measure the human-AI system, not just the agent in isolation. Two agents can score nearly the same on a benchmark while requiring very different amounts of human review to reach the reliability a workflow actually needs. The useful question isn't "what's the accuracy?" but "what does it cost to make this reliable enough to deploy?" For long-running agents specifically, that means benchmarking the controller separately from the worker, running evaluations long enough to see compounding failures, and being honest about what "success" means when the task takes months rather than minutes. ## Questions this post answers ### How well do AI agents perform on long-horizon tasks compared to humans according to recent research? The best-performing agent setup, Qwen3.7-Max combined with Hermes, ended a simulated year-long task with only 27.3% as much money as the average human participant. Eight leading models were tested, including GPT-5.6 Sol and Claude Opus 4.8, and all struggled with delayed feedback, recovering from earlier mistakes, and staying coherent across hundreds of sequential decisions. _Anyone evaluating agent reliability for production use can track findings like this on daily.dev._ ### Why do short AI agent benchmarks fail to reveal long-horizon reliability problems? Short benchmarks and single successful runs do not expose the compounding failure modes that appear over long-horizon tasks, such as handling delayed feedback and recovering from earlier mistakes across hundreds of steps. A year-long simulated task tested by Microsoft researchers showed these structural weaknesses clearly, since consequences accumulated from the agent's own past decisions rather than resetting each run. _Teams designing agent evaluations follow research like this on daily.dev before trusting a benchmark._ ## Community take How the wider developer community reacted, aggregated from 2 discussions and 7 comments across x (as of 2026-09-05). **TL;DR:** Commenters largely agree that raw benchmark accuracy is misleading, arguing that the real cost of deploying agents lies in human oversight, babysitting, and debugging unpredictable failures rather than in headline scores. **Sentiment:** 5% positive · 25% mixed · 70% skeptical **The pushback** - Benchmark leaderboard performance rarely reflects the true operational cost of human review needed to catch agent mistakes. - Enterprise budgets care more about the labor cost of babysitting agents than about scores like MMLU. - A major hidden cost is debugging non-deterministic failures in production when it's unclear why an agent took a given action. **By community** - x (skeptical): Replies converge on skepticism that benchmark scores matter, emphasizing hidden human-oversight and debugging costs as the real barrier to deployment. **Open questions** - How should evaluation frameworks quantify and standardize the human-oversight cost of deploying agents at scale? **Highlights** > @rohanpaul_ai The cost of the human review wrapped around an agent is the deployment metric that actually reaches the budget meeting, and benchmark accuracy rarely survives the trip. > — [ShinkaIoT on x](https://x.com/ShinkaIoT/status/2096073845291573433) > @rohanpaul_ai ranking by deployment cost misses the point because the real bottleneck is debugging non-deterministic failures in production when nobody knows why the agent took a weird route. > — [zerogtalent on x](https://x.com/zerogtalent/status/2096152389451649217) > @rohanpaul_ai This is such a necessary shift. Benchmarks often ignore the real-world operational overhead, so factoring in the true cost of human oversight is the only way to actually scale these agents. > — [sirshibaninja on x](https://x.com/sirshibaninja/status/2096087608711585864) > @rohanpaul_ai Enterprise budgets care way more about babysitting costs than MMLU scores. > — [gurtej\_\_gill\_ on x](https://x.com/gurtej__gill_/status/2096097362850836594) **Source threads** - [x](https://x.com/rohanpaul_ai/status/2095972497443414089) · 0 points · 0 comments - [x](https://x.com/rohanpaul_ai/status/2096069941405651374) · 0 points · 7 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#claude](https://daily.dev/tags/claude), [#gpt](https://daily.dev/tags/gpt) [View this post on daily.dev](https://daily.dev/posts/ai-agents-still-struggle-badly-at-long-horizon-tasks-new-research-shows-pxleh28n9) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"AI agents still struggle badly at long-horizon tasks, new research shows","url":"https://daily.dev/posts/ai-agents-still-struggle-badly-at-long-horizon-tasks-new-research-shows-pxleh28n9","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/ai-agents-still-struggle-badly-at-long-horizon-tasks-new-research-shows-pxleh28n9"},"datePublished":"2026-08-29T14:04:45.522Z","dateModified":"2026-09-05T08:57:58.957Z","description":"A Microsoft research paper tested eight leading AI models, including GPT-5.6 Sol and Claude Opus 4.8, on a simulated year-long task with interconnected...","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/ai-agents-still-struggle-badly-at-long-horizon-tasks-new-research-shows-pxleh28n9","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,ai-agents,claude,gpt","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"AI agents still struggle badly at long-horizon tasks, new research shows"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/ai-agents-still-struggle-badly-at-long-horizon-tasks-new-research-shows-pxleh28n9#faq","mainEntity":[{"@type":"Question","name":"How well do AI agents perform on long-horizon tasks compared to humans according to recent research?","acceptedAnswer":{"@type":"Answer","text":"The best-performing agent setup, Qwen3.7-Max combined with Hermes, ended a simulated year-long task with only 27.3% as much money as the average human participant. Eight leading models were tested, including GPT-5.6 Sol and Claude Opus 4.8, and all struggled with delayed feedback, recovering from earlier mistakes, and staying coherent across hundreds of sequential decisions. Anyone evaluating agent reliability for production use can track findings like this on daily.dev."}},{"@type":"Question","name":"Why do short AI agent benchmarks fail to reveal long-horizon reliability problems?","acceptedAnswer":{"@type":"Answer","text":"Short benchmarks and single successful runs do not expose the compounding failure modes that appear over long-horizon tasks, such as handling delayed feedback and recovering from earlier mistakes across hundreds of steps. A year-long simulated task tested by Microsoft researchers showed these structural weaknesses clearly, since consequences accumulated from the agent's own past decisions rather than resetting each run. Teams designing agent evaluations follow research like this on daily.dev before trusting a benchmark."}}]} ```