--- title: AWS Releases Aws-Bench to Evaluate Agents on Cloud Tasks description: AWS has open-sourced aws-bench, a benchmark for evaluating AI agents on real AWS tasks like diagnosing misconfigurations, provisioning infrastructure, and... canonical: https://daily.dev/posts/aws-releases-aws-bench-to-evaluate-agents-on-cloud-tasks-0shw5bgj3 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: AWS Releases Aws-Bench to Evaluate Agents on Cloud Tasks | daily.dev og:description: AWS has open-sourced aws-bench, a benchmark for evaluating AI agents on real AWS tasks like diagnosing misconfigurations, provisioning infrastructure, and... og:url: https://daily.dev/posts/aws-releases-aws-bench-to-evaluate-agents-on-cloud-tasks-0shw5bgj3 og:image: https://api.daily.dev/og/posts/0sHW5bgj3.png og:image:alt: AWS Releases Aws-Bench to Evaluate Agents on Cloud Tasks og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # AWS Releases Aws-Bench to Evaluate Agents on Cloud Tasks **[InfoQ](https://daily.dev/sources/infoq)** · 3 min read · 1 upvotes · 0 comments ## Summary AWS has open-sourced aws-bench, a benchmark for evaluating AI agents on real AWS tasks like diagnosing misconfigurations, provisioning infrastructure, and operating live cloud environments. Unlike static-fixture benchmarks, it spins up disposable AWS accounts with CDK-defined resources, runs the agent in a sandboxed container with scoped credentials, and scores results with an LLM judge or programmatic checks against live AWS state. It's built on the Harbor framework and includes adapters for Claude Code, Codex, Kiro CLI, Mini-SWE-Agent, Gemini CLI, and OpenCode. Running it requires management-account-level credentials, is pinned to us-east-1, and can incur costs. No baseline results or leaderboard have been published yet. The release lands amid growing skepticism about benchmark trustworthiness, with UC Berkeley researchers showing an agent scoring near-perfect on Terminal-Bench and SWE-Bench without solving any tasks. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is aws-bench and how does it evaluate AI agents on AWS tasks? Aws-bench is an open-source benchmark released by AWS that evaluates AI agents on real AWS tasks such as diagnosing misconfigurations, provisioning infrastructure, and operating live cloud environments. Instead of static fixtures, it deploys scenarios in disposable, isolated AWS accounts using CDK stacks, then scores agent runs with an LLM judge or programmatic checks against live AWS state. _daily.dev surfaces releases like aws-bench for engineers evaluating AI agents on real cloud workloads._ ### Which AI coding agents does aws-bench support out of the box? Built-in adapters cover Claude Code, Codex, Kiro CLI, and Mini-SWE-Agent, and it also supports any agent already included in the underlying Harbor framework, such as Gemini CLI and OpenCode. Aws-bench extends Harbor, an open-source agent evaluation framework, adding AWS resource provisioning, scenarios, and verifiers on top. _Teams comparing coding agents can track which benchmarks each tool supports via daily.dev._ ### What are the requirements and costs of running the aws-bench benchmark? Running aws-bench requires credentials with access to an organization's AWS management account plus permission to manage member accounts and organizational units, not just a local install. The setup is currently pinned to the us-east-1 region and creates persistent resources that can incur costs even when unused, and AWS has not yet published baseline results or a leaderboard. _Engineers weighing the operational cost of agent benchmarking can follow updates like this on daily.dev._ ## Similar posts on daily.dev - [AWS announces aws-bench, an open-source benchmark for AI agents on AWS](https://daily.dev/posts/aws-announces-aws-bench-an-open-source-benchmark-for-ai-agents-on-aws-5nxftt7ae) · AWS · 1 upvotes · 0 comments - [Center for Responsible, Decentralized Intelligence at Berkeley](https://daily.dev/posts/center-for-responsible-decentralized-intelligence-at-berkeley-lkuvgfu7q) · Hacker News · 0 upvotes · 0 comments - [The Open Agent Leaderboard](https://daily.dev/posts/the-open-agent-leaderboard-vs2qmkwoy) · Hugging Face · 0 upvotes · 0 comments - [How We Benchmark Deep Agents](https://daily.dev/posts/how-we-benchmark-deep-agents-nppzx6htk) · LangChain · 1 upvotes · 0 comments --- Tags: [#cloud](https://daily.dev/tags/cloud), [#aws](https://daily.dev/tags/aws), [#ai-agents](https://daily.dev/tags/ai-agents) [View this post on daily.dev](https://daily.dev/posts/aws-releases-aws-bench-to-evaluate-agents-on-cloud-tasks-0shw5bgj3) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"AWS Releases Aws-Bench to Evaluate Agents on Cloud Tasks","url":"https://daily.dev/posts/aws-releases-aws-bench-to-evaluate-agents-on-cloud-tasks-0shw5bgj3","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/aws-releases-aws-bench-to-evaluate-agents-on-cloud-tasks-0shw5bgj3"},"datePublished":"2026-08-22T08:06:55.365Z","dateModified":"2026-09-14T07:18:33.075Z","description":"AWS has open-sourced aws-bench, a benchmark for evaluating AI agents on real AWS tasks like diagnosing misconfigurations, provisioning infrastructure, and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/63c15fd41f2f8e3d3065cf00649dacad?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/63c15fd41f2f8e3d3065cf00649dacad?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"InfoQ","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"InfoQ","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/afc3bced3e1e4b188dd9127017a60e0c","url":"https://daily.dev/sources/infoq"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/aws-releases-aws-bench-to-evaluate-agents-on-cloud-tasks-0shw5bgj3","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"cloud,aws,ai-agents","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"InfoQ","item":"https://daily.dev/sources/infoq"},{"@type":"ListItem","position":3,"name":"AWS Releases Aws-Bench to Evaluate Agents on Cloud Tasks"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/aws-releases-aws-bench-to-evaluate-agents-on-cloud-tasks-0shw5bgj3#faq","mainEntity":[{"@type":"Question","name":"What is aws-bench and how does it evaluate AI agents on AWS tasks?","acceptedAnswer":{"@type":"Answer","text":"Aws-bench is an open-source benchmark released by AWS that evaluates AI agents on real AWS tasks such as diagnosing misconfigurations, provisioning infrastructure, and operating live cloud environments. Instead of static fixtures, it deploys scenarios in disposable, isolated AWS accounts using CDK stacks, then scores agent runs with an LLM judge or programmatic checks against live AWS state. daily.dev surfaces releases like aws-bench for engineers evaluating AI agents on real cloud workloads."}},{"@type":"Question","name":"Which AI coding agents does aws-bench support out of the box?","acceptedAnswer":{"@type":"Answer","text":"Built-in adapters cover Claude Code, Codex, Kiro CLI, and Mini-SWE-Agent, and it also supports any agent already included in the underlying Harbor framework, such as Gemini CLI and OpenCode. Aws-bench extends Harbor, an open-source agent evaluation framework, adding AWS resource provisioning, scenarios, and verifiers on top. Teams comparing coding agents can track which benchmarks each tool supports via daily.dev."}},{"@type":"Question","name":"What are the requirements and costs of running the aws-bench benchmark?","acceptedAnswer":{"@type":"Answer","text":"Running aws-bench requires credentials with access to an organization's AWS management account plus permission to manage member accounts and organizational units, not just a local install. The setup is currently pinned to the us-east-1 region and creates persistent resources that can incur costs even when unused, and AWS has not yet published baseline results or a leaderboard. Engineers weighing the operational cost of agent benchmarking can follow updates like this on daily.dev."}}]} ```