--- title: Perception agents: how Amazon AGI Lab is closing the... description: Amazon AGI Lab's Antje Barth introduced 'perception agents' at the AI Engineer Worlds Fair, arguing that current AI agents fail at end-to-end knowledge work... canonical: https://daily.dev/posts/perception-agents-how-amazon-agi-lab-is-closing-the-perceive-plan-act-loop-rszhxbrsl twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Perception agents: how Amazon AGI Lab is closing the perceive-plan-act loop | daily.dev og:description: Amazon AGI Lab's Antje Barth introduced 'perception agents' at the AI Engineer Worlds Fair, arguing that current AI agents fail at end-to-end knowledge work... og:url: https://daily.dev/posts/perception-agents-how-amazon-agi-lab-is-closing-the-perceive-plan-act-loop-rszhxbrsl og:image: https://api.daily.dev/og/posts/rSZhXbrsL.png og:image:alt: Perception agents: how Amazon AGI Lab is closing the perceive-plan-act loop og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Perception agents: how Amazon AGI Lab is closing the perceive-plan-act loop **[Collections](https://daily.dev/sources/collections)** · 3 min read · 1 upvotes · 0 comments ## Summary Amazon AGI Lab's Antje Barth introduced 'perception agents' at the AI Engineer Worlds Fair, arguing that current AI agents fail at end-to-end knowledge work because they can't perceive rendered screens or verify their own actions. Unlike coding agents where outputs are verifiable by running code, most knowledge work lacks clear success signals. Perception agents aim to close the perceive-plan-act loop by enabling agents to read rendered screens, confirm action results, and accept visual annotations as input. Amazon has open-sourced two components: a Chrome extension for screen annotation and a verification tool that checks agent work against design specs using visual comparisons. A demo also showed audio perception via wearable transcription devices feeding live meeting content into agent workflows. ## Content ## The problem with current AI agents AI agents have gotten surprisingly good at individual tasks — clicking buttons, calling APIs, writing code. But they keep falling apart on end-to-end knowledge work. Antje Barth from Amazon's AGI Lab has a clear diagnosis for why: agents act without perceiving. They dispatch actions into the void and have no reliable way to confirm what actually happened. Coding agents were the exception, not the rule. Code is verifiable — you run it and see if it works. Most knowledge work isn't like that. Writing a report, filling out a form, navigating a multi-step workflow — there's no automatic test suite to tell the agent whether it succeeded. The fix Barth proposes is completing what she calls the perceive-plan-act loop. Right now agents mostly plan and act. Perception — actually reading the rendered screen, confirming results, catching errors — is the missing piece. ## What perception agents actually do differently The core idea is that agents should interact with rendered screens the way humans do, rather than relying solely on DOM structure or API responses. This means: - Reading what's visually present on screen after each action - Accepting precise visual annotations as input instead of verbose text descriptions - Verifying their own outputs against what actually appeared Amazon has open-sourced two initial components to support this. The first is a Chrome extension for screen annotation — letting agents receive and interpret visual markup on what they're looking at. The second is a verification tool that checks agent work against design specs through visual comparisons and automated user flow walkthroughs. There's also an audio angle: wearable transcription devices can feed meeting discussions directly into the agent workflow, so perception isn't limited to screens. ## OS-level actions in Bedrock AgentCore Browser On the infrastructure side, Amazon has extended Bedrock AgentCore Browser with OS Level Actions — and this is where the perception loop gets real teeth. Previously, browser automation through Playwright and CDP could only reach the web layer: DOM elements, JavaScript, network requests. That's fine until you hit a native OS dialog, a certificate chooser, a right-click context menu, or a keyboard shortcut that the browser itself doesn't expose. Those interactions were simply out of reach. OS Level Actions change that. The agent now operates via an action-screenshot-reaction loop at the full desktop level: 1. Dispatch an action (mouse click, drag, scroll, key press, shortcut) 2. Capture a full-desktop screenshot 3. Send it to a vision model 4. Decide the next step Eight actions are supported across three categories: mouse control (mouseClick, mouseMove, mouseDrag, mouseScroll), keyboard input (keyType, keyPress, keyShortcut), and visual capture (screenshot). It's available through the InvokeBrowser API with no additional setup required for existing browser configurations. ## Why this matters together These two pieces — perception agents as a design philosophy, OS-level actions as infrastructure — are solving the same underlying problem from different angles. Agents that can only act are brittle. Agents that act, then look at what happened, then decide what to do next are something closer to actually useful. The honest caveat is that we're still early. Open-sourcing a Chrome extension and a verification tool is a starting point, not a finished system. But the framing is right: the gap in current agents isn't raw capability, it's the feedback loop. Perception is what closes it. ## Similar posts on daily.dev - [Amazon Bedrock AgentCore Browser adds OS-level interaction capabilities](https://daily.dev/posts/amazon-bedrock-agentcore-browser-adds-os-level-interaction-capabilities-dni3tekyi) · AWS · 1 upvotes · 0 comments - [Why Amazon hates 'human-in-the-loop' AI governance](https://daily.dev/posts/why-amazon-hates-human-in-the-loop-ai-governance-mydazda8o) · The Register · 1 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#amazon](https://daily.dev/tags/amazon), [#computer-vision](https://daily.dev/tags/computer-vision), [#multimodal](https://daily.dev/tags/multimodal) [View this post on daily.dev](https://daily.dev/posts/perception-agents-how-amazon-agi-lab-is-closing-the-perceive-plan-act-loop-rszhxbrsl) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Perception agents: how Amazon AGI Lab is closing the perceive-plan-act loop","url":"https://daily.dev/posts/perception-agents-how-amazon-agi-lab-is-closing-the-perceive-plan-act-loop-rszhxbrsl","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/perception-agents-how-amazon-agi-lab-is-closing-the-perceive-plan-act-loop-rszhxbrsl"},"datePublished":"2026-07-30T15:27:28.220Z","dateModified":"2026-08-06T10:39:55.073Z","description":"Amazon AGI Lab's Antje Barth introduced 'perception agents' at the AI Engineer Worlds Fair, arguing that current AI agents fail at end-to-end knowledge work...","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/perception-agents-how-amazon-agi-lab-is-closing-the-perceive-plan-act-loop-rszhxbrsl","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,amazon,computer-vision,multimodal","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"Perception agents: how Amazon AGI Lab is closing the perceive-plan-act loop"}]} ```