--- title: Perception Agents — Antje Barth, Amazon AGI Lab | daily.dev description: A talk from the AI Engineer Worlds Fair by an Amazon AGI Lab researcher introducing 'perception agents' — a new paradigm for AI agents that can perceive... canonical: https://daily.dev/posts/perception-agents-antje-barth-amazon-agi-lab-dpijkbp9u twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Perception Agents — Antje Barth, Amazon AGI Lab | daily.dev og:description: A talk from the AI Engineer Worlds Fair by an Amazon AGI Lab researcher introducing 'perception agents' — a new paradigm for AI agents that can perceive... og:url: https://daily.dev/posts/perception-agents-antje-barth-amazon-agi-lab-dpijkbp9u og:image: https://api.daily.dev/og/posts/DpIJkbp9U.png og:image:alt: Perception Agents — Antje Barth, Amazon AGI Lab og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Perception Agents — Antje Barth, Amazon AGI Lab **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 21 min read · 2 upvotes · 0 comments ## Summary A talk from the AI Engineer Worlds Fair by an Amazon AGI Lab researcher introducing 'perception agents' — a new paradigm for AI agents that can perceive rendered screens the way humans do, rather than just acting blindly. The core argument is that while agents have mastered individual tool use and clicking, they fail at end-to-end knowledge work because they lack shared visual context and the ability to verify their own outputs. Coding agents succeeded because code is verifiable; most knowledge work is not. Perception agents address this by completing the perceive-plan-act loop: reading rendered screens, confirming results, and accepting precise visual annotations as input instead of verbose text descriptions. Amazon has open-sourced two initial components — a Chrome extension for screen annotation and a verification tool that checks agent work against design specs through visual checks and automated user flow walkthroughs. A demo also shows audio perception via wearable transcription devices feeding meeting discussions directly into the agent workflow. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What are perception agents from Amazon AGI Lab? Perception agents are AI agents designed to perceive a rendered screen the way a human does - layout, state, and changes - rather than scraping underlying code, so they can confirm whether their own actions succeeded instead of just firing off actions and hoping. Amazon AGI Lab open sourced the first two components of this harness: an annotation tool and a verification tool. _Engineers building reliable computer-use agents can follow releases like this one on daily.dev._ ### What does the annotation tool in Amazon AGI Lab's perception agent harness do? It is a Chrome extension that lets a user visually mark up elements on a web page, such as circling a heading or selecting a section, and describe a change in a short phrase like 'double the font size' instead of writing a long text description. It captures the exact location and style feedback and produces a summary an agent can implement without further back and forth. _Anyone evaluating agent tooling for UI changes can track tools like this via daily.dev._ ### How does the verification tool check an AI agent's own work against design rules? It takes a design spec written in a markdown file describing colors, components, and layout rules, then runs two kinds of checks: a visual check confirming the output is on-brand and matches the layout, and a user-flow check where the agent walks through the app performing real user actions like adding or deleting a task. It produces a report listing which checks passed and which failed. _Teams deciding how to add self-verification to agent workflows can follow tooling updates on daily.dev._ ## Similar posts on daily.dev - [What are AI Agents?](https://daily.dev/posts/what-are-ai-agents--4jflfneiq) · ByteByteGo · 0 upvotes · 0 comments - [The End-to-End Agentic AI Pipeline](https://daily.dev/posts/the-end-to-end-agentic-ai-pipeline-gftp4wxu7) · Machine Learning Mastery · 2 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#human-computer-interaction](https://daily.dev/tags/human-computer-interaction) [View this post on daily.dev](https://daily.dev/posts/perception-agents-antje-barth-amazon-agi-lab-dpijkbp9u) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Perception Agents — Antje Barth, Amazon AGI Lab","url":"https://daily.dev/posts/perception-agents-antje-barth-amazon-agi-lab-dpijkbp9u","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/perception-agents-antje-barth-amazon-agi-lab-dpijkbp9u"},"datePublished":"2026-07-23T16:25:08.943Z","dateModified":"2026-09-13T18:52:43.992Z","description":"A talk from the AI Engineer Worlds Fair by an Amazon AGI Lab researcher introducing 'perception agents' — a new paradigm for AI agents that can perceive...","image":"https://i.ytimg.com/vi/2JX6JYyQG4Y/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/2JX6JYyQG4Y/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/perception-agents-antje-barth-amazon-agi-lab-dpijkbp9u","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,human-computer-interaction","timeRequired":"PT21M","video":{"@type":"VideoObject","name":"Perception Agents — Antje Barth, Amazon AGI Lab","description":"A talk from the AI Engineer Worlds Fair by an Amazon AGI Lab researcher introducing 'perception agents' — a new paradigm for AI agents that can perceive...","thumbnailUrl":"https://i.ytimg.com/vi/2JX6JYyQG4Y/sddefault.jpg","uploadDate":"2026-07-23T16:25:08.943Z","duration":"PT21M","url":"https://api.daily.dev/r/DpIJkbp9U","embedUrl":"https://www.youtube.com/embed/2JX6JYyQG4Y"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"Perception Agents — Antje Barth, Amazon AGI Lab"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/perception-agents-antje-barth-amazon-agi-lab-dpijkbp9u#faq","mainEntity":[{"@type":"Question","name":"What are perception agents from Amazon AGI Lab?","acceptedAnswer":{"@type":"Answer","text":"Perception agents are AI agents designed to perceive a rendered screen the way a human does - layout, state, and changes - rather than scraping underlying code, so they can confirm whether their own actions succeeded instead of just firing off actions and hoping. Amazon AGI Lab open sourced the first two components of this harness: an annotation tool and a verification tool. Engineers building reliable computer-use agents can follow releases like this one on daily.dev."}},{"@type":"Question","name":"What does the annotation tool in Amazon AGI Lab's perception agent harness do?","acceptedAnswer":{"@type":"Answer","text":"It is a Chrome extension that lets a user visually mark up elements on a web page, such as circling a heading or selecting a section, and describe a change in a short phrase like 'double the font size' instead of writing a long text description. It captures the exact location and style feedback and produces a summary an agent can implement without further back and forth. Anyone evaluating agent tooling for UI changes can track tools like this via daily.dev."}},{"@type":"Question","name":"How does the verification tool check an AI agent's own work against design rules?","acceptedAnswer":{"@type":"Answer","text":"It takes a design spec written in a markdown file describing colors, components, and layout rules, then runs two kinds of checks: a visual check confirming the output is on-brand and matches the layout, and a user-flow check where the agent walks through the app performing real user actions like adding or deleting a task. It produces a report listing which checks passed and which failed. Teams deciding how to add self-verification to agent workflows can follow tooling updates on daily.dev."}}]} ```