--- title: Building Closed-Loop Evals for a Multimodal Agent at... description: Engineers from Uber's computer vision team describe how they built a closed-loop evaluation system for a multimodal AI agent that automatically enhances food... canonical: https://daily.dev/posts/building-closed-loop-evals-for-a-multimodal-agent-at-scale-soumya-gupta-jai-chopra-uber-ltyp4yuew twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber | daily.dev og:description: Engineers from Uber's computer vision team describe how they built a closed-loop evaluation system for a multimodal AI agent that automatically enhances food... og:url: https://daily.dev/posts/building-closed-loop-evals-for-a-multimodal-agent-at-scale-soumya-gupta-jai-chopra-uber-ltyp4yuew og:image: https://api.daily.dev/og/posts/LTYP4yUeW.png og:image:alt: Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 21 min read · 1 upvotes · 0 comments ## Summary Engineers from Uber's computer vision team describe how they built a closed-loop evaluation system for a multimodal AI agent that automatically enhances food photos on Uber Eats at scale. The system includes an image understanding and routing agent, an iterative image editing agent with a QA feedback loop, and a final publish-ready QA gate. Key design elements include: human-labeled golden datasets as ground truth, confusion matrix-based router evaluation with precision/recall guardrails, a 'pass at K' metric for iterative image enhancement, and an automated diagnoser agent that detects drift, localizes failures, and triggers prompt auto-tuning without human intervention. The talk covers failure modes like faithfulness violations (adding/removing food items), reward hacking, and object coherence issues, as well as multiple feedback loops from internal dogfooding, merchant feedback, and production conversion metrics. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What metric does Uber Eats use to evaluate iterative AI image enhancement quality? Uber Eats measures pass@K, the pass rate at the Kth iteration of an image enhancement loop, where a QA agent rejects or accepts an edited photo and feeds rejection reasons back into the next generation attempt. Pass rate is expected to increase with more iterations since the model gets more specific feedback each round, such as fixing incorrect portion size or unrealistic plating. _Teams designing iterative AI QA loops can follow ongoing coverage of agent evaluation patterns on daily.dev._ ### How does a closed-loop system detect and fix AI agent drift without human intervention? Uber Eats uses a diagnoser agent that samples production data, compares agent outputs against fresh human-labeled data, and detects mismatches. When drift is found, it triggers an auto-tuning pipeline with a reflect sub-agent that identifies systemic issues and a synthesize sub-agent that updates the agent's config, benchmarks it against a golden dataset, and only ships the new version if it passes guardrail metrics. _Engineers building self-correcting agent pipelines can track similar architecture patterns via daily.dev._ ### What guardrail metric does Uber Eats prioritize for its image routing agent? Recall is the guardrail metric for the routing agent that decides whether a food photo needs enhancement, because the priority is ensuring no bad-quality image slips through the system undetected. Failures cited include a high-quality cheeseburger photo wrongly flagged for enhancement, wasting compute and risking degradation, and a six-wing photo mislabeled as eight pieces, risking hallucinated extra wings during enhancement. _Anyone tuning classifier-style routing thresholds can follow real-world guardrail metric tradeoffs on daily.dev._ ## Similar posts on daily.dev - [Human judgment in the agent improvement loop](https://daily.dev/posts/human-judgment-in-the-agent-improvement-loop-iwssfql5p) · LangChain · 1 upvotes · 0 comments - [AI Evals for Solo Developers 2026: A Practical Guide](https://daily.dev/posts/ai-evals-for-solo-developers-2026-a-practical-guide-ms6ijzdsf) · Alex CloudStar · 0 upvotes · 0 comments - [Eval-driven development: Build and evaluate reliable AI agents](https://daily.dev/posts/eval-driven-development-build-and-evaluate-reliable-ai-agents-xenqh9mz3) · Red Hat Developer · 1 upvotes · 0 comments - [Agentic AI Flywheels](https://daily.dev/posts/agentic-ai-flywheels-x9mcjmgv2) · SwirlAI · 8 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#computer-vision](https://daily.dev/tags/computer-vision), [#mlops](https://daily.dev/tags/mlops), [#multimodal](https://daily.dev/tags/multimodal) [View this post on daily.dev](https://daily.dev/posts/building-closed-loop-evals-for-a-multimodal-agent-at-scale-soumya-gupta-jai-chopra-uber-ltyp4yuew) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber","url":"https://daily.dev/posts/building-closed-loop-evals-for-a-multimodal-agent-at-scale-soumya-gupta-jai-chopra-uber-ltyp4yuew","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/building-closed-loop-evals-for-a-multimodal-agent-at-scale-soumya-gupta-jai-chopra-uber-ltyp4yuew"},"datePublished":"2026-07-24T22:00:49.567Z","dateModified":"2026-09-13T18:52:58.691Z","description":"Engineers from Uber's computer vision team describe how they built a closed-loop evaluation system for a multimodal AI agent that automatically enhances food...","image":"https://i.ytimg.com/vi/31GUkCBD-Uc/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/31GUkCBD-Uc/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/building-closed-loop-evals-for-a-multimodal-agent-at-scale-soumya-gupta-jai-chopra-uber-ltyp4yuew","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,computer-vision,mlops,multimodal","timeRequired":"PT21M","video":{"@type":"VideoObject","name":"Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber","description":"Engineers from Uber's computer vision team describe how they built a closed-loop evaluation system for a multimodal AI agent that automatically enhances food...","thumbnailUrl":"https://i.ytimg.com/vi/31GUkCBD-Uc/sddefault.jpg","uploadDate":"2026-07-24T22:00:49.567Z","duration":"PT21M","url":"https://api.daily.dev/r/LTYP4yUeW","embedUrl":"https://www.youtube.com/embed/31GUkCBD-Uc"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/building-closed-loop-evals-for-a-multimodal-agent-at-scale-soumya-gupta-jai-chopra-uber-ltyp4yuew#faq","mainEntity":[{"@type":"Question","name":"What metric does Uber Eats use to evaluate iterative AI image enhancement quality?","acceptedAnswer":{"@type":"Answer","text":"Uber Eats measures pass@K, the pass rate at the Kth iteration of an image enhancement loop, where a QA agent rejects or accepts an edited photo and feeds rejection reasons back into the next generation attempt. Pass rate is expected to increase with more iterations since the model gets more specific feedback each round, such as fixing incorrect portion size or unrealistic plating. Teams designing iterative AI QA loops can follow ongoing coverage of agent evaluation patterns on daily.dev."}},{"@type":"Question","name":"How does a closed-loop system detect and fix AI agent drift without human intervention?","acceptedAnswer":{"@type":"Answer","text":"Uber Eats uses a diagnoser agent that samples production data, compares agent outputs against fresh human-labeled data, and detects mismatches. When drift is found, it triggers an auto-tuning pipeline with a reflect sub-agent that identifies systemic issues and a synthesize sub-agent that updates the agent's config, benchmarks it against a golden dataset, and only ships the new version if it passes guardrail metrics. Engineers building self-correcting agent pipelines can track similar architecture patterns via daily.dev."}},{"@type":"Question","name":"What guardrail metric does Uber Eats prioritize for its image routing agent?","acceptedAnswer":{"@type":"Answer","text":"Recall is the guardrail metric for the routing agent that decides whether a food photo needs enhancement, because the priority is ensuring no bad-quality image slips through the system undetected. Failures cited include a high-quality cheeseburger photo wrongly flagged for enhancement, wasting compute and risking degradation, and a six-wing photo mislabeled as eight pieces, risking hallucinated extra wings during enhancement. Anyone tuning classifier-style routing thresholds can follow real-world guardrail metric tradeoffs on daily.dev."}}]} ```