--- title: Multimodal Collaborative Agents for Next-Gen Commerce —... description: A Google DeepMind product lead presents a framework for multimodal collaborative shopping agents designed to handle fuzzy, poorly articulated user intent... canonical: https://daily.dev/posts/multimodal-collaborative-agents-for-next-gen-commerce-nidhi-kaushik-vyas-google-deepmind-gpkk8wptp twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind | daily.dev og:description: A Google DeepMind product lead presents a framework for multimodal collaborative shopping agents designed to handle fuzzy, poorly articulated user intent... og:url: https://daily.dev/posts/multimodal-collaborative-agents-for-next-gen-commerce-nidhi-kaushik-vyas-google-deepmind-gpkk8wptp og:image: https://api.daily.dev/og/posts/GpkK8WPTp.png og:image:alt: Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 21 min read · 0 upvotes · 0 comments ## Summary A Google DeepMind product lead presents a framework for multimodal collaborative shopping agents designed to handle fuzzy, poorly articulated user intent instead of assuming well-formed search queries. The framework breaks the interaction into a discovery phase (building a working state from context, session history, and reference images with confidence scoring), a research phase (multimodal preference elicitation using visual boards rather than text, plus background comparison work), and an adaptive response phase (choosing bulleted lists, comparison tables, or visual boards based on query type). The talk covers auto-rater evaluation strategies for each phase, including counterfactual sensitivity, question utility, turn efficiency, and format accuracy, and closes with a Q&A touching on merchant ontology integration, UCP, and agent-to-agent commerce. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How should a shopping agent handle users who don't know exactly what they want yet? A collaborative agent framework breaks the interaction into three phases: discovery, where it builds a working state from past conversation, personal context, and reference images; research, where it elicits missing preferences using visual boards instead of forcing text answers, since subjective style preferences are hard to put into words; and adaptive response, where output format (bulleted list, comparison table, or visual board) matches the query type. _daily.dev surfaces practical patterns for teams designing agents that guide fuzzy, underspecified user intent._ ### What is counterfactual sensitivity testing for AI agent preference extraction? It is an evaluation method where parts of a user query are flipped to check whether an agent's extracted constraints change appropriately. If a query detail changes, the corresponding constraint the agent pulls out should also change, while unrelated constraints should stay the same, measuring sensitivity in both directions to catch agents that over- or under-react to input changes. _Developers building evaluation pipelines for AI agents can track techniques like this through daily.dev._ ## Similar posts on daily.dev - [How to design conversational AI agents](https://daily.dev/posts/how-to-design-conversational-ai-agents-iri8eb2pj) · Google Cloud · 0 upvotes · 0 comments - [Multi-Modal AI Agents In Production 2026: Vision, Audio, Glue](https://daily.dev/posts/multi-modal-ai-agents-in-production-2026-vision-audio-glue-xmtiipzk9) · Alex CloudStar · 1 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#conversational-ai](https://daily.dev/tags/conversational-ai), [#multimodal](https://daily.dev/tags/multimodal), [#recommendation-systems](https://daily.dev/tags/recommendation-systems), [#google-deepmind](https://daily.dev/tags/google-deepmind) [View this post on daily.dev](https://daily.dev/posts/multimodal-collaborative-agents-for-next-gen-commerce-nidhi-kaushik-vyas-google-deepmind-gpkk8wptp) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind","url":"https://daily.dev/posts/multimodal-collaborative-agents-for-next-gen-commerce-nidhi-kaushik-vyas-google-deepmind-gpkk8wptp","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/multimodal-collaborative-agents-for-next-gen-commerce-nidhi-kaushik-vyas-google-deepmind-gpkk8wptp"},"datePublished":"2026-09-01T17:23:09.734Z","dateModified":"2026-09-01T17:28:48.878Z","description":"A Google DeepMind product lead presents a framework for multimodal collaborative shopping agents designed to handle fuzzy, poorly articulated user intent...","image":"https://i.ytimg.com/vi/AhQpRalYlyg/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/AhQpRalYlyg/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/multimodal-collaborative-agents-for-next-gen-commerce-nidhi-kaushik-vyas-google-deepmind-gpkk8wptp","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,conversational-ai,multimodal,recommendation-systems,google-deepmind","timeRequired":"PT21M","video":{"@type":"VideoObject","name":"Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind","description":"A Google DeepMind product lead presents a framework for multimodal collaborative shopping agents designed to handle fuzzy, poorly articulated user intent...","thumbnailUrl":"https://i.ytimg.com/vi/AhQpRalYlyg/sddefault.jpg","uploadDate":"2026-09-01T17:23:09.734Z","duration":"PT21M","url":"https://api.daily.dev/r/GpkK8WPTp","embedUrl":"https://www.youtube.com/embed/AhQpRalYlyg"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/multimodal-collaborative-agents-for-next-gen-commerce-nidhi-kaushik-vyas-google-deepmind-gpkk8wptp#faq","mainEntity":[{"@type":"Question","name":"How should a shopping agent handle users who don't know exactly what they want yet?","acceptedAnswer":{"@type":"Answer","text":"A collaborative agent framework breaks the interaction into three phases: discovery, where it builds a working state from past conversation, personal context, and reference images; research, where it elicits missing preferences using visual boards instead of forcing text answers, since subjective style preferences are hard to put into words; and adaptive response, where output format (bulleted list, comparison table, or visual board) matches the query type. daily.dev surfaces practical patterns for teams designing agents that guide fuzzy, underspecified user intent."}},{"@type":"Question","name":"What is counterfactual sensitivity testing for AI agent preference extraction?","acceptedAnswer":{"@type":"Answer","text":"It is an evaluation method where parts of a user query are flipped to check whether an agent's extracted constraints change appropriately. If a query detail changes, the corresponding constraint the agent pulls out should also change, while unrelated constraints should stay the same, measuring sensitivity in both directions to catch agents that over- or under-react to input changes. Developers building evaluation pipelines for AI agents can track techniques like this through daily.dev."}}]} ```