At OpenSearchCon Europe 2026, Bobby Mohammed (AWS) and Daniel Wrigley (OpenSource Connections) introduced Agentic Relevance Tuning (ART), a framework that uses specialized LLM agents to automate the full search relevance lifecycle in OpenSearch. Separate agents monitor User Behavior Insights data, generate tuning hypotheses, validate them through offline evaluations using the Search Relevance Workbench, and orchestrate deployment—all within human-managed guardrails. The framework composes existing OpenSearch components (UBI, Search Relevance Workbench, neural/hybrid ranking) into a closed loop, aiming to turn relevance tuning from a staffing and expertise problem into more of an infrastructure capability, though the presenters were careful to frame it as augmentation rather than full replacement of search engineers.
Table of contents
What ART actually isClosing the loop between data and improvementWhy this is timelyQuestions this post answers
What is Agentic Relevance Tuning (ART) in OpenSearch?
Agentic Relevance Tuning (ART) is a framework introduced by AWS and OpenSource Connections that uses specialized LLM agents to automate the full search relevance lifecycle in OpenSearch. One agent monitors User Behavior Insights (UBI) data for degradations, another generates tuning hypotheses, and another validates fixes using the Search Relevance Workbench before orchestrating deployment, all within human-managed guardrails. Teams weighing how much of search tuning to automate can follow ongoing OpenSearch relevance coverage on daily.dev.
How does ART decide when search relevance needs to be fixed in OpenSearch?
ART relies on an agent that continuously monitors relevance signals from OpenSearch's User Behavior Insights (UBI), which captures click, conversion, and session data showing how users respond to search results. When that agent detects a degradation or improvement opportunity, it hands off to another agent that hypothesizes changes and tests them via the Search Relevance Workbench before deployment. Developers tracking how UBI-driven signals feed automated tuning can follow OpenSearch updates on daily.dev.