InfoQ

<h1>Microsoft Open Sources Evals for Agent Interop Starter Kit to Benchmark Enterprise AI Agents</h1>
<p>Microsoft has taken a significant step forward by open sourcing an innovative evaluation framework named ‘Evals for Agent Interop Starter Kit.’ This framework is developed to benchmark AI agents specifically within enterprise settings, with a particular focus on assessing and ensuring interoperability among these agents.</p>
<p>The Evals for Agent Interop Starter Kit provides a comprehensive toolkit to evaluate AI agents in realistic enterprise scenarios. It includes curated use cases and datasets, along with an evaluation harness that meticulously measures various aspects such as schema adherence, tool call correctness, and qualities like coherence and helpfulness as determined by AI assessments.</p>
<p>Initially, this starter kit targets evaluating interactions related to email and calendar functionalities. The package comes with declarative JSON evaluation specifications and introduces the concept of a leaderboard to facilitate comparison among agents developed on different technological stacks. This setup allows developers to easily deploy their evaluations using Docker Compose, draw baselines from provided samples, and tailor evaluation rubrics to meet specific workflow requirements.</p>
<p>By providing this open-source tool, Microsoft aims to streamline the process for developers to benchmark and improve AI agent interoperability and efficiency, thereby enhancing the overall functionality and integration of AI systems in enterprise environments.</p>


Collections

Microsoft has open sourced the 'Evals for Agent Interop Starter Kit,' an evaluation framework for benchmarking AI agents in enterprise environments. It focuses on measuring interoperability between agents and includes curated datasets, declarative JSON evaluation specs, and an evaluation harness that tracks schema adherence, tool call correctness, coherence, and helpfulness. Initial use cases cover email and calendar interactions. The kit supports Docker Compose deployment, provides baseline samples, and introduces a leaderboard for comparing agents built on different tech stacks.