--- title: Evaluate AI models for observability: o11y-bench joins... description: Grafana Labs' open benchmark o11y-bench has joined Fireworks' Specialized Intelligence Index and reached version 1.1. The benchmark tests AI models on 64... canonical: https://daily.dev/posts/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-zloeonj1b twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1 | daily.dev og:description: Grafana Labs' open benchmark o11y-bench has joined Fireworks' Specialized Intelligence Index and reached version 1.1. The benchmark tests AI models on 64... og:url: https://daily.dev/posts/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-zloeonj1b og:image: https://api.daily.dev/og/posts/zloEonj1b.png og:image:alt: Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1 og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1 **[Grafana Labs](https://daily.dev/sources/grafana)** · 5 min read · 0 upvotes · 0 comments ## Summary Grafana Labs' open benchmark o11y-bench has joined Fireworks' Specialized Intelligence Index and reached version 1.1. The benchmark tests AI models on 64 observability and SRE tasks across six areas, including PromQL, LogQL, TraceQL, incident investigation, and Grafana dashboard editing, using the Grafana MCP server. Each task runs three times, and results are scored with Pass^3 (average across runs) and Pass@3 (success at least once) to measure consistency, not just one-off success. Version 1.1 adds grounded, outcome-based grading verified against reference data, and a no-tools control failed all 64 tasks. Results from 1.1 aren't comparable to 1.0 since grading and calibration changed. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is o11y-bench and what does it test in AI models? o11y-bench is Grafana Labs' open benchmark for evaluating AI models on observability and SRE tasks. It runs agents against a live Grafana stack with synthetic telemetry, completing 64 tasks across six areas: Prometheus/PromQL, Loki/LogQL, Tempo/TraceQL, multi-step incident investigations, Grafana API workflows, and dashboard creation and repair, using the Grafana MCP server. _Developers comparing AI agents for incident response can track benchmark updates like this on daily.dev._ ### What is the difference between Pass^3 and Pass@3 in model benchmarking? Pass^3 is the average benchmark score across three repeated runs of the same task, while Pass@3 measures whether the model succeeded at least once in three attempts. o11y-bench prioritizes Pass^3 because it reflects consistent, repeatable performance rather than an occasional lucky result, which matters for trusting a model during an incident. _Teams weighing model reliability for production use can follow benchmarking methodology discussions on daily.dev._ ### What changed in o11y-bench version 1.1 compared to 1.0? Version 1.1 makes grading more outcome-based: reported numbers are checked against reference values, cited traces are read back from Tempo to verify claims, dashboard edits are graded via before-and-after diffs, and failed ground-truth queries count as grading errors rather than skewed pass rates. Because of these calibration changes, 1.1 results are not directly comparable to 1.0 and establish a new baseline. _Anyone tracking AI benchmark methodology shifts can keep up with releases like this on daily.dev._ ## Similar posts on daily.dev - [Introducing o11y-bench: an open benchmark for AI agents running observability workflows](https://daily.dev/posts/introducing-o11y-bench-an-open-benchmark-for-ai-agents-running-observability-workflows-psucsa908) · Grafana Labs · 20 upvotes · 0 comments - [Has Grafana Taken Lead in AI for Observability?](https://daily.dev/posts/has-grafana-taken-lead-in-ai-for-observability--q9hgfklco) · The New Stack · 0 upvotes · 0 comments - [Grafana offers AI assistant for free, warns users not to go mad](https://daily.dev/posts/grafana-offers-ai-assistant-for-free-warns-users-not-to-go-mad-stwrwmzko) · DEVCLASS · 0 upvotes · 0 comments - [Evaluate agent performance](https://daily.dev/posts/evaluate-agent-performance-znjiurbqt) · Google Cloud · 1 upvotes · 0 comments --- Tags: [#observability](https://daily.dev/tags/observability), [#grafana](https://daily.dev/tags/grafana) [View this post on daily.dev](https://daily.dev/posts/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-zloeonj1b) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1","url":"https://daily.dev/posts/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-zloeonj1b","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-zloeonj1b"},"datePublished":"2026-10-28T16:33:46.376Z","dateModified":"2026-10-28T16:35:58.607Z","description":"Grafana Labs' open benchmark o11y-bench has joined Fireworks' Specialized Intelligence Index and reached version 1.1. The benchmark tests AI models on 64...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/bfcaa7d681d91c601ff400b2df40457d?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/bfcaa7d681d91c601ff400b2df40457d?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Grafana Labs","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Grafana Labs","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/769cb8df467e4c428abd75e0c2491d1d","url":"https://daily.dev/sources/grafana"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-zloeonj1b","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"observability,grafana","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Grafana Labs","item":"https://daily.dev/sources/grafana"},{"@type":"ListItem","position":3,"name":"Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-zloeonj1b#faq","mainEntity":[{"@type":"Question","name":"What is o11y-bench and what does it test in AI models?","acceptedAnswer":{"@type":"Answer","text":"o11y-bench is Grafana Labs' open benchmark for evaluating AI models on observability and SRE tasks. It runs agents against a live Grafana stack with synthetic telemetry, completing 64 tasks across six areas: Prometheus/PromQL, Loki/LogQL, Tempo/TraceQL, multi-step incident investigations, Grafana API workflows, and dashboard creation and repair, using the Grafana MCP server. Developers comparing AI agents for incident response can track benchmark updates like this on daily.dev."}},{"@type":"Question","name":"What is the difference between Pass^3 and Pass@3 in model benchmarking?","acceptedAnswer":{"@type":"Answer","text":"Pass^3 is the average benchmark score across three repeated runs of the same task, while Pass@3 measures whether the model succeeded at least once in three attempts. o11y-bench prioritizes Pass^3 because it reflects consistent, repeatable performance rather than an occasional lucky result, which matters for trusting a model during an incident. Teams weighing model reliability for production use can follow benchmarking methodology discussions on daily.dev."}},{"@type":"Question","name":"What changed in o11y-bench version 1.1 compared to 1.0?","acceptedAnswer":{"@type":"Answer","text":"Version 1.1 makes grading more outcome-based: reported numbers are checked against reference values, cited traces are read back from Tempo to verify claims, dashboard edits are graded via before-and-after diffs, and failed ground-truth queries count as grading errors rather than skewed pass rates. Because of these calibration changes, 1.1 results are not directly comparable to 1.0 and establish a new baseline. Anyone tracking AI benchmark methodology shifts can keep up with releases like this on daily.dev."}}]} ```