--- title: Designing Agents (The Floor Is the Frontier) — Ben... description: A conference talk from Raindrop's CTO and co-founder argues that most public discourse about evaluating AI agents is stuck in the 'chatbot era' and doesn't... canonical: https://daily.dev/posts/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-eswjlaapy twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop | daily.dev og:description: A conference talk from Raindrop's CTO and co-founder argues that most public discourse about evaluating AI agents is stuck in the 'chatbot era' and doesn't... og:url: https://daily.dev/posts/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-eswjlaapy og:image: https://api.daily.dev/og/posts/esWJLaaPY.png og:image:alt: Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 19 min read · 0 upvotes · 0 comments ## Summary A conference talk from Raindrop's CTO and co-founder argues that most public discourse about evaluating AI agents is stuck in the 'chatbot era' and doesn't reflect how real teams operate. The core framing splits teams into 'benchmark maxers' versus 'floor raisers,' arguing the floor (the worst thing an agent can do, like leaking data or recommending a competitor) matters more for user trust than the ceiling (peak capability). Practical lessons shared include: evals should look like code/tests rather than prompt-playground tools since the prompt is now the entire harness; issue detection needs to track when a problem started and what percentage of users it affects; clustering traces is useful for one-off analysis but doesn't scale as a way to track issues over time because boundaries are ambiguous and root causes get conflated; 'code mode' classifiers run in a sandbox can analyze traces at production scale; and agents are poor at detecting anomalies themselves, so they should be asked to investigate anomalies already surfaced by deterministic signals like keyword frequency spikes. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why do static eval datasets for AI agents keep breaking? Static eval sets built around fixed prompts or tool-call expectations break whenever the underlying model or agent harness changes, such as switching to a new model or moving to a different CLI tool like Claude Code. Because agent behavior depends heavily on the model and harness, a large fraction of a hand-built eval set can become invalid overnight, making heavy investment in fixed eval datasets risky. _Anyone maintaining agent evals can follow how teams are adapting their testing approach on daily.dev._ ### Why isn't clustering AI agent traces a good way to track issues over time? Clustering agent traces into groups is useful for one-off exploratory analysis but does not scale as a way to monitor issues over time, because it is hard to reliably track cluster boundaries and growth (a problem sometimes called temporal clustering), and what counts as 'the same issue' varies by product. Clusters can lump together problems with very different root causes, such as grouping unrelated pricing and refund calculation bugs under one vague 'price issues' label. _Teams debugging agent failures can compare monitoring approaches like this through daily.dev._ ### Should AI agents be used to detect anomalies in their own production traces? No, agents are described as very bad at anomaly detection and should not be asked to find anomalies on their own. Instead, deterministic signals like keyword frequency spikes should be extracted first, and the agent should only be tasked with investigating anomalies that have already been surfaced this way, since that gives it a more tangible, tractable problem to work on. _Developers building agent monitoring pipelines can track practical patterns like this on daily.dev._ ## Similar posts on daily.dev - [Evaluating AI Agents in Practice: Benchmarks, Frameworks, and Lessons Learned](https://daily.dev/posts/evaluating-ai-agents-in-practice-benchmarks-frameworks-and-lessons-learned-tohpewvfd) · InfoQ · 2 upvotes · 0 comments - [Making AI work through eval hygiene](https://daily.dev/posts/making-ai-work-through-eval-hygiene-9yhow3tph) · InfoWorld · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#observability](https://daily.dev/tags/observability) [View this post on daily.dev](https://daily.dev/posts/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-eswjlaapy) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop","url":"https://daily.dev/posts/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-eswjlaapy","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-eswjlaapy"},"datePublished":"2026-08-12T18:24:54.377Z","dateModified":"2026-09-14T07:54:14.578Z","description":"A conference talk from Raindrop's CTO and co-founder argues that most public discourse about evaluating AI agents is stuck in the 'chatbot era' and doesn't...","image":"https://i.ytimg.com/vi/jHMiYtjoJfA/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/jHMiYtjoJfA/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-eswjlaapy","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,ai-agents,observability","timeRequired":"PT19M","video":{"@type":"VideoObject","name":"Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop","description":"A conference talk from Raindrop's CTO and co-founder argues that most public discourse about evaluating AI agents is stuck in the 'chatbot era' and doesn't...","thumbnailUrl":"https://i.ytimg.com/vi/jHMiYtjoJfA/sddefault.jpg","uploadDate":"2026-08-12T18:24:54.377Z","duration":"PT19M","url":"https://api.daily.dev/r/esWJLaaPY","embedUrl":"https://www.youtube.com/embed/jHMiYtjoJfA"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-eswjlaapy#faq","mainEntity":[{"@type":"Question","name":"Why do static eval datasets for AI agents keep breaking?","acceptedAnswer":{"@type":"Answer","text":"Static eval sets built around fixed prompts or tool-call expectations break whenever the underlying model or agent harness changes, such as switching to a new model or moving to a different CLI tool like Claude Code. Because agent behavior depends heavily on the model and harness, a large fraction of a hand-built eval set can become invalid overnight, making heavy investment in fixed eval datasets risky. Anyone maintaining agent evals can follow how teams are adapting their testing approach on daily.dev."}},{"@type":"Question","name":"Why isn't clustering AI agent traces a good way to track issues over time?","acceptedAnswer":{"@type":"Answer","text":"Clustering agent traces into groups is useful for one-off exploratory analysis but does not scale as a way to monitor issues over time, because it is hard to reliably track cluster boundaries and growth (a problem sometimes called temporal clustering), and what counts as 'the same issue' varies by product. Clusters can lump together problems with very different root causes, such as grouping unrelated pricing and refund calculation bugs under one vague 'price issues' label. Teams debugging agent failures can compare monitoring approaches like this through daily.dev."}},{"@type":"Question","name":"Should AI agents be used to detect anomalies in their own production traces?","acceptedAnswer":{"@type":"Answer","text":"No, agents are described as very bad at anomaly detection and should not be asked to find anomalies on their own. Instead, deterministic signals like keyword frequency spikes should be extracted first, and the agent should only be tasked with investigating anomalies that have already been surfaced this way, since that gives it a more tangible, tractable problem to work on. Developers building agent monitoring pipelines can track practical patterns like this on daily.dev."}}]} ```