--- title: How we use Insights to improve the agent behind Insights description: Datadog used its own Insights feature in Agent Observability to debug and improve the agent that powers Insights. Traces showed search calls returning roughly... canonical: https://daily.dev/posts/how-we-use-insights-to-improve-the-agent-behind-insights-tuxppw9s1 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How we use Insights to improve the agent behind Insights | daily.dev og:description: Datadog used its own Insights feature in Agent Observability to debug and improve the agent that powers Insights. Traces showed search calls returning roughly... og:url: https://daily.dev/posts/how-we-use-insights-to-improve-the-agent-behind-insights-tuxppw9s1 og:image: https://api.daily.dev/og/posts/tUxpPW9S1.png og:image:alt: How we use Insights to improve the agent behind Insights og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How we use Insights to improve the agent behind Insights **[Datadog](https://daily.dev/sources/datadog)** · 7 min read · 0 upvotes · 0 comments ## Summary Datadog used its own Insights feature in Agent Observability to debug and improve the agent that powers Insights. Traces showed search calls returning roughly 80,000 tokens of content across five calls, with one result adding 19,000 tokens to a 97,000-token request. A separate insight found task-specific context placed before tool definitions was preventing prompt caching, with an estimated $2,734 monthly savings possible by reordering requests. Other insights caught five repeated invalid tool calls over 18 minutes and 193 sessions where the agent guessed resource names. Datadog's 'Fix with Bits' option then passed findings to a coding agent to prepare a pull request. Insights in Agent Observability is now generally available with a free tier up to 40,000 LLM spans per month. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can misplaced context break LLM prompt caching and increase costs? Placing task-specific or changing information before stable tool definitions and system instructions in a prompt prevents the provider from reusing the cached prefix. Datadog found this caused requests to repeatedly rewrite nearly the entire prompt to cache, with an estimated $2,734 per month in avoidable costs from one recurring pattern. The fix is to keep stable instructions and tool definitions first, before task-specific or conversation content. _Anyone tuning LLM costs can track prompt caching techniques like this one on daily.dev._ ### What causes an AI agent to send tool calls with missing required arguments? An agent can repeatedly send invalid tool calls when it omits required fields, such as a complete object needed to replace an existing record, even while supplying other identifiers correctly. In one observed case, an agent made five identical invalid calls over about 18 minutes, all rejected by the tool, ending the run in error after four retries since repeating the same incomplete request never resolves the issue. _daily.dev helps engineers debugging agent tool-call failures stay current on patterns like this._ ### How many free LLM spans per month does Datadog Agent Observability include? Datadog Agent Observability's Insights feature offers a free tier of up to 40,000 LLM spans per month, and is generally available. It analyzes agent traces to identify recurring cost and reliability problems, explain their causes, assess impact, and recommend evidence-backed fixes, including an option to pass findings directly to a coding agent to prepare a pull request. _Teams evaluating agent observability tools can compare pricing tiers like this one on daily.dev._ ## Similar posts on daily.dev - [Optimize AI costs with Agent Observability](https://daily.dev/posts/optimize-ai-costs-with-agent-observability-lnsouoalu) · Datadog · 0 upvotes · 0 comments - [Making agentic token costs visible in production](https://daily.dev/posts/making-agentic-token-costs-visible-in-production-fejyo9qbg) · Datadog · 2 upvotes · 0 comments - [How we eliminated $1 million a year of wasted AI agent spend in one hour](https://daily.dev/posts/how-we-eliminated-1-million-a-year-of-wasted-ai-agent-spend-in-one-hour-sl4y8p7za) · databricks · 1 upvotes · 0 comments - [I Measured 23,968 Agent Turns. The Prompt Was 309× the Output.](https://daily.dev/posts/i-measured-23-968-agent-turns-the-prompt-was-309-the-output--livgzsnpr) · The Infinity · 2 upvotes · 0 comments - [AI Cost Visibility: How to Track and Optimize Token Spend Before the Invoice Arrives](https://daily.dev/posts/ai-cost-visibility-how-to-track-and-optimize-token-spend-before-the-invoice-arrives-133h7kulz) · Telerik · 1 upvotes · 1 comments --- Tags: [#devops](https://daily.dev/tags/devops), [#ai-agents](https://daily.dev/tags/ai-agents), [#llm](https://daily.dev/tags/llm), [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/how-we-use-insights-to-improve-the-agent-behind-insights-tuxppw9s1) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How we use Insights to improve the agent behind Insights","url":"https://daily.dev/posts/how-we-use-insights-to-improve-the-agent-behind-insights-tuxppw9s1","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-we-use-insights-to-improve-the-agent-behind-insights-tuxppw9s1"},"datePublished":"2026-10-30T19:17:51.891Z","dateModified":"2026-10-30T19:24:32.132Z","description":"Datadog used its own Insights feature in Agent Observability to debug and improve the agent that powers Insights. Traces showed search calls returning roughly...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a1bee1eefdc9943fd56987fd7b5183a6?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a1bee1eefdc9943fd56987fd7b5183a6?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Datadog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Datadog","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/8f9d7e3e4bd64bd7b9db9b2ea83e0c6a","url":"https://daily.dev/sources/datadog"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-we-use-insights-to-improve-the-agent-behind-insights-tuxppw9s1","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"devops,ai-agents,llm,ai-inference","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Datadog","item":"https://daily.dev/sources/datadog"},{"@type":"ListItem","position":3,"name":"How we use Insights to improve the agent behind Insights"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-we-use-insights-to-improve-the-agent-behind-insights-tuxppw9s1#faq","mainEntity":[{"@type":"Question","name":"How can misplaced context break LLM prompt caching and increase costs?","acceptedAnswer":{"@type":"Answer","text":"Placing task-specific or changing information before stable tool definitions and system instructions in a prompt prevents the provider from reusing the cached prefix. Datadog found this caused requests to repeatedly rewrite nearly the entire prompt to cache, with an estimated $2,734 per month in avoidable costs from one recurring pattern. The fix is to keep stable instructions and tool definitions first, before task-specific or conversation content. Anyone tuning LLM costs can track prompt caching techniques like this one on daily.dev."}},{"@type":"Question","name":"What causes an AI agent to send tool calls with missing required arguments?","acceptedAnswer":{"@type":"Answer","text":"An agent can repeatedly send invalid tool calls when it omits required fields, such as a complete object needed to replace an existing record, even while supplying other identifiers correctly. In one observed case, an agent made five identical invalid calls over about 18 minutes, all rejected by the tool, ending the run in error after four retries since repeating the same incomplete request never resolves the issue. daily.dev helps engineers debugging agent tool-call failures stay current on patterns like this."}},{"@type":"Question","name":"How many free LLM spans per month does Datadog Agent Observability include?","acceptedAnswer":{"@type":"Answer","text":"Datadog Agent Observability's Insights feature offers a free tier of up to 40,000 LLM spans per month, and is generally available. It analyzes agent traces to identify recurring cost and reliability problems, explain their causes, assess impact, and recommend evidence-backed fixes, including an option to pass findings directly to a coding agent to prepare a pull request. Teams evaluating agent observability tools can compare pricing tiers like this one on daily.dev."}}]} ```