--- title: AI Agent Observability: Traces, Costs and Tool Failures description: An observable AI agent records each model call, tool call and decision as a span inside one trace, with token counts, latency and errors attached.... canonical: https://daily.dev/posts/ai-agent-observability-traces-costs-and-tool-failures-metjhul62 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: AI Agent Observability: Traces, Costs and Tool Failures | daily.dev og:description: An observable AI agent records each model call, tool call and decision as a span inside one trace, with token counts, latency and errors attached.... og:url: https://daily.dev/posts/ai-agent-observability-traces-costs-and-tool-failures-metjhul62 og:image: https://api.daily.dev/og/posts/meTJhUL62.png og:image:alt: AI Agent Observability: Traces, Costs and Tool Failures og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # AI Agent Observability: Traces, Costs and Tool Failures **[Anubhav Bhatt](https://daily.dev/sources/piirjq3y7ofa7m8zrpdg8)** · [@anubhavbhatt](https://daily.dev/anubhavbhatt) · 7 min read · 16 upvotes · 3 comments ## Summary An observable AI agent records each model call, tool call and decision as a span inside one trace, with token counts, latency and errors attached. OpenTelemetry's GenAI conventions, LangSmith, Datadog and SigNoz describe the same core data, though OpenTelemetry's conventions remain in development status. Tool call arguments and results are opt-in fields because they may hold sensitive data. LangSmith computes cost from token counts and per-model prices and offers AI Search, a public beta announced October 13, 2026, that finds traces by described behavior but reads only about an 8K-token budget per trajectory, so evidence in long sessions can get trimmed. ## Content An observable agent records each model call, tool call and decision as a span inside one trace, with token counts, latency and errors attached. The OpenTelemetry GenAI conventions, LangSmith, Datadog and SigNoz all describe the same core data, though the conventions are still marked as in development. ### Key Takeaways ``` - Capture one trace per request, with child spans for model calls, tool calls and agent steps. Attach token counts, latency and an error type to each. - OpenTelemetry's GenAI conventions define an `execute_tool` span, but tool arguments and results are opt-in because they may contain sensitive data. - Token counts are the base for cost. LangSmith computes cost from token counts and per-model prices, and lets you set cost by hand on non-LLM runs. - LangSmith's AI Search (public beta, announced 2026-10-13) finds traces by described behavior, but it reads only a fixed token budget per trajectory. ``` ## What should an agent trace contain? A trace should cover one operation end to end, with each step as its own unit. LangSmith's [observability concepts](https://docs.langchain.com/langsmith/observability-concepts) define a run as "a single unit of work executed by an agent, such as calling an LLM, formatting a prompt, or retrieving documents." A trace is "a collection of runs for a single operation," and a thread is "a sequence of traces representing a single multi-turn session."  Datadog's [LLM Observability docs](https://docs.datadoghq.com/llm_observability/) use the same shape. Each trace contains "spans representing each choice made by an agent or each step of a given workflow." Tool calls appear as operations inside those workflows. The practical rule is to keep the unit small. If a failure happens in the third tool call of a five-step plan, the span for that call is where you want the input, output and error. ## What does OpenTelemetry standardize? OpenTelemetry defines attribute and span names so that different backends can read the same data. The GenAI conventions now live in their own [repository](https://github.com/open-telemetry/semantic-conventions-genai), and the older pages in the main semantic-conventions docs say they have moved there. The [spans document](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md) carries the status "Development", and so does the [agent spans document](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md). Individual attributes can have their own badges; `error.type` is marked Stable. Names may change between releases, and SigNoz's docs warn that "These conventions are still evolving." The documents define these pieces:  The metrics page lists further agent and tool metrics, such as `genai.invokeagent.toolcalls` _and_ `genai.execute_tool.duration`, in its table of contents. Their details are in a part of the page that is easy to miss, so check the spec before depending on them. The page also says token metrics for inference are defined in a separate token-metrics document. ## How do you inspect tool calls? Use the `executetool` _span and decide deliberately what content to record. Per the spec,_ `genai.tool.name` is required. `genai.tool.call.id` _is conditionally required when the model or provider supplied an identifier. The tool call arguments (_`genai.tool.call.arguments`) and result (`gen_ai.tool.call.result`) are both Opt-In, and each carries the warning "This attribute may contain sensitive information." That default matters for debugging. Without arguments and results, a span tells you which tool ran and whether it errored, but not what it was asked or what it returned. Turning content capture on gives you the evidence and also puts user data into your telemetry store. The spec says instrumentations "MAY provide a way for users to filter or truncate output messages," and it marks `gen_ai.output.messages` with a warning that it is likely to contain PII. SigNoz's [docs](https://signoz.io/docs/llm-community-integrations) make the same point: prompt and response content is opt-in, not captured by default, for privacy. The same page says agent tool-call spans help spot loops and silent failures. ## How do you track latency and token spend? Record token counts on every model span and aggregate them upward. LangSmith's [cost tracking docs](https://docs.langchain.com/langsmith/cost-tracking) say it "automatically records LLM token usage and costs for major providers." Cost is derived from token counts and per-model prices, matched on the run's `lsmodelname` and `ls_provider` metadata. Price changes do not apply to traces already logged. Three details from those docs are easy to get wrong: - Cost is computed "greedily from most-to-least specific token type," so cache-read tokens are priced before general input tokens. - Non-LLM runs, such as a tool call or retrieval step, can carry a `totalcost` _value in_ `usagemetadata`. - For thread-level totals, child runs need `sessionid` _or_ `threadid` metadata, or their usage is excluded. SigNoz's page warns that cached and reasoning token counts are subsets of the input and output totals, so adding them again double-counts. It also recommends tracking latency as percentiles, with time to first token tracked separately for streaming. Datadog says its auto-instrumentation captures "latency, errors, and token usage metrics" without code changes. It also notes that billing is based on LLM spans and that "one agent workflow can produce several LLM spans." That is a cost of the observability tool itself, separate from model spend. ## What does LangSmith's AI Search add for debugging? AI Search lets you describe a behavior in plain language and returns traces that match. LangChain's [announcement](https://www.langchain.com/blog/agent-traces-ai-search-langsmith), dated 2026-10-13, says it "now enables users to find conversations based on semantic understanding of traces." A decision model reads each trajectory and returns matches. Structured filters, such as errored runs or a tag, can narrow the set first. LangChain's example queries include whether a user got frustrated, whether PII leaked, and an agent that said it finished a task but did not. [Developer Tech News](https://www.developer-tech.com/news/langsmith-ai-search-debugging-ai-agents/) reported the launch on 2026-10-14. It says the feature can "distinguish between an agent that completed a task and one that only claimed to have completed it." Availability per LangChain: public beta on LangSmith Cloud in the GCP US, EU and APAC regions, at no additional cost, enabled by an organization admin in settings. This complements structured filters rather than replacing them. A filter finds errored runs. A behavioral query is aimed at runs that finished without an error flag but did the wrong thing. ## What are the limits? The main limit of AI Search is the token budget. LangChain says each trajectory gets a fixed budget of about 8K tokens, "filled in priority order": user messages and agent replies first, then intermediate AI messages, tool results and system messages. Messages that do not fit are trimmed and later ones dropped. Developer Tech News reports that evidence in lower-priority tool outputs can be missed in long sessions. It also reports LangChain's own sample of 150 conversations from its Open SWE coding agent, where the median conversation was about 72,000 tokens and tool outputs were about 85% of the content. That is vendor-reported. Per the same coverage, detecting an agent that ignored a tool error requires the tool result, which LangChain says it is still working on. Other open points: - The OpenTelemetry GenAI conventions are in Development status, so attribute and metric names may change. - Content capture for tool arguments and results is opt-in. Searching for exposed PII afterwards is separate from preventing sensitive data from being recorded, as Developer Tech News notes. - The LangSmith concepts page does not describe latency or cost; those come from other docs pages. - Independent evaluations of AI Search accuracy are not published, and the feature is in beta. - Datadog's docs describe OpenTelemetry GenAI support only through a link, not in detail. ## Sources - [OpenTelemetry semantic conventions for GenAI (repository)](https://github.com/open-telemetry/semantic-conventions-genai) - [GenAI spans specification](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md) - [GenAI agent spans specification](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md) - [GenAI metrics specification](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md) - [LangSmith observability concepts](https://docs.langchain.com/langsmith/observability-concepts) - [LangSmith cost tracking](https://docs.langchain.com/langsmith/cost-tracking) - [LangChain: Search agent traces by behavior with AI Search in LangSmith](https://www.langchain.com/blog/agent-traces-ai-search-langsmith) - [Developer Tech News: LangChain adds AI Search to LangSmith for debugging AI agents](https://www.developer-tech.com/news/langsmith-ai-search-debugging-ai-agents/) - [Datadog LLM Observability docs](https://docs.datadoghq.com/llm_observability/) - [SigNoz LLM monitoring docs](https://signoz.io/docs/llm-community-integrations) ## Community discussion Top comments from developers on daily.dev. **@pdfopsdev** · 7 upvotes > the 8k budget on LangSmith's AI search is the part people miss, if your agent loop has a long tool-call chain it'll just drop the early spans and you get a confident wrong answer about what happened. we ended up grepping raw traces for anything beyond a few steps instead of trusting the search. **@diyweb** · 5 upvotes > Making tool arguments and results opt-in is the right default, because those payloads often contain customer data that should not end up in a trace store. I would also tag every span with a prompt or version identifier, so a cost or failure spike can be tied to a specific change. Without that, you can see that something got worse but not which release caused it. **@aliahmeddiyweb** · 2 upvotes > The detail that OpenTelemetry's execute_tool span keeps tool arguments and results opt-in because they may contain sensitive data is a sensible default. It does raise a question for the example of a failure in the third tool call of a five-step plan, since the input that caused it may not be captured. Would redacting specific fields, or recording full payloads only for failed spans, be a workable middle ground? ## Similar posts on daily.dev - [AI Agent Observability: Logging, Tracing, and Debugging Explained](https://daily.dev/posts/ai-agent-observability-logging-tracing-and-debugging-explained-95izimukh) · Machine Learning Mastery · 1 upvotes · 0 comments - [The AI observability race will be won on interoperability](https://daily.dev/posts/the-ai-observability-race-will-be-won-on-interoperability-bhlqq3vjj) · The New Stack · 0 upvotes · 0 comments - [Why Your AI Agent is a Black Box and How to fix it With OpenTelemetry](https://daily.dev/posts/why-your-ai-agent-is-a-black-box-and-how-to-fix-it-with-opentelemetry-oakutrahj) · DevOps.com · 0 upvotes · 0 comments - [Optimize AI costs with Agent Observability](https://daily.dev/posts/optimize-ai-costs-with-agent-observability-lnsouoalu) · Datadog · 0 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#observability](https://daily.dev/tags/observability), [#opentelemetry](https://daily.dev/tags/opentelemetry), [#langsmith](https://daily.dev/tags/langsmith), [#llm-observability](https://daily.dev/tags/llm-observability) [View this post on daily.dev](https://daily.dev/posts/ai-agent-observability-traces-costs-and-tool-failures-metjhul62) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/ai-agent-observability-traces-costs-and-tool-failures-metjhul62","headline":"AI Agent Observability: Traces, Costs and Tool Failures","text":"An observable AI agent records each model call, tool call and decision as a span inside one trace, with token counts, latency and errors attached. OpenTelemetry's GenAI conventions, LangSmith, Datadog and SigNoz describe the same core data, though OpenTelemetry's conventions remain in development status. Tool call arguments and results are opt-in fields because they may hold sensitive data. LangSmith computes cost from token counts and per-model prices and offers AI Search, a public beta announced October 13, 2026, that finds traces by described behavior but reads only about an 8K-token budget per trajectory, so evidence in long sessions can get trimmed.","url":"https://daily.dev/posts/ai-agent-observability-traces-costs-and-tool-failures-metjhul62","datePublished":"2026-10-24T14:54:00.161Z","dateModified":"2026-10-24T14:54:01.241Z","author":{"@type":"Person","name":"Anubhav Bhatt","url":"https://daily.dev/anubhavbhatt","image":"https://media.daily.dev/image/upload/s--btrq4aRY--/f_auto/v1776340073/avatars/avatar_pIiRJq3y7oFA7M8ZRPDG8?_a=BAMAMiWQ0","description":"Python + System Design | Turning Complex Ideas into Scalable Solutions","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":4080}},"image":"https://media.daily.dev/image/upload/s--Gf13dH7u--/f_auto,pg_1/v1792828116/posts/meTJhUL62?_a=BAMAMiiw0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":16},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":3}],"comment":[{"@type":"Comment","text":"the 8k budget on LangSmith’s AI search is the part people miss, if your agent loop has a long tool-call chain it’ll just drop the early spans and you get a confident wrong answer about what happened. we ended up grepping raw traces for anything beyond a few steps instead of trusting the search.","datePublished":"2026-10-25T18:05:10.009Z","url":"https://daily.dev/posts/meTJhUL62#c-6BkBVfSU2","author":{"@type":"Person","name":"PDFops","url":"https://daily.dev/pdfopsdev","image":"https://media.daily.dev/image/upload/s---8isRBKc--/f_auto/v1782922291/avatars/avatar_orjMeK8QKaaVZwGq7ScPz?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":7}},{"@type":"Comment","text":"Making tool arguments and results opt-in is the right default, because those payloads often contain customer data that should not end up in a trace store. I would also tag every span with a prompt or version identifier, so a cost or failure spike can be tied to a specific change. Without that, you can see that something got worse but not which release caused it.","datePublished":"2026-10-25T18:12:56.345Z","url":"https://daily.dev/posts/meTJhUL62#c-sm9hDezQh","author":{"@type":"Person","name":"DiyWeb","url":"https://daily.dev/diyweb","image":"https://media.daily.dev/image/upload/s--dgS6jDeN--/f_auto/v1790879420/avatars/avatar_774zDwMaRTNS1fv5OKStX?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":5}},{"@type":"Comment","text":"The detail that OpenTelemetry’s execute_tool span keeps tool arguments and results opt-in because they may contain sensitive data is a sensible default. It does raise a question for the example of a failure in the third tool call of a five-step plan, since the input that caused it may not be captured. Would redacting specific fields, or recording full payloads only for failed spans, be a workable middle ground?","datePublished":"2026-10-26T12:11:56.199Z","url":"https://daily.dev/posts/meTJhUL62#c-D2kCNpm4r","author":{"@type":"Person","name":"Ali Ahmed","url":"https://daily.dev/aliahmeddiyweb","image":"https://media.daily.dev/image/upload/s--b4sqNNv5--/f_auto,pg_1/v1792538576/avatars/avatar_lsiC0qqIxPUeSVlhCejTA?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/piirjq3y7ofa7m8zrpdg8","name":"Anubhav Bhatt"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Anubhav Bhatt","item":"https://daily.dev/sources/piirjq3y7ofa7m8zrpdg8"},{"@type":"ListItem","position":3,"name":"AI Agent Observability: Traces, Costs and Tool Failures"}]} ```