AI agent teams need full execution traces, not just pass/fail results, to understand why agents succeed, fail, or waste tokens. The piece argues that 'trace is evals' and walks through why agent traces differ from traditional distributed tracing: they are long-running, nested, schema-drifting JSON that can span hundreds of MB. An internal experiment comparing Claude Code, Evot, and Pi on the same task (all using DeepSeek V4 Pro) shows execution time varies wildly due to harness-model fit rather than raw model capability. The piece then describes what a data layer needs (native JSON storage, cleaning, hot-field acceleration, full-text search, path-level governance, cheap object storage, incremental computation) and presents Databend's architecture combining VARIANT storage, accelerated columns, full-text indexes, and Stream/Task for incremental aggregation as a foundation for evaluation, replay, attribution, and training data generation.
Table of contents
From Prompt Engineering to Harness EngineeringWhy Agent Evaluation Needs Trace DataHarness and Model Fit: Claude Code, Evot, and PiAgent Paths Branch EarlyWhat an LLM Request Actually ContainsAgent Trace Is Not Traditional TraceWhat the Data Layer Needs for Agent TraceDatabend’s Minimal Trace Storage and Analytics ArchitectureOne Trace Dataset, Many Upper-Layer WorkflowsTrace Is the Foundation for Reliable AgentsTry Databend CloudQuestions this post answers
Why did Claude Code take much longer to complete a task when using a third-party model instead of its own model?
The slowdown came from harness-model mismatch rather than weaker model capability. In an internal experiment, Claude Code took about 15 minutes on a task using DeepSeek V4 Pro, but only about 3 minutes using its own Opus 4.6 model, because Claude Code's system prompts, tool names, and execution conventions are optimized specifically for models it was designed around. daily.dev surfaces this kind of harness-versus-model tradeoff for teams picking an agent stack.
How is tracing AI agent execution different from traditional distributed tracing?
Agent traces run for minutes to hours instead of seconds, evolve as spans are appended over time, and originate from prompts, tool calls, and model responses rather than stable SDK instrumentation. A single agent trace can range from hundreds of KB to hundreds of MB, with deeply nested and schema-drifting JSON, unlike the relatively stable schemas of traditional service traces. Engineers designing agent observability pipelines track these distinctions on daily.dev.
What capabilities does a data warehouse need to support AI agent trace analysis?
It needs native JSON storage so traces aren't forced into rigid schemas, strong JSON cleaning functions, fast access to hot fields like trace_id and model, full-text search across messages and errors, path-level governance for sensitive data, low-cost object storage for retention, and high-throughput ingestion with incremental aggregation instead of full historical rescans. daily.dev helps teams evaluating data infrastructure choices for agent workloads stay informed.