Similarweb's AI engineering team shares how they built an evaluation workflow for long-form agent research reports in Data Studio using LangSmith. The post covers two evaluation approaches: deterministic checks (tool usage, structured output) and LLM-as-judge scoring. For open-ended deep research reports with no single correct answer, they use rubric-based scoring with explicit anchors per quality dimension (e.g., source integration, faithfulness) plus A/B comparisons against a saved baseline. A key lesson: miscalibrated rubrics can make good updates appear as regressions — they lost a week to conflicting criteria before inspecting per-criterion comments. The workflow that worked combines hypothesis-driven small evaluations, trace inspection, full benchmark runs with repetitions, and baseline comparisons, treating evaluation as an engineering workflow rather than a release checklist.
