When you ship a prompt change to a production LangGraph agent, hand-written test fixtures miss regressions that only appear against customers with real conversation history. This post walks through building an evaluation harness that forks a production Tigris object storage bucket (containing all agent checkpoints), replays real customer threads through both the baseline and candidate prompts, and uses an LLM judge with position-bias cancellation to score them head-to-head — all without touching live data. The key insight is that object storage enables O(1) copy-on-write bucket forking, making per-variant eval environments practical where a relational database would require expensive full copies. The post includes a full runnable example using the langgraph-checkpoint-tigris checkpointer and explains why object storage is architecturally superior to relational checkpointers for multi-tenant agent platforms.
Table of contents
What a state-blind regression looks like The whole loop on one page Why this only works on object storage The short version 451 Impressions1 Comment