What We Learned Evaluating Agent Memory:The Setup (Part 1)

Agent memory differs fundamentally from RAG: the corpus is personal and dynamic, queries are autobiographical, and time is a first-class dimension. This post details an evaluation framework using two benchmarks — LoCoMo (10 long naturalistic conversations, 1,540 questions) and LongMemEval (500 questions in session haystacks at two scales: ~46 sessions vs. ~500 sessions). Key findings include that optimal retrieval depth (k) and prompt variants differ significantly between the small and large scale variants, making LME-S an insufficient proxy for production. Three metrics are used: BLEU, F1, and an LLM Judge score — with the judge prompt itself shown to dramatically shift results. The ingestion pipeline uses dual point-of-view extraction, async LLM context extraction (~3.6s), and stores raw text, summaries, and discrete fact strings in Couchbase with human-readable timestamps. Embedding-based vector search returns in ~298ms p50, with the embedding step as the bottleneck rather than Couchbase retrieval.