--- title: "What We Learned Evaluating Agent Memory:The Setup (Part 1)" url: https://daily.dev/posts/what-we-learned-evaluating-agent-memory-the-setup-part-1--tyaycnpsz source_url: https://www.couchbase.com/blog/what-we-learned-evaluating-agent-memorythe-setup-part-1 type: article source: "Couchbase" published: 2026-07-28T15:02:03.186Z updated: 2026-07-28T20:18:13.821Z tags: ["llm", "rag", "vector-search", "couchbase"] reading_time: 12 upvotes: 1 comments: 1 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # What We Learned Evaluating Agent Memory:The Setup (Part 1) **[Couchbase](https://daily.dev/sources/couchbase)** · 12 min read · 1 upvotes · 1 comments ## Summary Agent memory differs fundamentally from RAG: the corpus is personal and dynamic, queries are autobiographical, and time is a first-class dimension. This post details an evaluation framework using two benchmarks — LoCoMo (10 long naturalistic conversations, 1,540 questions) and LongMemEval (500 questions in session haystacks at two scales: ~46 sessions vs. ~500 sessions). Key findings include that optimal retrieval depth (k) and prompt variants differ significantly between the small and large scale variants, making LME-S an insufficient proxy for production. Three metrics are used: BLEU, F1, and an LLM Judge score — with the judge prompt itself shown to dramatically shift results. The ingestion pipeline uses dual point-of-view extraction, async LLM context extraction (~3.6s), and stores raw text, summaries, and discrete fact strings in Couchbase with human-readable timestamps. Embedding-based vector search returns in ~298ms p50, with the embedding step as the bottleneck rather than Couchbase retrieval. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community discussion Top comments from developers on daily.dev. **@kartiknvj** · 0 upvotes > The finding that the LLM judge prompt alone dramatically shifts results matches what I keep running into, the judge is a model you have to evaluate before you trust it to evaluate anything else. I would want to see BLEU and F1 broken out against the judge on the temporal questions specifically, since autobiographical recall with time as a first-class dimension is exactly where lexical-overlap metrics tend to reward the wrong answer. Are you pinning the judge model version across LoCoMo and LongMemEval runs, or does that drift too? ## Similar posts on daily.dev - [What We Learned Evaluating Agent Memory:The Results \(Part 2\)](https://daily.dev/posts/what-we-learned-evaluating-agent-memory-the-results-part-2--cfjxen3ox) · Couchbase · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#rag](https://daily.dev/tags/rag), [#vector-search](https://daily.dev/tags/vector-search), [#couchbase](https://daily.dev/tags/couchbase) [View this post on daily.dev](https://daily.dev/posts/what-we-learned-evaluating-agent-memory-the-setup-part-1--tyaycnpsz)