---
title: "Better Experiments with LLM Evals — A funnel, not a fork"
url: https://daily.dev/posts/better-experiments-with-llm-evals-a-funnel-not-a-fork-lzwtdyy08
source_url: https://engineering.atspotify.com/2026/5/better-experiments-with-llm-evals-a-funnel-not-a-fork
type: article
source: "Spotify Labs"
published: 2026-05-18T13:52:49.755Z
updated: 2026-05-18T13:53:11.857Z
tags: ["machine-learning", "llm", "spotify", "ab-testing"]
reading_time: 5
upvotes: 0
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Better Experiments with LLM Evals — A funnel, not a fork

**[Spotify Labs](https://daily.dev/sources/spotify)** · 5 min read · 0 upvotes · 0 comments

## Summary

Spotify engineers argue that LLM evals and A/B experiments should form a funnel, not competing alternatives. LLM evals (automated judges assessing relevance, coherence, tone) belong before experiments to filter out weak candidates and raise the hit rate of what gets tested. Experiments then validate whether real users respond as predicted and catch regressions in secondary metrics that evals miss. A key insight is that evals need continuous calibration against online outcomes — without this, eval scores are opinions, not evidence. The post describes a feedback loop where running evals on A/B test data helps calibrate judges over time, making both evals and experiments progressively smarter.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://engineering.atspotify.com/2026/5/better-experiments-with-llm-evals-a-funnel-not-a-fork>

## Similar posts on daily.dev

- [Most LLM-powered evals are BROKEN\!](https://daily.dev/posts/most-llm-powered-evals-are-broken--hp4gepz0s) · Daily Dose of Data Science \| Avi Chawla \| Substack · 1 upvotes · 0 comments
- [Why evals are essential for AI product managers](https://daily.dev/posts/why-evals-are-essential-for-ai-product-managers-ofqo83cnz) · LogRocket · 0 upvotes · 0 comments
- [A Guide to LLM Evals](https://daily.dev/posts/a-guide-to-llm-evals-y9wpcrtpx) · ByteByteGo · 1 upvotes · 0 comments

---

Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#llm](https://daily.dev/tags/llm), [#spotify](https://daily.dev/tags/spotify), [#ab-testing](https://daily.dev/tags/ab-testing)

[View this post on daily.dev](https://daily.dev/posts/better-experiments-with-llm-evals-a-funnel-not-a-fork-lzwtdyy08)