---
title: "Search Quality Assurance with AI as a Judge"
url: https://daily.dev/posts/search-quality-assurance-with-ai-as-a-judge-ckpklk2ko
source_url: https://engineering.zalando.com/posts/2026/03/search-quality-assurance-with-llm-judge.html
type: article
source: "Zalando"
published: 2026-03-17T08:15:48.138Z
updated: 2026-03-17T08:16:13.868Z
tags: ["nlp", "gpt", "apache-airflow"]
reading_time: 14
upvotes: 3
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Search Quality Assurance with AI as a Judge

**[Zalando](https://daily.dev/sources/zalando)** · 14 min read · 3 upvotes · 0 comments

## Summary

Zalando built a search quality assurance framework using LLM-as-a-judge to evaluate search relevance at scale, enabling pre-launch validation for new markets (Luxembourg, Portugal, Greece). The system clusters search queries by NER tags to group similar search intents, translates them to target languages via LLM, then evaluates search results using GPT-4o scoring relevance on a 0–4 scale. Pipelines run on Apache Airflow with Kubernetes PodOperators and an Elasticache layer to avoid redundant product data fetches. A full evaluation of 1,500 search segments with 25 results each costs ~$250 and takes 3–5 hours. The framework proactively identified NER lemmatization bugs, unrecognized Portuguese and Greek terms, and undiscoverable product categories before go-live, replacing a slow, reactive manual review process.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://engineering.zalando.com/posts/2026/03/search-quality-assurance-with-llm-judge.html>

## Similar posts on daily.dev

- [LLM-as-a-Judge: Evaluating natural language search](https://daily.dev/posts/llm-as-a-judge-evaluating-natural-language-search-qzclaql93) · Doordash · 0 upvotes · 0 comments
- [LLM-Powered Relevance Assessment for Pinterest Search](https://daily.dev/posts/llm-powered-relevance-assessment-for-pinterest-search-zx5akjrjt) · Pinterest Engineering · 1 upvotes · 0 comments

---

Tags: [#nlp](https://daily.dev/tags/nlp), [#gpt](https://daily.dev/tags/gpt), [#apache-airflow](https://daily.dev/tags/apache-airflow)

[View this post on daily.dev](https://daily.dev/posts/search-quality-assurance-with-ai-as-a-judge-ckpklk2ko)