Airbnb's ML team describes a four-layer framework for making LLM evaluation fast and trustworthy enough to iterate on daily. Layer 1 diagnoses two distinct sources of evaluation noise: epistemic (judge/model limits) and aleatoric (task ambiguity). Layer 2 eliminates that noise by caching judge scores and reference outputs per-sample, making evaluation deterministic and resumable. Layer 3 introduces 'micro adapters' — low-rank LoRA patches (rank < 50) that train in under an hour on a single GPU, enabling same-day hotfixes for specific model bugs with automatic rollback. Layer 4 adds end-to-end validation across the full production path, catching seam-level failures that component-level tests miss. The key insight is that most evaluation unreliability comes from infrastructure, not model quality, and classical software engineering patterns — caching, scoped mutations, integration testing — solve it.
