--- title: From weeks to a day: how we made LLM evaluation fast... description: Airbnb's ML team describes a four-layer framework for making LLM evaluation fast and trustworthy enough to iterate on daily. Layer 1 diagnoses two distinct... canonical: https://daily.dev/posts/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-bo32oeyen twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: From weeks to a day: how we made LLM evaluation fast enough to iterate on | daily.dev og:description: Airbnb's ML team describes a four-layer framework for making LLM evaluation fast and trustworthy enough to iterate on daily. Layer 1 diagnoses two distinct... og:url: https://daily.dev/posts/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-bo32oeyen og:image: https://api.daily.dev/og/posts/bO32oeYeN.png og:image:alt: From weeks to a day: how we made LLM evaluation fast enough to iterate on og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # From weeks to a day: how we made LLM evaluation fast enough to iterate on **[Airbnb](https://daily.dev/sources/airbnb)** · 12 min read · 0 upvotes · 0 comments ## Summary Airbnb's ML team describes a four-layer framework for making LLM evaluation fast and trustworthy enough to iterate on daily. Layer 1 diagnoses two distinct sources of evaluation noise: epistemic (judge/model limits) and aleatoric (task ambiguity). Layer 2 eliminates that noise by caching judge scores and reference outputs per-sample, making evaluation deterministic and resumable. Layer 3 introduces 'micro adapters' — low-rank LoRA patches (rank < 50) that train in under an hour on a single GPU, enabling same-day hotfixes for specific model bugs with automatic rollback. Layer 4 adds end-to-end validation across the full production path, catching seam-level failures that component-level tests miss. The key insight is that most evaluation unreliability comes from infrastructure, not model quality, and classical software engineering patterns — caching, scoped mutations, integration testing — solve it. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is a micro adapter in the context of LoRA fine-tuning for LLMs? A micro adapter is a small LoRA patch with rank under 50, layered on top of an existing shared adapter without modifying its weights, used to make a scoped correction for a specific bug. It trains in under an hour on one GPU, unlike full high-rank adapter retraining on an 8-billion-parameter model which takes days, and is validated behind regression and uncertainty-flagging gates before canary deployment. _Teams weighing fast patches versus full retraining can track emerging LoRA and fine-tuning patterns on daily.dev._ ### Why does caching LLM judge scores and references help reduce evaluation noise? Caching stabilizes the inputs to the judge rather than trying to model around judge instability, since more than half of model outputs across candidates were found to be identical strings and most reference regenerations would have matched anyway. A per-sample cache keyed by sample, model output, judge configuration, and metric makes evaluation deterministic, resumable, and comparable across runs. _Engineers building reproducible LLM eval pipelines can follow infrastructure techniques like this via daily.dev._ ### What is the difference between epistemic and aleatoric uncertainty in LLM evaluation? Epistemic uncertainty comes from model or judge limits and is actionable through better judges or methods, while aleatoric uncertainty reflects genuine task ambiguity that no judge improvement can resolve. Conflating the two causes wrong conclusions, such as misclassifying high-entropy responses as hallucinations, so diagnosing which type is causing a score movement is necessary before deciding whether a change is a real improvement. _Anyone debugging noisy model evaluation results can dig into related uncertainty-quantification research through daily.dev._ ## Similar posts on daily.dev - [The best approach to compare LLM outputs](https://daily.dev/posts/the-best-approach-to-compare-llm-outputs-qev5ctp3d) · portkey · 1 upvotes · 0 comments - [Best Practices and Methods for LLM Evaluation](https://daily.dev/posts/best-practices-and-methods-for-llm-evaluation-x3yfemf90) · databricks · 1 upvotes · 0 comments - [Stop Evaluating LLMs with “Vibe Checks”](https://daily.dev/posts/stop-evaluating-llms-with-vibe-checks--aiyxjrdvh) · Towards Data Science · 1 upvotes · 1 comments - [AI Evaluation Engineering: Build a Production-Grade LLM Evaluation Platform from Scratch \[Full Handbook\]](https://daily.dev/posts/ai-evaluation-engineering-build-a-production-grade-llm-evaluation-platform-from-scratch-full-handb-ju9v0rogq) · freeCodeCamp · 5 upvotes · 0 comments - [How to evaluate LLMs before production](https://daily.dev/posts/how-to-evaluate-llms-before-production-ps4g4zhbm) · GitHub Blog · 2 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#lora](https://daily.dev/tags/lora) [View this post on daily.dev](https://daily.dev/posts/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-bo32oeyen) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"From weeks to a day: how we made LLM evaluation fast enough to iterate on","url":"https://daily.dev/posts/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-bo32oeyen","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-bo32oeyen"},"datePublished":"2026-07-14T19:24:58.319Z","dateModified":"2026-09-13T20:16:45.940Z","description":"Airbnb's ML team describes a four-layer framework for making LLM evaluation fast and trustworthy enough to iterate on daily. Layer 1 diagnoses two distinct...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fdc2f1a1cc143b2a6ee5cb59d92ca23f?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fdc2f1a1cc143b2a6ee5cb59d92ca23f?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Airbnb","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Airbnb","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/airbnb","url":"https://daily.dev/sources/airbnb"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-bo32oeyen","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,lora","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Airbnb","item":"https://daily.dev/sources/airbnb"},{"@type":"ListItem","position":3,"name":"From weeks to a day: how we made LLM evaluation fast enough to iterate on"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-bo32oeyen#faq","mainEntity":[{"@type":"Question","name":"What is a micro adapter in the context of LoRA fine-tuning for LLMs?","acceptedAnswer":{"@type":"Answer","text":"A micro adapter is a small LoRA patch with rank under 50, layered on top of an existing shared adapter without modifying its weights, used to make a scoped correction for a specific bug. It trains in under an hour on one GPU, unlike full high-rank adapter retraining on an 8-billion-parameter model which takes days, and is validated behind regression and uncertainty-flagging gates before canary deployment. Teams weighing fast patches versus full retraining can track emerging LoRA and fine-tuning patterns on daily.dev."}},{"@type":"Question","name":"Why does caching LLM judge scores and references help reduce evaluation noise?","acceptedAnswer":{"@type":"Answer","text":"Caching stabilizes the inputs to the judge rather than trying to model around judge instability, since more than half of model outputs across candidates were found to be identical strings and most reference regenerations would have matched anyway. A per-sample cache keyed by sample, model output, judge configuration, and metric makes evaluation deterministic, resumable, and comparable across runs. Engineers building reproducible LLM eval pipelines can follow infrastructure techniques like this via daily.dev."}},{"@type":"Question","name":"What is the difference between epistemic and aleatoric uncertainty in LLM evaluation?","acceptedAnswer":{"@type":"Answer","text":"Epistemic uncertainty comes from model or judge limits and is actionable through better judges or methods, while aleatoric uncertainty reflects genuine task ambiguity that no judge improvement can resolve. Conflating the two causes wrong conclusions, such as misclassifying high-entropy responses as hallucinations, so diagnosing which type is causing a score movement is necessary before deciding whether a change is a real improvement. Anyone debugging noisy model evaluation results can dig into related uncertainty-quantification research through daily.dev."}}]} ```