DoorDash engineering describes how they built a query-item relevance prediction system for search ads using a teacher-student model architecture. A fine-tuned LLM acts as an offline teacher, generating relevance labels (on a 0-1-2 scale) for millions of query-item pairs derived from six months of production traffic. A compact BERT-based bi-encoder (DistilBERT) serves as the student model for low-latency online inference. Embeddings are precomputed and cached; a bilinear layer scores relevance at serving time. The system filters irrelevant items before the ad auction, preventing well-funded but irrelevant campaigns from winning. The SLM-based model achieved a 5.2% relative gain in Precision@2 over the previous gradient-boosted decision tree baseline in A/B testing.
