Yelp replaced its legacy rule-based customer support chatbot with an LLM-assisted system powered by a RAG pipeline. The new system routes queries into five specialized workflows (QA, Billing, Refund, Cancel, Review). The RAG pipeline uses metadata-only embeddings (title, summary, headers) from ~370 support articles rather than full-text or chunked content, which proved more accurate for semantic matching. FAISS is used for fast similarity search, achieving ~94% recall@5. A daily batch job keeps the vectorstore fresh via AWS S3. Key challenges included LLM hallucination of hyperlinks, addressed with dedicated validation. The system doubled the chatbot resolution rate in A/B testing.
Table of contents
The Evolution of Support: From Fixed Phrases to ConversationStructuring Intelligence: The Chatbot with Specialized WorkflowsDeep Dive: Building a Lean and Accurate RAG PipelineConclusionAcknowledgements351 Impressions