From Prompt to Classifier: A Production Case Study

Gusto's engineering team shares how they built an AI-to-human escalation classifier for their customer support system. Instead of the traditional ML pipeline (collect data, train, deploy), they started with an LLM-based prompt on day one, used production traffic to build a labeled dataset of ~3,500 turns, and iterated on rules with real conversation data. Once evaluation metrics plateaued and the problem definition stabilized, they fine-tuned a lightweight BERT-based binary classifier achieving ~94% precision and ~93% recall. The post outlines when this LLM-first-then-specialize pattern works best: high-volume, narrow, stable decisions like intent classification or routing — and where it doesn't, such as open-ended generation or shifting rules. The broader takeaway is to use LLMs to bootstrap understanding, then replace well-understood nodes with smaller, faster, cheaper purpose-built models.