Your AI feature works in the demo.
It fails in production 3 weeks later. Nobody touched the model. Nobody changed the code.
The only thing that changed: the inputs got messier.
Here’s the setup:
You’re at a SaaS company. 50,000 support tickets a week. Your team builds an AI triage system — GPT-4o classifies each ticket into 6 categories (billing, bug, feature request, account access, security, other) so the right team gets it instantly.
In dev, it nails 71% accuracy. You need 90%+ to cut manual review.
The model is locked. The budget for inference isn’t unlimited. You need to close the 19-point gap.
Here are your four options:
A) Zero-shot with a better system prompt — rewrite the instructions, add explicit category definitions, specify edge case rules. No examples.
B) Few-shot examples — add 3–5 real classified tickets directly in the prompt. One example per category edge case.
C) Chain-of-Thought — add “think step-by-step before answering” to the prompt. Force the model to reason through the ticket before outputting the category.
D) Self-Consistency — run each ticket through the model 5 times with temperature=0.7, take the majority vote across outputs.
Same model. Same ticket. Four different accuracy + cost profiles.
Pick one — A, B, C, or D — and tell me why. Full breakdown in the comments.
#30DaysOfSystemDesign #AI #SystemDesign #BackendEngineering

Joud Awad

A system design challenge exploring how to improve an AI triage system's classification accuracy from 71% to 90%+ without changing the model or exceeding inference budget. Four prompt engineering strategies are presented: zero-shot with improved system prompt, few-shot examples, chain-of-thought reasoning, and self-consistency via majority voting across multiple runs. Readers are asked to evaluate the accuracy and cost trade-offs of each approach.

34/60 Days System Design Questions

B — Few-shot Best answer here
Accuracy: 88–93%. Cost increase: near-zero.
Few-shot shifts the model from reading rules to pattern-matching against real examples. Fundamentally stronger signal.
The part most teams get wrong: they pick the obvious examples. A crystal-clear billing ticket. A textbook bug report. Those teach the model nothing it didn’t already know.
You want the ambiguous ones. The ticket that looks like billing but is account access. The feature request that reads like a bug. Those are where the model currently fails — and those are the examples that close the gap.
Get the selection right, you close most of the 19 points without touching anything else.

A — Zero-shot with a better system prompt
Accuracy ceiling: ~78–82%.
This is where every team starts. You iterate the prompt, add category definitions, enumerate edge cases. It feels like progress. And for a while it is.
The hard wall: instructions describe categories. They can’t show the model where the ambiguous cases land.
Real example: you define “billing” as anything about charges or invoices. Then you get this ticket — “I can’t log in and I was charged twice last month.” Billing? Account access? Both?
No amount of rewriting tells the model which bucket that gets. It guesses wrong 20–25% of the time on edge cases. And your dataset is full of edge cases. The ceiling is real.

D — Self-Consistency
Accuracy: 93–96%. Cost: 5x. Latency: 5x.
Run the same ticket 5–10x at temperature=0.7, take the majority vote. Works because each run samples a slightly different path — noise cancels out across runs.
The math is good. The cost is brutal.
50K tickets/week × $0.002 × 5 runs = $500/week vs $100/week with few-shot. You’re paying $400/week extra for a 3–5% gain on the hardest cases.
Worth it when being wrong is expensive: fraud detection (a missed flag costs hundreds), medical triage (wrong category = real harm). For support routing, a misclassified ticket costs ~$0.10 of human rerouting time. Use B.

C — Chain-of-Thought
Accuracy delta: ±2%. Latency: +200–400ms.
CoT is the most misapplied technique in production AI right now. Teams add “think step-by-step” to everything. For some tasks it works. Not this one.
Why it struggles here: the model generates a reasoning chain, then commits to a label. On ambiguous tickets, that chain becomes a liability. “Mentions a charge… but also a login issue… charge is mentioned first… probably billing.” It picks billing. Right answer is account access.
CoT helps when the answer emerges from reasoning — math, debugging, planning. For classification, the answer emerges from calibration. Reasoning doesn’t substitute for calibration.
Layer it on top of few-shot if you’re still missing edge cases. Don’t reach for it first.

Also, it would mean a lot to me if you could support my content and stay in touch 🙏
<ul>
<li>YouTube: <a href="https://www.youtube.com/@system-design-lab" target="_blank" rel="noopener nofollow">https://www.youtube.com/@system-design-lab</a></li>
<li>LinkedIn: <a href="https://www.linkedin.com/in/joud-awad/" target="_blank" rel="noopener nofollow">https://www.linkedin.com/in/joud-awad/</a></li>
<li>Medium Blog: <a href="https://joudwawad.medium.com/" target="_blank" rel="noopener nofollow">https://joudwawad.medium.com/</a></li>
<li>Substack: <a href="https://joudawad.substack.com/" target="_blank" rel="noopener nofollow">https://joudawad.substack.com/</a></li>
</ul>