Flipkart's search team replaced manual human annotation for query-product relevance labeling with an LLM-based system called Product Analyser (PA). The system uses a two-stage training approach: Supervised Fine-Tuning (SFT) on gold-standard human-labeled data with balanced sampling across relevance buckets, followed by GRPO (Grouped Relative Policy Optimization) alignment to make reasoning consistent and robust. A Reasoning Reward Model (~8B parameters) acts as a coach, rewarding not just correct labels but well-reasoned explanations. Results show the PA exceeds human annotator accuracy by 2.3–2.9%, outperforms general-purpose proprietary LLMs by 12% at ~30% of the cost, and produces NDCG reports within 1% of manual reports across 22 weeks of data. The system scales to millions of query-product pairs on demand, provides diagnostic reasoning traces, and frees expert annotators for harder edge cases.
Table of contents
The Solution: The Product Analyser LLMStage I: Supervised Fine-Tuning (Teaching with Solved Problems)1. Starting from a gold standard2. Making sure the model sees the hard cases too3. Letting the model generate its own high-quality reasoning (with a safety check)4. Scaling upStage II: GRPO Alignment (Teaching Reliable Reasoning)1. The intuition behind GRPO (no separate “grader” needed)Get Amey Patil ’s stories in your inbox2. What we reward: a three-part signal3. The Reasoning Reward Model: a coach for how the model thinks12.5K Impressions1 Comment