--- title: "LLMs for Relevance: Automating High-Quality Product Relevance Labeling in Flipkart Search" url: https://daily.dev/posts/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-s41higuux source_url: https://blog.flipkart.tech/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-ddd5ca50b584 type: article source: "Flipkart Tech" published: 2026-08-03T12:36:02.336Z updated: 2026-08-03T14:18:15.802Z tags: ["data-science", "llm", "deep-learning", "reinforcement-learning"] reading_time: 12 upvotes: 9 comments: 1 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # LLMs for Relevance: Automating High-Quality Product Relevance Labeling in Flipkart Search **[Flipkart Tech](https://daily.dev/sources/flipkart)** · 12 min read · 9 upvotes · 1 comments ## Summary Flipkart's search team replaced manual human annotation for query-product relevance labeling with an LLM-based system called Product Analyser (PA). The system uses a two-stage training approach: Supervised Fine-Tuning (SFT) on gold-standard human-labeled data with balanced sampling across relevance buckets, followed by GRPO (Grouped Relative Policy Optimization) alignment to make reasoning consistent and robust. A Reasoning Reward Model (~8B parameters) acts as a coach, rewarding not just correct labels but well-reasoned explanations. Results show the PA exceeds human annotator accuracy by 2.3–2.9%, outperforms general-purpose proprietary LLMs by 12% at ~30% of the cost, and produces NDCG reports within 1% of manual reports across 22 weeks of data. The system scales to millions of query-product pairs on demand, provides diagnostic reasoning traces, and frees expert annotators for harder edge cases. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community discussion Top comments from developers on daily.dev. **@agustinbarrientos** · 0 upvotes > How often does the Product Analyser disagree with all three human annotators, and who gets the final say on those edge cases? ## Similar posts on daily.dev - [LLM-Powered Relevance Assessment for Pinterest Search](https://daily.dev/posts/llm-powered-relevance-assessment-for-pinterest-search-zx5akjrjt) · Pinterest Engineering · 1 upvotes · 0 comments - [LLM-as-a-Judge: Evaluating natural language search](https://daily.dev/posts/llm-as-a-judge-evaluating-natural-language-search-qzclaql93) · Doordash · 0 upvotes · 0 comments - [Scaling Human Judgment: How Dropbox Uses LLMs to Improve Labeling for RAG Systems](https://daily.dev/posts/scaling-human-judgment-how-dropbox-uses-llms-to-improve-labeling-for-rag-systems-vsvb6du0n) · InfoQ · 1 upvotes · 0 comments - [Our Early Journey to Transform Instacart’s Discovery Recommendations with LLMs](https://daily.dev/posts/our-early-journey-to-transform-instacart-s-discovery-recommendations-with-llms-wpv1ryxzx) · Instacart · 1 upvotes · 0 comments - [Search Quality Assurance with AI as a Judge](https://daily.dev/posts/search-quality-assurance-with-ai-as-a-judge-ckpklk2ko) · Zalando · 3 upvotes · 0 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#llm](https://daily.dev/tags/llm), [#deep-learning](https://daily.dev/tags/deep-learning), [#reinforcement-learning](https://daily.dev/tags/reinforcement-learning) [View this post on daily.dev](https://daily.dev/posts/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-s41higuux)