--- title: LLMs for Relevance: Automating High-Quality Product... description: Flipkart's search team replaced manual human annotation for query-product relevance labeling with an LLM-based system called Product Analyser (PA). The system... canonical: https://daily.dev/posts/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-s41higuux twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: LLMs for Relevance: Automating High-Quality Product Relevance Labeling in Flipkart Search | daily.dev og:description: Flipkart's search team replaced manual human annotation for query-product relevance labeling with an LLM-based system called Product Analyser (PA). The system... og:url: https://daily.dev/posts/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-s41higuux og:image: https://api.daily.dev/og/posts/s41hIGuuX.png og:image:alt: LLMs for Relevance: Automating High-Quality Product Relevance Labeling in Flipkart Search og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # LLMs for Relevance: Automating High-Quality Product Relevance Labeling in Flipkart Search **[Flipkart Tech](https://daily.dev/sources/flipkart)** · 12 min read · 9 upvotes · 1 comments ## Summary Flipkart's search team replaced manual human annotation for query-product relevance labeling with an LLM-based system called Product Analyser (PA). The system uses a two-stage training approach: Supervised Fine-Tuning (SFT) on gold-standard human-labeled data with balanced sampling across relevance buckets, followed by GRPO (Grouped Relative Policy Optimization) alignment to make reasoning consistent and robust. A Reasoning Reward Model (~8B parameters) acts as a coach, rewarding not just correct labels but well-reasoned explanations. Results show the PA exceeds human annotator accuracy by 2.3–2.9%, outperforms general-purpose proprietary LLMs by 12% at ~30% of the cost, and produces NDCG reports within 1% of manual reports across 22 weeks of data. The system scales to millions of query-product pairs on demand, provides diagnostic reasoning traces, and frees expert annotators for harder edge cases. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is GRPO (Grouped Relative Policy Optimization) and how does it avoid needing a separate critic model? GRPO trains a model by generating a group of answers for the same query, scoring each one, and judging each answer relative to the group's own average score rather than against a separately trained critic model. Answers scoring above the group average are reinforced, those below are discouraged, making the group itself the yardstick and eliminating the cost of training and running a separate grader model. _Engineers comparing RL alignment techniques for LLMs can track approaches like GRPO on daily.dev._ ### How much better did Flipkart's fine-tuned Product Analyser LLM perform compared to human annotators and off-the-shelf LLMs for product relevance labeling? The fine-tuned model beat human annotator benchmarks by 2.3% on three-level accuracy (Relevant/Partially Relevant/Irrelevant) and 2.9% on binary accuracy, and outperformed best general-purpose proprietary LLMs used off the shelf by 12%, while costing only about 30% as much to run. _Teams evaluating custom LLM fine-tuning against off-the-shelf models can follow results like these on daily.dev._ ### How can I generate high-quality reasoning traces for training an LLM without paying humans to write every explanation? Have the LLM generate multiple candidate reasoning paths for the same query-product pair, then keep only the reasoning that demonstrably arrives at the correct gold label (determined by majority vote among human annotators), discarding reasoning that reaches the wrong conclusion even if plausible-sounding. This generate-filter-drop approach scaled a training set from a few thousand examples to millions. _Developers building LLM training pipelines can follow data curation techniques like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@agustinbarrientos** · 0 upvotes > How often does the Product Analyser disagree with all three human annotators, and who gets the final say on those edge cases? ## Similar posts on daily.dev - [LLM-Powered Relevance Assessment for Pinterest Search](https://daily.dev/posts/llm-powered-relevance-assessment-for-pinterest-search-zx5akjrjt) · Pinterest Engineering · 1 upvotes · 0 comments - [LLM-as-a-Judge: Evaluating natural language search](https://daily.dev/posts/llm-as-a-judge-evaluating-natural-language-search-qzclaql93) · Doordash · 0 upvotes · 0 comments - [Scaling Human Judgment: How Dropbox Uses LLMs to Improve Labeling for RAG Systems](https://daily.dev/posts/scaling-human-judgment-how-dropbox-uses-llms-to-improve-labeling-for-rag-systems-vsvb6du0n) · InfoQ · 1 upvotes · 0 comments - [Our Early Journey to Transform Instacart’s Discovery Recommendations with LLMs](https://daily.dev/posts/our-early-journey-to-transform-instacart-s-discovery-recommendations-with-llms-wpv1ryxzx) · Instacart · 1 upvotes · 0 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#llm](https://daily.dev/tags/llm), [#deep-learning](https://daily.dev/tags/deep-learning), [#reinforcement-learning](https://daily.dev/tags/reinforcement-learning) [View this post on daily.dev](https://daily.dev/posts/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-s41higuux) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"LLMs for Relevance: Automating High-Quality Product Relevance Labeling in Flipkart Search","url":"https://daily.dev/posts/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-s41higuux","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-s41higuux"},"datePublished":"2026-08-03T12:36:02.336Z","dateModified":"2026-09-14T08:04:59.569Z","description":"Flipkart's search team replaced manual human annotation for query-product relevance labeling with an LLM-based system called Product Analyser (PA). The system...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/063912808045f7e24b693f18bb5923ee?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/063912808045f7e24b693f18bb5923ee?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Flipkart Tech","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Flipkart Tech","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/37f0fa00abf44cfc963bf3f3fd888b7e","url":"https://daily.dev/sources/flipkart"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-s41higuux","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":9},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"data-science,llm,deep-learning,reinforcement-learning","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Flipkart Tech","item":"https://daily.dev/sources/flipkart"},{"@type":"ListItem","position":3,"name":"LLMs for Relevance: Automating High-Quality Product Relevance Labeling in Flipkart Search"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-s41higuux","comment":[{"@type":"Comment","text":"How often does the Product Analyser disagree with all three human annotators, and who gets the final say on those edge cases?","datePublished":"2026-08-03T17:16:23.313Z","url":"https://daily.dev/posts/s41hIGuuX#c-LHKdfazFa","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-s41higuux#faq","mainEntity":[{"@type":"Question","name":"What is GRPO (Grouped Relative Policy Optimization) and how does it avoid needing a separate critic model?","acceptedAnswer":{"@type":"Answer","text":"GRPO trains a model by generating a group of answers for the same query, scoring each one, and judging each answer relative to the group's own average score rather than against a separately trained critic model. Answers scoring above the group average are reinforced, those below are discouraged, making the group itself the yardstick and eliminating the cost of training and running a separate grader model. Engineers comparing RL alignment techniques for LLMs can track approaches like GRPO on daily.dev."}},{"@type":"Question","name":"How much better did Flipkart's fine-tuned Product Analyser LLM perform compared to human annotators and off-the-shelf LLMs for product relevance labeling?","acceptedAnswer":{"@type":"Answer","text":"The fine-tuned model beat human annotator benchmarks by 2.3% on three-level accuracy (Relevant/Partially Relevant/Irrelevant) and 2.9% on binary accuracy, and outperformed best general-purpose proprietary LLMs used off the shelf by 12%, while costing only about 30% as much to run. Teams evaluating custom LLM fine-tuning against off-the-shelf models can follow results like these on daily.dev."}},{"@type":"Question","name":"How can I generate high-quality reasoning traces for training an LLM without paying humans to write every explanation?","acceptedAnswer":{"@type":"Answer","text":"Have the LLM generate multiple candidate reasoning paths for the same query-product pair, then keep only the reasoning that demonstrably arrives at the correct gold label (determined by majority vote among human annotators), discarding reasoning that reaches the wrong conclusion even if plausible-sounding. This generate-filter-drop approach scaled a training set from a few thousand examples to millions. Developers building LLM training pipelines can follow data curation techniques like this on daily.dev."}}]} ```