--- title: Scaling document classification to 100k+ labels | daily.dev description: Scaling document classification to taxonomies with 100k+ labels is challenging due to context window limits, hallucination, and cost when using frontier LLMs... canonical: https://daily.dev/posts/scaling-document-classification-to-100k-labels-605vzyplf twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Scaling document classification to 100k+ labels | daily.dev og:description: Scaling document classification to taxonomies with 100k+ labels is challenging due to context window limits, hallucination, and cost when using frontier LLMs... og:url: https://daily.dev/posts/scaling-document-classification-to-100k-labels-605vzyplf og:image: https://api.daily.dev/og/posts/605VzyPLF.png og:image:alt: Scaling document classification to 100k+ labels og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Scaling document classification to 100k+ labels **[databricks](https://daily.dev/sources/databricks)** · 8 min read · 2 upvotes · 0 comments ## Summary Scaling document classification to taxonomies with 100k+ labels is challenging due to context window limits, hallucination, and cost when using frontier LLMs directly. Databricks benchmarks three approaches: pure vector search using Qwen3-Embedding-8B with hybrid BM25 and semantic scoring, a two-step workflow combining vector search retrieval with the Databricks AI Classify function, and direct frontier LLM calls (GPT-5.6 Luna, Gemini 3.5 Flash, Claude Sonnet 5, etc.). The hybrid AI Classify workflow achieves 0.81 average accuracy across three datasets (spanning 35k–100k labels), outperforming the best direct frontier model (Gemini 3.5 Flash at 0.76) by five points at roughly 1/100th the per-document cost. The recommended approach shortlists the top 20 labels via vector search, then passes only that shortlist to AI Classify, avoiding context window overflow and reducing token costs dramatically. Taxonomy updates require only re-embedding new labels with no model retraining. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What accuracy does combining vector search with AI Classify achieve compared to calling a frontier LLM directly for large-taxonomy document classification? A two-step workflow that shortlists the top 20 labels via vector search before passing them to the Databricks AI Classify function reaches 0.81 average accuracy across three benchmarks spanning 35,000 to 100,000 labels, beating the strongest direct frontier model tested, Gemini 3.5 Flash, at 0.76, while costing roughly a hundredth as much per document. _Teams weighing LLM classification approaches can track workflow benchmarks like this one on daily.dev._ ### Why do large language models struggle when classifying documents against taxonomies with hundreds of thousands of labels? At that scale, models struggle to fit and reason over the full label list within their context window, begin hallucinating labels that do not exist in the taxonomy, and become costly to run across many documents even with prompt caching. On the MedMentions benchmark, only GPT-5.6 Luna's 1M-token context could hold the entire taxonomy; every other tested model required cropping the label list. _Developers scaling LLM-based classification can follow context-window workarounds like this on daily.dev._ ### How much storage does embedding a 100,000-label taxonomy require for vector search based document classification? Embedding 100k labels requires about 1.6 GB of storage using Qwen3-Embedding-8B 4096-dimensional float32 embeddings, and takes roughly 1-3 minutes using the Databricks AI Query function. This one-time embedding cost does not scale with the number of documents processed afterward, unlike per-document LLM inference costs. _Engineers sizing infrastructure for large-taxonomy classification can compare setups like this on daily.dev._ ## Similar posts on daily.dev - [Your Data Model Is the Semantic Layer](https://daily.dev/posts/your-data-model-is-the-semantic-layer-ro8bktzht) · MotherDuck · 0 upvotes · 0 comments - [How Many Labeled Examples Does a Text Classifier Actually Need? I Measured It.](https://daily.dev/posts/how-many-labeled-examples-does-a-text-classifier-actually-need-i-measured-it--b4sbjuyeq) · Towards Data Science · 0 upvotes · 0 comments - [Databricks Document Intelligence: pushing the frontier for complex document extraction](https://daily.dev/posts/databricks-document-intelligence-pushing-the-frontier-for-complex-document-extraction-lcp55fadj) · databricks · 0 upvotes · 0 comments - [Cut Text2SQL token costs up to 81%, and stop paying for wrong answers: a Neo4j semantic layer across BigQuery and Databricks](https://daily.dev/posts/cut-text2sql-token-costs-up-to-81-and-stop-paying-for-wrong-answers-a-neo4j-semantic-layer-across-w7f20iaii) · neo4j · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#nlp](https://daily.dev/tags/nlp), [#vector-search](https://daily.dev/tags/vector-search), [#databricks](https://daily.dev/tags/databricks) [View this post on daily.dev](https://daily.dev/posts/scaling-document-classification-to-100k-labels-605vzyplf) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Scaling document classification to 100k+ labels","url":"https://daily.dev/posts/scaling-document-classification-to-100k-labels-605vzyplf","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/scaling-document-classification-to-100k-labels-605vzyplf"},"datePublished":"2026-07-20T18:31:30.258Z","dateModified":"2026-09-14T08:45:41.489Z","description":"Scaling document classification to taxonomies with 100k+ labels is challenging due to context window limits, hallucination, and cost when using frontier LLMs...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e38386be9687c134e67b39a0475850f2?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e38386be9687c134e67b39a0475850f2?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"databricks","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"databricks","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/fa7aa720f2db4d1eba826814730482c8","url":"https://daily.dev/sources/databricks"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/scaling-document-classification-to-100k-labels-605vzyplf","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,nlp,vector-search,databricks","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"databricks","item":"https://daily.dev/sources/databricks"},{"@type":"ListItem","position":3,"name":"Scaling document classification to 100k+ labels"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/scaling-document-classification-to-100k-labels-605vzyplf#faq","mainEntity":[{"@type":"Question","name":"What accuracy does combining vector search with AI Classify achieve compared to calling a frontier LLM directly for large-taxonomy document classification?","acceptedAnswer":{"@type":"Answer","text":"A two-step workflow that shortlists the top 20 labels via vector search before passing them to the Databricks AI Classify function reaches 0.81 average accuracy across three benchmarks spanning 35,000 to 100,000 labels, beating the strongest direct frontier model tested, Gemini 3.5 Flash, at 0.76, while costing roughly a hundredth as much per document. Teams weighing LLM classification approaches can track workflow benchmarks like this one on daily.dev."}},{"@type":"Question","name":"Why do large language models struggle when classifying documents against taxonomies with hundreds of thousands of labels?","acceptedAnswer":{"@type":"Answer","text":"At that scale, models struggle to fit and reason over the full label list within their context window, begin hallucinating labels that do not exist in the taxonomy, and become costly to run across many documents even with prompt caching. On the MedMentions benchmark, only GPT-5.6 Luna's 1M-token context could hold the entire taxonomy; every other tested model required cropping the label list. Developers scaling LLM-based classification can follow context-window workarounds like this on daily.dev."}},{"@type":"Question","name":"How much storage does embedding a 100,000-label taxonomy require for vector search based document classification?","acceptedAnswer":{"@type":"Answer","text":"Embedding 100k labels requires about 1.6 GB of storage using Qwen3-Embedding-8B 4096-dimensional float32 embeddings, and takes roughly 1-3 minutes using the Databricks AI Query function. This one-time embedding cost does not scale with the number of documents processed afterward, unlike per-document LLM inference costs. Engineers sizing infrastructure for large-taxonomy classification can compare setups like this on daily.dev."}}]} ```