--- title: Machine Learning Blog | ML@CMU | Carnegie Mellon University description: LumberChunker is a document segmentation method that uses an LLM to detect semantic boundaries in long-form narrative text, producing more coherent chunks for... canonical: https://daily.dev/posts/machine-learning-blog-ml-cmu-carnegie-mellon-university-vxdxthphs twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Machine Learning Blog | ML@CMU | Carnegie Mellon University | daily.dev og:description: LumberChunker is a document segmentation method that uses an LLM to detect semantic boundaries in long-form narrative text, producing more coherent chunks for... og:url: https://daily.dev/posts/machine-learning-blog-ml-cmu-carnegie-mellon-university-vxdxthphs og:image: https://api.daily.dev/og/posts/VxdXtHphS.png og:image:alt: Machine Learning Blog | ML@CMU | Carnegie Mellon University og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Machine Learning Blog | ML@CMU | Carnegie Mellon University **[ML CMU](https://daily.dev/sources/mlcmu)** · 7 min read · 0 upvotes · 0 comments ## Summary LumberChunker is a document segmentation method that uses an LLM to detect semantic boundaries in long-form narrative text, producing more coherent chunks for RAG pipelines. Instead of fixed token windows or structural cues, it feeds rolling groups of paragraphs (around 550 tokens) to a language model and asks it to identify the earliest point where content meaningfully shifts. Evaluated on GutenQA—a new benchmark of 100 public-domain books with 3,000 retrieval questions—LumberChunker achieves DCG ≈ 62.1% and Recall ≈ 77.9% at k=20, outperforming semantic, recursive, paragraph-level, and proposition-level chunking methods. Downstream QA results show that focused retrieval with LumberChunker beats large-context non-retrieval setups and approaches human-curated segmentation quality. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Essential Chunking Techniques for Building Better LLM Applications](https://daily.dev/posts/essential-chunking-techniques-for-building-better-llm-applications-hr2idjqb4) · Machine Learning Mastery · 2 upvotes · 0 comments - [\[2510.02361\] ChunkLLM: A Lightweight Pluggable Framework for Accelerating LLMs Inference](https://daily.dev/posts/2510-02361-chunkllm-a-lightweight-pluggable-framework-for-accelerating-llms-inference-zp24bruoi) · Hacker News · 4 upvotes · 0 comments - [The Chunking Dilemma: Why RAG Fails Before Retrieval Even Starts](https://daily.dev/posts/the-chunking-dilemma-why-rag-fails-before-retrieval-even-starts-wj6xg3kix) · Medium · 2 upvotes · 0 comments - [RAG Explained Simply with a Real Project](https://daily.dev/posts/rag-explained-simply-with-a-real-project-pgy15jzg7) · freeCodeCamp · 23 upvotes · 1 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#nlp](https://daily.dev/tags/nlp), [#rag](https://daily.dev/tags/rag) [View this post on daily.dev](https://daily.dev/posts/machine-learning-blog-ml-cmu-carnegie-mellon-university-vxdxthphs) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Machine Learning Blog | ML@CMU | Carnegie Mellon University","url":"https://daily.dev/posts/machine-learning-blog-ml-cmu-carnegie-mellon-university-vxdxthphs","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/machine-learning-blog-ml-cmu-carnegie-mellon-university-vxdxthphs"},"datePublished":"2026-03-17T17:04:22.069Z","dateModified":"2026-03-17T17:04:51.440Z","description":"LumberChunker is a document segmentation method that uses an LLM to detect semantic boundaries in long-form narrative text, producing more coherent chunks for...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/03e1d8a7b88e5a9cf87357c0e87406c3?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/03e1d8a7b88e5a9cf87357c0e87406c3?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ML CMU","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ML CMU","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/f6a0448c0a90422d9a19c706a926c0ad","url":"https://daily.dev/sources/mlcmu"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/machine-learning-blog-ml-cmu-carnegie-mellon-university-vxdxthphs","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,nlp,rag","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ML CMU","item":"https://daily.dev/sources/mlcmu"},{"@type":"ListItem","position":3,"name":"Machine Learning Blog | ML@CMU | Carnegie Mellon University"}]} ```