--- title: Don’t stop early: Case-folding source code at memory speed description: GitHub engineers explain how they achieved >45 GiB/s case-folding throughput for their Blackbird code search engine, which indexes 480TB of source code. The... canonical: https://daily.dev/posts/don-t-stop-early-case-folding-source-code-at-memory-speed-0kmr0gljo twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Don’t stop early: Case-folding source code at memory speed | daily.dev og:description: GitHub engineers explain how they achieved >45 GiB/s case-folding throughput for their Blackbird code search engine, which indexes 480TB of source code. The... og:url: https://daily.dev/posts/don-t-stop-early-case-folding-source-code-at-memory-speed-0kmr0gljo og:image: https://api.daily.dev/og/posts/0Kmr0GljO.png og:image:alt: Don’t stop early: Case-folding source code at memory speed og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Don’t stop early: Case-folding source code at memory speed **[GitHub Blog](https://daily.dev/sources/ghblog)** · 22 min read · 3 upvotes · 0 comments ## Summary GitHub engineers explain how they achieved >45 GiB/s case-folding throughput for their Blackbird code search engine, which indexes 480TB of source code. The key insight is counterintuitive: removing an early-exit branch (that stopped at the first non-ASCII byte) and making the loop entirely branch-free enables LLVM to auto-vectorize it, yielding a 15x speedup over the naive approach. The post details the full design: a branch-free ASCII sweep using byte-space arithmetic, a compact 1776-byte Unicode lookup table using a page bitmap and run-length encoding, and a novel decode-free fold technique that performs UTF-8 case folding as direct byte arithmetic without ever decoding to a code point. The result is open-sourced as the 'casefold' Rust crate. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community take How the wider developer community reacted, aggregated from 3 discussions and 25 comments across hackernews, lobsters (as of 2026-08-05). **TL;DR:** The community finds the technical content genuinely interesting — especially the branch-free SIMD auto-vectorization insight and the compact Unicode table design — but a significant portion of the discussion is dominated by suspicion that the post was AI-written, with some commenters also pointing out factual errors in the introduction's examples. **Sentiment:** 30% positive · 50% mixed · 20% skeptical **The case for** - The core insight of removing an early-exit branch to enable LLVM auto-vectorization is clever and counterintuitive. - The suggestion to reverse the fold direction for the two problematic code points (U+023A, U+023E) could eliminate the extra buffer allocation code path entirely. - A follow-up optimization idea — building a block-wise bitmap during the ASCII pass to skip non-ASCII ranges — was raised as a promising further improvement. **The pushback** - Multiple commenters are confident the post was written or heavily assisted by an LLM, citing abrupt tone shifts, poor paragraph flow, and characteristic phrasing. - The introduction contains factual errors: the ß and İ examples used to distinguish case folding from lowercasing are incorrect for the crate's actual behavior. - The article is longer and more verbose than necessary, making it harder to follow. **By community** - hackernews (mixed): Engagement is split between genuine technical interest — spawning substantive follow-up ideas — and frustration over apparent AI authorship and inaccuracies in the introductory examples. - lobsters (skeptical): The lone comment ignores the technical content entirely, wishing GitHub would apply the same performance mindset to its UI. **Hottest debate:** Whether the post was written by an LLM, and whether AI-generated technical writing is acceptable even when the underlying content is sound. **Open questions** - How does this implementation compare to other highly optimized string libraries like StringZilla? - Would folding to uppercase (instead of lowercase) strictly reduce the number of code points that expand in byte length? - How often do the two problematic expanding code points (U+023A, U+023E) actually appear in real-world source code, making the extra-buffer path worth optimizing? **Highlights** > TLDR: they implemented case folding with a lot more SIMD via autovectorization. > almost every fold preserves the UTF-8 length or shrinks it, but two outliers grow—U+023A (Ⱥ) and U+023E (Ɀ) are 2 bytes each yet fold to 3-byte characters (ⱥ, ɀ) Fix this by reversing it. Fold ⱥ to Ⱥ instead of the other way around. The search index won't only consist of lowercase characters any more, but that never mattered. > — [inigyou on hackernews · 5 comments](https://news.ycombinator.com/item?id=49175881) > There’s some interesting information in there. Unfortunately the person or LLM writing this got pretty confused right in the introduction already. > “Suppose […] they type straße and you’ve stored STRASSE. To make these count as matches, you need […]” Really bad example, because as the article says later on, this casefold crate won’t match those two strings because the ß → ss conversion isn’t done. > “[str::to_lowercase and case folding] diverge on real characters—ß, İ, final sigma” The main point is true (case folding is different from lowercasing), but two of the three examples are wrong. The casefold operation that they use maps ß to itself, as does str::to_lowercase. The casefold operation maps İ to U+0069 U+0307 regardless of locale, as does str::to_lowercase. When I’m reading an article, these kind of mistakes in the introduction make me doubt the accuracy of the whole article. Which is a shame, because again, it’s an interesting write-up. The mistakes also make the article harder to follow, since the examples imply ß is folded to ss. > — [lukasgelbmann on hackernews](https://news.ycombinator.com/item?id=49178496) > One thing that wasn't tried is that the ASCII path could build a block-wise bitmap of non-ASCII blocks. Then the unicode pass need only process the contiguous ranges within the bitmap. This would be simpler to implement when combined with inigyou's no-reallocate suggestion. > — [RossBencina on hackernews](https://news.ycombinator.com/item?id=49178831) > Agreed. This is genuinely interesting content, but there is no doubt in my mind that "The two operations diverge on real characters—ß, İ, final sigma—which is why lowercasing as a stand-in silently produces wrong matches." is LLM output. Are we doomed to spend the rest of our professional and personal lives reading AI output? > — [agency on hackernews · 1 comments](https://news.ycombinator.com/item?id=49175873) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49127983) · 61 points · 24 comments - [hackernews](https://news.ycombinator.com/item?id=49134468) · 3 points · 0 comments - [lobsters](https://lobste.rs/s/qsgcgz/don_t_stop_early_case_folding_source_code) · 3 points · 1 comments ## Similar posts on daily.dev - [Full Unicode Search at 50× ICU Speed with AVX‑512](https://daily.dev/posts/full-unicode-search-at-50-icu-speed-with-avx-512-s9bwdg8zf) · Hacker News · 0 upvotes · 0 comments - [6× faster binary search: from compiled code to mechanical sympathy](https://daily.dev/posts/6-faster-binary-search-from-compiled-code-to-mechanical-sympathy-kooq3tqdq) · Planet Python · 0 upvotes · 0 comments - [Rust zero-cost abstractions vs. SIMD](https://daily.dev/posts/rust-zero-cost-abstractions-vs-simd-u3pwvyd72) · Lobsters · 34 upvotes · 2 comments --- Tags: [#performance](https://daily.dev/tags/performance), [#rust](https://daily.dev/tags/rust) [View this post on daily.dev](https://daily.dev/posts/don-t-stop-early-case-folding-source-code-at-memory-speed-0kmr0gljo) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Don’t stop early: Case-folding source code at memory speed","url":"https://daily.dev/posts/don-t-stop-early-case-folding-source-code-at-memory-speed-0kmr0gljo","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/don-t-stop-early-case-folding-source-code-at-memory-speed-0kmr0gljo"},"datePublished":"2026-07-31T16:04:49.562Z","dateModified":"2026-08-05T05:33:20.300Z","description":"GitHub engineers explain how they achieved >45 GiB/s case-folding throughput for their Blackbird code search engine, which indexes 480TB of source code. The...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9ea263f0304dda09a41d0cdb582de2ec?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9ea263f0304dda09a41d0cdb582de2ec?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"GitHub Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"GitHub Blog","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/106cf162b88840808484d4b5429b59b1","url":"https://daily.dev/sources/ghblog"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/don-t-stop-early-case-folding-source-code-at-memory-speed-0kmr0gljo","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"performance,rust","timeRequired":"PT22M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"GitHub Blog","item":"https://daily.dev/sources/ghblog"},{"@type":"ListItem","position":3,"name":"Don’t stop early: Case-folding source code at memory speed"}]} ```