--- title: GitHub - marcelroed/gigatoken: Language model... description: Gigatoken is a new tokenizer library claiming ~1000x faster throughput than HuggingFace's tokenizers and tiktoken for language model tokenization. It achieves... canonical: https://daily.dev/posts/github---marcelroed-gigatoken-language-model-tokenization-at-gb-s-eobew1umo twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: GitHub - marcelroed/gigatoken: Language model tokenization at GB/s | daily.dev og:description: Gigatoken is a new tokenizer library claiming ~1000x faster throughput than HuggingFace's tokenizers and tiktoken for language model tokenization. It achieves... og:url: https://daily.dev/posts/github---marcelroed-gigatoken-language-model-tokenization-at-gb-s-eobew1umo og:image: https://api.daily.dev/og/posts/EoBEw1UMO.png og:image:alt: GitHub - marcelroed/gigatoken: Language model tokenization at GB/s og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # GitHub - marcelroed/gigatoken: Language model tokenization at GB/s **[Hacker News](https://daily.dev/sources/hn)** · 14 min read · 23 upvotes · 3 comments ## Summary Gigatoken is a new tokenizer library claiming ~1000x faster throughput than HuggingFace's tokenizers and tiktoken for language model tokenization. It achieves GB/s speeds through SIMD-optimized pretokenization (replacing regex engines), aggressive caching of pretoken mappings, and minimized Python overhead. It supports most common BPE-based tokenizers (GPT-2, Llama 3, Qwen, DeepSeek, etc.) and offers both a native API and drop-in compatibility modes for HuggingFace and tiktoken. Benchmarks show 24+ GB/s on a 144-core AMD EPYC and 8+ GB/s on Apple M4 Max. SentencePiece-based tokenizers (Gemma, Mistral) see smaller but still significant gains. Install via pip and use with HuggingFace model names directly. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community take How the wider developer community reacted, aggregated from 1 discussion and 208 comments across hackernews (as of 2026-07-23). **TL;DR:** The community is broadly impressed by Gigatoken's ~1000x tokenization speedup, praising the engineering ingenuity and real-world use cases (pretraining data pipelines, TTFT reduction, latency-critical routing). A minority debate whether optimizing <0.1% of inference time is worth the effort, though the author's own benchmarks showing up to 10% TTFT reduction largely defuse that criticism. **Sentiment:** 72% positive · 20% mixed · 8% skeptical **The case for** - Fast tokenization meaningfully reduces time-to-first-token (up to ~10% on benchmarks), which is observable by end users. - Pretraining data pipelines that tokenize terabytes of text benefit enormously, enabling faster iteration cycles. - Latency-critical inference paths (routing, rate limiting, KV-cache lookup) benefit even when overall inference time is dominated by GPU compute. - Even the Python compatibility mode yields a ~200–300x speedup over existing libraries. - The work is architecturally clean, using SIMD, smart caching of pretoken mappings, and minimized Python interop rather than being narrowly tuned to one CPU. **The pushback** - Tokenization is typically <0.1% of total inference time, so Amdahl's Law limits the end-to-end impact for most workloads. - CPU tokenization is largely hidden by GPU compute in practice, making the real-world savings smaller than raw throughput numbers suggest. - A potential correctness concern was raised around 42-bit hash collisions in the caching layer. - Throughput benchmarks use a 12 GiB file; cold-start performance on small inputs (e.g., 32 KiB) is not yet shown. - Adoption requires inference frameworks (vLLM, SGLang, llama.cpp) or HuggingFace to integrate the library. **By community** - hackernews (positive): Commenters are enthusiastic about the engineering achievement and surface concrete use cases (pretraining, TTFT, routing), with only a vocal minority questioning the practical impact given Amdahl's Law. **Hottest debate:** Whether a 1000x speedup on a component that is <0.1% of inference time constitutes meaningful real-world impact, versus the author's evidence of up to 10% TTFT reduction and significant pretraining pipeline savings. **Open questions** - What is the cold-start throughput for small inputs (e.g., 32 KiB) where the pretoken cache is not yet warm? - Could KV-cache prefix lookup be restructured to walk a chunk tree by characters instead of tokens, allowing tokenization and cache lookup to proceed in parallel? - Will HuggingFace, vLLM, or SGLang officially adopt Gigatoken, and what is the integration path? - How does the 42-bit hash collision risk manifest in practice, and what is the failure mode? **Highlights** > Author here: Actually, depending on the nature of the inference you're doing it can be quite significant. Here are some numbers for time-to-first-token (time to process the entire input and produce the first token of output) for an 8B Qwen3 model running on a single B200. Obviously these numbers are more significant with smaller models and on faster GPUs. Credit to fastokens [0] for the benchmark. sglang_speed [huggingface]: mean=10.31ms median=6.48ms p99=45.98ms rps=96.8 sglang_speed [gigatoken]: mean=10.13ms median=6.54ms p99=45.16ms rps=98.4 input_len= 2048: TTFT mean 30.74 -> 29.05 ms (+5.5% reduction) | median 31.00 -> 28.80 (+7.1%) | p99 33.02 -> 32.02 (+3.0%) input_len= 8192: TTFT mean 105.20 -> 96.36 ms (+8.4% reduction) | median 103.87 -> 95.49 (+8.1%) | p99 126.88 -> 113.84 (+10.3%) input_len= 32768: TTFT mean 687.05 -> 633.66 ms (+7.8% reduction) | median 708.14 -> 657.35 (+7.2%) | p99 728.95 -> 678.79 (+6.9%) These are preliminary numbers, so I will need to do some more testing before including this in the README. [0] https://github.com/crusoecloud/fastokens > — [marcelroed on hackernews](https://news.ycombinator.com/item?id=49015014) > I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request. > — [scottcha on hackernews · 4 comments](https://news.ycombinator.com/item?id=49012078) > Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM. From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0] I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo. [0] https://github.com/crusoecloud/fastokens > — [marcelroed on hackernews · 3 comments](https://news.ycombinator.com/item?id=49011028) > Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas. And screw all the 0.1% haters on here, this is great stuff. > — [cschmidt on hackernews · 4 comments](https://news.ycombinator.com/item?id=49015607) > hm, maybe not so trivially correct here. Do I understand correctly that incorrect results can happen as a result of a 42-bit hash collision? That could happen after less than one MB of input, given the simple one-mul hash. BTW throughput is measured for a 12 GiB file. Would be interesting to see the throughput for something more like 32 KiB, with cold start (token cache not yet populated). > — [janwas on hackernews](https://news.ycombinator.com/item?id=49017220) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49010167) · 472 points · 208 comments ## Community discussion Top comments from developers on daily.dev. **@lfaoliveira** · 1 upvotes > If these results actually hold up in production, It is a fenomenal engineering feat. Even if it doesnt (personally think it is unlikely), keep going in the direction of LLM-related optimizations, there is a world of possibility there **@jtgsystems** · 0 upvotes > rust is where its at these days. ## Similar posts on daily.dev - [Google’s DiffusionGemma is 4x faster than its other Gemma models](https://daily.dev/posts/google-s-diffusiongemma-is-4x-faster-than-its-other-gemma-models-vjw4zukla) · The New Stack · 2 upvotes · 0 comments - [Gemma 4 Multi-Token Prediction Delivers Up to ~3x Faster Token Generation](https://daily.dev/posts/gemma-4-multi-token-prediction-delivers-up-to-3x-faster-token-generation-n9rqcmn5a) · InfoQ · 1 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#nlp](https://daily.dev/tags/nlp), [#rust](https://daily.dev/tags/rust) [View this post on daily.dev](https://daily.dev/posts/github---marcelroed-gigatoken-language-model-tokenization-at-gb-s-eobew1umo) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"GitHub - marcelroed/gigatoken: Language model tokenization at GB/s","url":"https://daily.dev/posts/github---marcelroed-gigatoken-language-model-tokenization-at-gb-s-eobew1umo","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/github---marcelroed-gigatoken-language-model-tokenization-at-gb-s-eobew1umo"},"datePublished":"2026-07-22T18:43:17.958Z","dateModified":"2026-07-23T13:53:51.377Z","description":"Gigatoken is a new tokenizer library claiming ~1000x faster throughput than HuggingFace's tokenizers and tiktoken for language model tokenization. It achieves...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/687903981e9957c07fc39902237fdc06?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/687903981e9957c07fc39902237fdc06?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":3,"discussionUrl":"https://daily.dev/posts/github---marcelroed-gigatoken-language-model-tokenization-at-gb-s-eobew1umo","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":23},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":3}],"keywords":"llm,nlp,rust","timeRequired":"PT14M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"GitHub - marcelroed/gigatoken: Language model tokenization at GB/s"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/github---marcelroed-gigatoken-language-model-tokenization-at-gb-s-eobew1umo","comment":[{"@type":"Comment","text":"If these results actually hold up in production, It is a fenomenal engineering feat. Even if it doesnt (personally think it is unlikely), keep going in the direction of LLM-related optimizations, there is a world of possibility there","datePublished":"2026-07-23T12:21:17.859Z","url":"https://daily.dev/posts/EoBEw1UMO#c-WmgFXdpHT","author":{"@type":"Person","name":"Luis Felipe","url":"https://daily.dev/lfaoliveira","image":"https://media.daily.dev/image/upload/s--O0TOmw4y--/f_auto/v1715772965/public/noProfile"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"rust is where its at these days.","datePublished":"2026-07-29T07:41:55.572Z","url":"https://daily.dev/posts/EoBEw1UMO#c-1RnDmFLf6","author":{"@type":"Person","name":"John Gallie","url":"https://daily.dev/jtgsystems","image":"https://lh3.googleusercontent.com/a/AEdFTp6DSayJzUafmDYsZ3NsMm_v-1lKJFdfIVNKu0x6Qw=s96-c"}}]} ```