--- title: "From NanoChat to Dagster: Building Your Ingestion Pipeline and Custom Tokenizer" url: https://daily.dev/posts/from-nanochat-to-dagster-building-your-ingestion-pipeline-and-custom-tokenizer-rz93amuvy source_url: https://dagster.io/blog/orchestrating-nanochat-building-the-tokenizer type: article source: "Dagster Blog" published: 2026-05-31T07:44:06.521Z updated: 2026-06-04T02:16:57.685Z tags: ["backend", "dagster", "nlp", "rust"] reading_time: 7 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # From NanoChat to Dagster: Building Your Ingestion Pipeline and Custom Tokenizer **[Dagster Blog](https://daily.dev/sources/dagster-blog)** · 7 min read · 0 upvotes · 0 comments ## Summary A tutorial series on wrapping Karpathy's nanochat educational LLM project with Dagster for orchestration. Covers downloading and partitioning 1,823 FineWeb Parquet shards as Dagster assets, building a high-speed BPE tokenizer using a Rust extension (rustbpe) via PyO3 bindings with Apache Arrow for zero-copy Parquet reads, and promoting tokenizer validation checks to first-class Dagster asset checks. The approach makes the full LLM training pipeline modular, observable, and reproducible. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Creative Dagster Projects from the Community](https://daily.dev/posts/creative-dagster-projects-from-the-community-ibhvtpjdv) · Dagster Blog · 1 upvotes · 0 comments --- Tags: [#backend](https://daily.dev/tags/backend), [#dagster](https://daily.dev/tags/dagster), [#nlp](https://daily.dev/tags/nlp), [#rust](https://daily.dev/tags/rust) [View this post on daily.dev](https://daily.dev/posts/from-nanochat-to-dagster-building-your-ingestion-pipeline-and-custom-tokenizer-rz93amuvy)