A tutorial series on wrapping Karpathy's nanochat educational LLM project with Dagster for orchestration. Covers downloading and partitioning 1,823 FineWeb Parquet shards as Dagster assets, building a high-speed BPE tokenizer using a Rust extension (rustbpe) via PyO3 bindings with Apache Arrow for zero-copy Parquet reads, and promoting tokenizer validation checks to first-class Dagster asset checks. The approach makes the full LLM training pipeline modular, observable, and reproducible.
