DuckDB v2.0 (scheduled for fall 2026) introduces asynchronous I/O for Parquet and CSV files, targeting remote storage scenarios like EC2/S3 setups where synchronous reads leave network bandwidth underutilized. The implementation uses two thread pools — REGULAR workers for computation and ASYNC threads (defaulting to 4× system threads, capped at 256) for blocking I/O — combined with a read-ahead queue and memory-governed backlog control. Benchmarks on TPC-H Q6 at SF100 show Parquet queries running ~3× faster on S3 (8.2s → 2.8s), with a tuned configuration reaching ~3.7× speedup by saturating a 25 Gbit/s network. CSV sees the most dramatic gain: ~20× faster (878s → 45s) due to its row-oriented nature requiring far more data transfer. Cold local SSD reads improve ~1.5×. Concurrent query workloads jump from ~6 average busy cores to ~48 out of 64, nearly eliminating idle wait time. The read-ahead depth is configurable via `read_ahead_depth`, and memory pressure automatically throttles prefetching. Future plans include async reads for JSON and DuckDB-native formats, plus potential io_uring integration.

17m read timeFrom duckdb.org
Post cover image
Table of contents
Asynchronous I/OBenchmarksConclusion
3K Impressions