Spotify engineers describe Random Access Parquet (RAP), a system that enables low-latency point queries directly against data lake Parquet files without copying data into a key-value store. The core idea is an external index that maps lookup keys to exact file locations and row numbers, replacing the chain of dependent reads that distributed SQL engines like Trino or BigQuery require. The post details several Parquet write-time optimizations — sorting by key, co-grouping, one-page-per-key, ZSTD frame resets, column interleaving, and covering indexes — that can reduce a point query to a single ranged read of a few kilobytes or eliminate the storage read entirely. This approach lets the same files serve both batch analytics and interactive online workloads, including AI agent context retrieval, without duplicating storage or running ETL pipelines.
