--- title: "Indexing the Data Lake for Online Point Queries" url: https://daily.dev/posts/indexing-the-data-lake-for-online-point-queries-y3p7o9ihw source_url: https://engineering.atspotify.com/2026/7/indexing-the-data-lake-for-online-point-queries type: article source: "Spotify Labs" published: 2026-07-27T21:02:52.059Z updated: 2026-07-28T02:18:21.468Z tags: ["big-data", "data-lake"] reading_time: 13 upvotes: 7 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Indexing the Data Lake for Online Point Queries **[Spotify Labs](https://daily.dev/sources/spotify)** · 13 min read · 7 upvotes · 0 comments ## Summary Spotify engineers describe Random Access Parquet (RAP), a system that enables low-latency point queries directly against data lake Parquet files without copying data into a key-value store. The core idea is an external index that maps lookup keys to exact file locations and row numbers, replacing the chain of dependent reads that distributed SQL engines like Trino or BigQuery require. The post details several Parquet write-time optimizations — sorting by key, co-grouping, one-page-per-key, ZSTD frame resets, column interleaving, and covering indexes — that can reduce a point query to a single ranged read of a few kilobytes or eliminate the storage read entirely. This approach lets the same files serve both batch analytics and interactive online workloads, including AI agent context retrieval, without duplicating storage or running ETL pipelines. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Spotify Builds External Index to Enable Low Latency Point Queries on Its Data Lake](https://daily.dev/posts/spotify-builds-external-index-to-enable-low-latency-point-queries-on-its-data-lake-utxkvessc) · InfoQ · 0 upvotes · 0 comments - [Parquet Files Explained: Why Every Data Engineer should Use](https://daily.dev/posts/parquet-files-explained-why-every-data-engineer-should-use-mz7jytdxj) · Medium · 1 upvotes · 0 comments --- Tags: [#big-data](https://daily.dev/tags/big-data), [#data-lake](https://daily.dev/tags/data-lake) [View this post on daily.dev](https://daily.dev/posts/indexing-the-data-lake-for-online-point-queries-y3p7o9ihw)