--- title: Indexing the Data Lake for Online Point Queries | daily.dev description: Spotify engineers describe Random Access Parquet (RAP), a system that enables low-latency point queries directly against data lake Parquet files without... canonical: https://daily.dev/posts/indexing-the-data-lake-for-online-point-queries-y3p7o9ihw twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Indexing the Data Lake for Online Point Queries | daily.dev og:description: Spotify engineers describe Random Access Parquet (RAP), a system that enables low-latency point queries directly against data lake Parquet files without... og:url: https://daily.dev/posts/indexing-the-data-lake-for-online-point-queries-y3p7o9ihw og:image: https://api.daily.dev/og/posts/y3p7o9Ihw.png og:image:alt: Indexing the Data Lake for Online Point Queries og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Indexing the Data Lake for Online Point Queries **[Spotify Labs](https://daily.dev/sources/spotify)** · 13 min read · 7 upvotes · 0 comments ## Summary Spotify engineers describe Random Access Parquet (RAP), a system that enables low-latency point queries directly against data lake Parquet files without copying data into a key-value store. The core idea is an external index that maps lookup keys to exact file locations and row numbers, replacing the chain of dependent reads that distributed SQL engines like Trino or BigQuery require. The post details several Parquet write-time optimizations — sorting by key, co-grouping, one-page-per-key, ZSTD frame resets, column interleaving, and covering indexes — that can reduce a point query to a single ranged read of a few kilobytes or eliminate the storage read entirely. This approach lets the same files serve both batch analytics and interactive online workloads, including AI agent context retrieval, without duplicating storage or running ETL pipelines. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is Random Access Parquet (RAP) and how does it enable fast point queries on data lake files? Random Access Parquet is an external indexing technique that maps keys directly to the file, row number, and location where their data resides in existing Parquet files. Instead of scanning files with a distributed SQL engine, a reader looks up the index (O(1)), resolves row numbers to page locations using cached metadata, and issues parallel ranged reads to fetch exactly the needed bytes, avoiding the chain of dependent reads that normally makes point lookups slow. _daily.dev surfaces engineering deep dives like this for teams weighing lake-native indexing against maintaining a separate key-value store._ ### Why are Trino and BigQuery slow for single-row lookups even though cloud storage itself is fast? Trino and BigQuery add seconds of job scheduling and query planning overhead per query because they are built for analytical throughput across large scans, not single-row interactive lookups. Meanwhile GCS delivers 30-100ms per request and newer options like S3 Express One Zone and GCS Rapid Storage reach single-digit millisecond latency, meaning the query engine, not the storage layer, has become the bottleneck for point queries. _engineers debating whether to route point lookups through Trino/BigQuery or a dedicated index can compare tradeoffs like these on daily.dev._ ### How does one-page-per-key layout reduce Parquet read overhead compared to standard Parquet pages? Flushing a new page whenever the key changes means an entire decompressed page belongs to a single key, so no row extraction is needed and page locations can be stored directly in the index entry, eliminating the need for a separate page index. Each key boundary only costs about 20 bytes of page header overhead, and a side effect is that PageIndex size grows with the number of keys rather than the number of conventional pages. _teams tuning Parquet file layout for low-latency reads can track techniques like this via daily.dev._ ## Similar posts on daily.dev - [Spotify Builds External Index to Enable Low Latency Point Queries on Its Data Lake](https://daily.dev/posts/spotify-builds-external-index-to-enable-low-latency-point-queries-on-its-data-lake-utxkvessc) · InfoQ · 0 upvotes · 0 comments - [Parquet Files Explained: Why Every Data Engineer should Use](https://daily.dev/posts/parquet-files-explained-why-every-data-engineer-should-use-mz7jytdxj) · Medium · 1 upvotes · 0 comments - [Lakehouses: the path to low-cost, infinitely scalable, and no-lock-in observability?](https://daily.dev/posts/lakehouses-the-path-to-low-cost-infinitely-scalable-and-no-lock-in-observability--ihpmenhk9) · ClickHouse · 0 upvotes · 0 comments - [Beyond Indexes: How Open Table Formats Optimize Query Performance — Jack Vanlightly](https://daily.dev/posts/beyond-indexes-how-open-table-formats-optimize-query-performance-jack-vanlightly-inrhgb2yf) · Programming Digest · 0 upvotes · 0 comments - [From monolith to Lakebase to LTAP: rethinking the database from storage up](https://daily.dev/posts/from-monolith-to-lakebase-to-ltap-rethinking-the-database-from-storage-up-gsup5t8nr) · databricks · 6 upvotes · 0 comments --- Tags: [#big-data](https://daily.dev/tags/big-data), [#data-lake](https://daily.dev/tags/data-lake) [View this post on daily.dev](https://daily.dev/posts/indexing-the-data-lake-for-online-point-queries-y3p7o9ihw) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Indexing the Data Lake for Online Point Queries","url":"https://daily.dev/posts/indexing-the-data-lake-for-online-point-queries-y3p7o9ihw","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/indexing-the-data-lake-for-online-point-queries-y3p7o9ihw"},"datePublished":"2026-07-27T21:02:52.059Z","dateModified":"2026-09-14T07:51:34.693Z","description":"Spotify engineers describe Random Access Parquet (RAP), a system that enables low-latency point queries directly against data lake Parquet files without...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d2704bae42e9bb483bda07392ef5e676?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d2704bae42e9bb483bda07392ef5e676?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Spotify Labs","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Spotify Labs","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/spotify","url":"https://daily.dev/sources/spotify"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/indexing-the-data-lake-for-online-point-queries-y3p7o9ihw","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":7},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"big-data,data-lake","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Spotify Labs","item":"https://daily.dev/sources/spotify"},{"@type":"ListItem","position":3,"name":"Indexing the Data Lake for Online Point Queries"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/indexing-the-data-lake-for-online-point-queries-y3p7o9ihw#faq","mainEntity":[{"@type":"Question","name":"What is Random Access Parquet (RAP) and how does it enable fast point queries on data lake files?","acceptedAnswer":{"@type":"Answer","text":"Random Access Parquet is an external indexing technique that maps keys directly to the file, row number, and location where their data resides in existing Parquet files. Instead of scanning files with a distributed SQL engine, a reader looks up the index (O(1)), resolves row numbers to page locations using cached metadata, and issues parallel ranged reads to fetch exactly the needed bytes, avoiding the chain of dependent reads that normally makes point lookups slow. daily.dev surfaces engineering deep dives like this for teams weighing lake-native indexing against maintaining a separate key-value store."}},{"@type":"Question","name":"Why are Trino and BigQuery slow for single-row lookups even though cloud storage itself is fast?","acceptedAnswer":{"@type":"Answer","text":"Trino and BigQuery add seconds of job scheduling and query planning overhead per query because they are built for analytical throughput across large scans, not single-row interactive lookups. Meanwhile GCS delivers 30-100ms per request and newer options like S3 Express One Zone and GCS Rapid Storage reach single-digit millisecond latency, meaning the query engine, not the storage layer, has become the bottleneck for point queries. engineers debating whether to route point lookups through Trino/BigQuery or a dedicated index can compare tradeoffs like these on daily.dev."}},{"@type":"Question","name":"How does one-page-per-key layout reduce Parquet read overhead compared to standard Parquet pages?","acceptedAnswer":{"@type":"Answer","text":"Flushing a new page whenever the key changes means an entire decompressed page belongs to a single key, so no row extraction is needed and page locations can be stored directly in the index entry, eliminating the need for a separate page index. Each key boundary only costs about 20 bytes of page header overhead, and a side effect is that PageIndex size grows with the number of keys rather than the number of conventional pages. teams tuning Parquet file layout for low-latency reads can track techniques like this via daily.dev."}}]} ```