--- title: Spotify Builds External Index to Enable Low Latency... description: Spotify introduced Random Access Parquet (RAP), a storage architecture that adds an external indexing layer over Apache Parquet files to enable low-latency... canonical: https://daily.dev/posts/spotify-builds-external-index-to-enable-low-latency-point-queries-on-its-data-lake-utxkvessc twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Spotify Builds External Index to Enable Low Latency Point Queries on Its Data Lake | daily.dev og:description: Spotify introduced Random Access Parquet (RAP), a storage architecture that adds an external indexing layer over Apache Parquet files to enable low-latency... og:url: https://daily.dev/posts/spotify-builds-external-index-to-enable-low-latency-point-queries-on-its-data-lake-utxkvessc og:image: https://api.daily.dev/og/posts/UtxKvessc.png og:image:alt: Spotify Builds External Index to Enable Low Latency Point Queries on Its Data Lake og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Spotify Builds External Index to Enable Low Latency Point Queries on Its Data Lake **[InfoQ](https://daily.dev/sources/infoq)** · 3 min read · 0 upvotes · 0 comments ## Summary Spotify introduced Random Access Parquet (RAP), a storage architecture that adds an external indexing layer over Apache Parquet files to enable low-latency point queries directly against its data lake. The index maps lookup keys like user IDs to specific Parquet files and row locations, letting queries perform targeted ranged reads on Google Cloud Storage instead of scanning thousands of files, avoiding replication into operational databases like Bigtable. Index fragments are built append-only as new data lands in Apache Iceberg tables, keeping Parquet files immutable. Additional layout optimizations - sorting by lookup key, grouping related records, interleaving value columns, covering indexes, and secondary indexes with Z-order or Hilbert curve locality - further cut latency, with some queries served by a single ranged read of a few kilobytes. The design drew commentary from data engineers Andrew Lamb and Vikas Singh on how object storage speed shifts the bottleneck toward query planning and metadata access. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does Spotify's Random Access Parquet (RAP) enable low latency point queries on a data lake? RAP adds an external indexing layer that maps lookup keys, such as user IDs, directly to specific Parquet files and row locations, letting a query resolve the key through the index and issue a single targeted ranged read against object storage instead of scanning thousands of files. Index fragments are generated append-only as data lands in Apache Iceberg tables, leaving Parquet files immutable. _See how teams are evolving data lake architecture for real-time lookups by following backend engineering coverage on daily.dev._ ### What storage layout techniques reduce point query latency on Parquet data lakes? Sorting data by lookup key, grouping related records together, interleaving value columns for single contiguous reads, and using covering indexes all reduce the number of storage operations needed. These trade modest increases in file or index size for fewer reads, and some point queries can be served through a single ranged read of only a few kilobytes. _Engineers optimizing storage layouts for latency can track these architecture patterns on daily.dev._ ## Similar posts on daily.dev - [Indexing the Data Lake for Online Point Queries](https://daily.dev/posts/indexing-the-data-lake-for-online-point-queries-y3p7o9ihw) · Spotify Labs · 7 upvotes · 0 comments - [Parquet Files Explained: Why Every Data Engineer should Use](https://daily.dev/posts/parquet-files-explained-why-every-data-engineer-should-use-mz7jytdxj) · Medium · 1 upvotes · 0 comments - [Parquet and Iceberg: An Overview — CommandPrompt Inc.](https://daily.dev/posts/parquet-and-iceberg-an-overview-commandprompt-inc--pv4vposch) · Planet PostgreSQL · 0 upvotes · 0 comments - [Apache Iceberg vs Apache Parquet Explained](https://daily.dev/posts/apache-iceberg-vs-apache-parquet-explained-9kadpsw5t) · Dremio Blog · 0 upvotes · 0 comments - [Borderless Lakehouse cross-cloud caching and connections](https://daily.dev/posts/borderless-lakehouse-cross-cloud-caching-and-connections-xctvphcsx) · Google Cloud · 0 upvotes · 0 comments --- Tags: [#distributed-systems](https://daily.dev/tags/distributed-systems), [#spotify](https://daily.dev/tags/spotify), [#data-lake](https://daily.dev/tags/data-lake), [#apache-iceberg](https://daily.dev/tags/apache-iceberg) [View this post on daily.dev](https://daily.dev/posts/spotify-builds-external-index-to-enable-low-latency-point-queries-on-its-data-lake-utxkvessc) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Spotify Builds External Index to Enable Low Latency Point Queries on Its Data Lake","url":"https://daily.dev/posts/spotify-builds-external-index-to-enable-low-latency-point-queries-on-its-data-lake-utxkvessc","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/spotify-builds-external-index-to-enable-low-latency-point-queries-on-its-data-lake-utxkvessc"},"datePublished":"2026-08-12T14:42:45.687Z","dateModified":"2026-08-12T14:43:10.130Z","description":"Spotify introduced Random Access Parquet (RAP), a storage architecture that adds an external indexing layer over Apache Parquet files to enable low-latency...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ec634cab7bb81233b2e0fdf954689916?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ec634cab7bb81233b2e0fdf954689916?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"InfoQ","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"InfoQ","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/afc3bced3e1e4b188dd9127017a60e0c","url":"https://daily.dev/sources/infoq"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/spotify-builds-external-index-to-enable-low-latency-point-queries-on-its-data-lake-utxkvessc","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"distributed-systems,spotify,data-lake,apache-iceberg","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"InfoQ","item":"https://daily.dev/sources/infoq"},{"@type":"ListItem","position":3,"name":"Spotify Builds External Index to Enable Low Latency Point Queries on Its Data Lake"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/spotify-builds-external-index-to-enable-low-latency-point-queries-on-its-data-lake-utxkvessc#faq","mainEntity":[{"@type":"Question","name":"How does Spotify's Random Access Parquet (RAP) enable low latency point queries on a data lake?","acceptedAnswer":{"@type":"Answer","text":"RAP adds an external indexing layer that maps lookup keys, such as user IDs, directly to specific Parquet files and row locations, letting a query resolve the key through the index and issue a single targeted ranged read against object storage instead of scanning thousands of files. Index fragments are generated append-only as data lands in Apache Iceberg tables, leaving Parquet files immutable. See how teams are evolving data lake architecture for real-time lookups by following backend engineering coverage on daily.dev."}},{"@type":"Question","name":"What storage layout techniques reduce point query latency on Parquet data lakes?","acceptedAnswer":{"@type":"Answer","text":"Sorting data by lookup key, grouping related records together, interleaving value columns for single contiguous reads, and using covering indexes all reduce the number of storage operations needed. These trade modest increases in file or index size for fewer reads, and some point queries can be served through a single ranged read of only a few kilobytes. Engineers optimizing storage layouts for latency can track these architecture patterns on daily.dev."}}]} ```