--- title: Efficient Delta Vacuum with File Inventory | daily.dev description: Grab's data engineering team describes how they cut Delta Lake vacuum times on large partitioned tables by switching from recursive file listing to an... canonical: https://daily.dev/posts/efficient-delta-vacuum-with-file-inventory-qwa46ea1n twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Efficient Delta Vacuum with File Inventory | daily.dev og:description: Grab's data engineering team describes how they cut Delta Lake vacuum times on large partitioned tables by switching from recursive file listing to an... og:url: https://daily.dev/posts/efficient-delta-vacuum-with-file-inventory-qwa46ea1n og:image: https://api.daily.dev/og/posts/Qwa46Ea1n.png og:image:alt: Efficient Delta Vacuum with File Inventory og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Efficient Delta Vacuum with File Inventory **[Delta Lake Blog](https://daily.dev/sources/delta-lake)** · 7 min read · 0 upvotes · 0 comments ## Summary Grab's data engineering team describes how they cut Delta Lake vacuum times on large partitioned tables by switching from recursive file listing to an inventory-based vacuum, a feature released in Delta 3.2. Instead of relying on slow, throttled S3 list API calls, the new approach uses object store inventory reports (from S3, Azure ADLS, or GCS) as a precomputed file list, turning vacuum into a two-stage comparison-and-delete process. This reduced vacuum runtime from 30-60 minutes to under 5 minutes, cut listing costs by roughly 44%, and saved the team $30-40 per day on compute and storage for one large table, enabling more frequent vacuum runs across many tables in a single job. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is inventory-based vacuum in Delta Lake and which version introduced it? Inventory-based vacuum is a Delta Lake feature, released in Delta 3.2, that lets vacuum use a precomputed file inventory (from an object store inventory report or a Spark SQL query) instead of performing a recursive file listing. It requires a schema with path, length, isDir, and modificationTime columns, and is invoked with the USING INVENTORY clause, turning vacuum into a two-stage compare-and-delete process. _Track new Delta Lake releases like this one on daily.dev before they hit your production pipelines._ ### Why does Delta Lake vacuum take so long on large partitioned tables in S3? Vacuum's recursive file listing is the bottleneck, since parallelism depends on the cardinality of the top directory levels and partitioning by year/month/day/hour limits that parallelism. On heavily loaded S3 buckets, the large number of list API calls needed becomes both slow and expensive, and this listing step dominates the total vacuum time as tables grow into hundreds of thousands of objects. _Developers troubleshooting slow storage maintenance jobs can follow data engineering fixes like this on daily.dev._ ### How much can inventory-based vacuum reduce Delta Lake maintenance costs? One large table's vacuum runtime dropped from 30-60 minutes to under 5 minutes after switching to inventory-based vacuum, and listing files via inventory reports costs about 44% less than using S3 List API calls for a million objects. This also enabled vacuuming multiple tables in a single daily job instead of one job per table every 6 hours, saving roughly $30-40 per day in compute and EBS costs on one table alone. _Teams weighing lakehouse cost optimizations can compare real savings data like this on daily.dev._ ## Similar posts on daily.dev - [Optimizing our Datalake's Raw Data Layer by Smart Storage](https://daily.dev/posts/optimizing-our-datalake-s-raw-data-layer-by-smart-storage-wlnbqtoz5) · Halodoc · 2 upvotes · 0 comments - [650GB of Data \(Delta Lake on S3\). Polars vs DuckDB vs Daft vs Spark.](https://daily.dev/posts/650gb-of-data-delta-lake-on-s3-polars-vs-duckdb-vs-daft-vs-spark--bbkmmear9) · Hacker News · 1 upvotes · 0 comments - [How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%](https://daily.dev/posts/how-partition-access-visualizations-reduced-our-data-lake-s3-cost-by-33--dnz5wsm95) · Yelp Engineering · 0 upvotes · 0 comments - [Expensive Delta Lake S3 Storage Mistakes \(And How to Fix Them\)](https://daily.dev/posts/expensive-delta-lake-s3-storage-mistakes-and-how-to-fix-them--iyqaxbwgz) · databricks · 3 upvotes · 0 comments - [Enhancing Data Efficiency with Snowflake Storage Lifecycle Management](https://daily.dev/posts/enhancing-data-efficiency-with-snowflake-storage-lifecycle-management-qutldtwaq) · Faun · 0 upvotes · 0 comments --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark), [#aws-s3](https://daily.dev/tags/aws-s3), [#finops](https://daily.dev/tags/finops) [View this post on daily.dev](https://daily.dev/posts/efficient-delta-vacuum-with-file-inventory-qwa46ea1n) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Efficient Delta Vacuum with File Inventory","url":"https://daily.dev/posts/efficient-delta-vacuum-with-file-inventory-qwa46ea1n","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/efficient-delta-vacuum-with-file-inventory-qwa46ea1n"},"datePublished":"2026-08-16T08:53:44.027Z","dateModified":"2026-08-16T08:57:02.777Z","description":"Grab's data engineering team describes how they cut Delta Lake vacuum times on large partitioned tables by switching from recursive file listing to an...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/99a934f013a939ed9d29b82103280d1c?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/99a934f013a939ed9d29b82103280d1c?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Delta Lake Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Delta Lake Blog","logo":"https://media.daily.dev/image/upload/s--7f2q9Asf--/f_auto,q_auto/v1786870393/logos/delta-lake?_a=BAMAMicg0","url":"https://daily.dev/sources/delta-lake"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/efficient-delta-vacuum-with-file-inventory-qwa46ea1n","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,apache-spark,aws-s3,finops","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Delta Lake Blog","item":"https://daily.dev/sources/delta-lake"},{"@type":"ListItem","position":3,"name":"Efficient Delta Vacuum with File Inventory"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/efficient-delta-vacuum-with-file-inventory-qwa46ea1n#faq","mainEntity":[{"@type":"Question","name":"What is inventory-based vacuum in Delta Lake and which version introduced it?","acceptedAnswer":{"@type":"Answer","text":"Inventory-based vacuum is a Delta Lake feature, released in Delta 3.2, that lets vacuum use a precomputed file inventory (from an object store inventory report or a Spark SQL query) instead of performing a recursive file listing. It requires a schema with path, length, isDir, and modificationTime columns, and is invoked with the USING INVENTORY clause, turning vacuum into a two-stage compare-and-delete process. Track new Delta Lake releases like this one on daily.dev before they hit your production pipelines."}},{"@type":"Question","name":"Why does Delta Lake vacuum take so long on large partitioned tables in S3?","acceptedAnswer":{"@type":"Answer","text":"Vacuum's recursive file listing is the bottleneck, since parallelism depends on the cardinality of the top directory levels and partitioning by year/month/day/hour limits that parallelism. On heavily loaded S3 buckets, the large number of list API calls needed becomes both slow and expensive, and this listing step dominates the total vacuum time as tables grow into hundreds of thousands of objects. Developers troubleshooting slow storage maintenance jobs can follow data engineering fixes like this on daily.dev."}},{"@type":"Question","name":"How much can inventory-based vacuum reduce Delta Lake maintenance costs?","acceptedAnswer":{"@type":"Answer","text":"One large table's vacuum runtime dropped from 30-60 minutes to under 5 minutes after switching to inventory-based vacuum, and listing files via inventory reports costs about 44% less than using S3 List API calls for a million objects. This also enabled vacuuming multiple tables in a single daily job instead of one job per table every 6 hours, saving roughly $30-40 per day in compute and EBS costs on one table alone. Teams weighing lakehouse cost optimizations can compare real savings data like this on daily.dev."}}]} ```