--- title: Building a more efficient data infrastructure for... description: Explains how to combine Delta Lake, Amazon SageMaker Studio, and Amazon EMR to build an end-to-end data infrastructure for machine learning workflows. Covers... canonical: https://daily.dev/posts/building-a-more-efficient-data-infrastructure-for-machine-learning-with-open-source-using-delta-lake-phdxn9dlh twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Building a more efficient data infrastructure for machine learning with Open Source using Delta Lake, Amazon SageMaker, and EMR | daily.dev og:description: Explains how to combine Delta Lake, Amazon SageMaker Studio, and Amazon EMR to build an end-to-end data infrastructure for machine learning workflows. Covers... og:url: https://daily.dev/posts/building-a-more-efficient-data-infrastructure-for-machine-learning-with-open-source-using-delta-lake-phdxn9dlh og:image: https://api.daily.dev/og/posts/PhdxN9DLh.png og:image:alt: Building a more efficient data infrastructure for machine learning with Open Source using Delta Lake, Amazon SageMaker, and EMR og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Building a more efficient data infrastructure for machine learning with Open Source using Delta Lake, Amazon SageMaker, and EMR **[Delta Lake Blog](https://daily.dev/sources/delta-lake)** · 8 min read · 0 upvotes · 0 comments ## Summary Explains how to combine Delta Lake, Amazon SageMaker Studio, and Amazon EMR to build an end-to-end data infrastructure for machine learning workflows. Covers reading and writing Delta Lake tables from SageMaker Studio notebooks in single-node mode for prototyping, and cluster mode using EMR-backed Spark for distributed processing via the delta-spark PyPI library, SparkMagic, and Livy. Highlights benefits like ACID transactions, scalable metadata, Parquet-based storage on S3, and time travel for reproducible ML data pipelines. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I read Delta Lake tables from Amazon SageMaker Studio? Use the delta-spark PyPI library, which provides Python APIs for reading and writing Delta Lake tables with Apache Spark inside a SageMaker Studio JupyterLab notebook. You can run Spark commands locally on the studio instance for prototyping on smaller datasets, or connect to a distributed Amazon EMR cluster for larger workloads once the delta-spark library is installed there. _daily.dev surfaces practical guides for teams wiring Delta Lake into their SageMaker ML pipelines._ ### What is the difference between single-node mode and cluster mode when using SageMaker Studio with Delta Lake? Single-node mode runs Spark locally on the SageMaker Studio instance, ideal for prototyping on smaller datasets to save cost. Cluster mode connects SageMaker Studio to a distributed Amazon EMR cluster running Spark, using tools like SparkMagic and Livy or PyHive, for scaling to larger Delta Lake workloads after prototyping is complete. _Compare data pipeline scaling options like this on daily.dev before choosing your ML infrastructure setup._ ### Why use Delta Lake instead of raw Parquet files for machine learning data pipelines? Delta Lake adds ACID transactions, scalable metadata handling, and unified streaming/batch processing on top of the open-source Parquet file format, plus a transaction log with granular per-file metadata that speeds up metadata searches compared to a plain files-in-object-store approach. It also supports time travel to view or revert to previous data versions. _daily.dev helps engineers weighing storage layer choices like Delta Lake for lakehouse architectures._ ## Similar posts on daily.dev - [Amazon SageMaker Unified Studio Notebooks now support EMR Serverless](https://daily.dev/posts/amazon-sagemaker-unified-studio-notebooks-now-support-emr-serverless-1izj5pt53) · AWS · 0 upvotes · 0 comments - [Delta Lake Explained: Boost Data Reliability in Cloud Storage](https://daily.dev/posts/delta-lake-explained-boost-data-reliability-in-cloud-storage-oechbmvm4) · databricks · 0 upvotes · 0 comments - [Bootstrap a Data Lakehouse in an Afternoon](https://daily.dev/posts/bootstrap-a-data-lakehouse-in-an-afternoon-co4kw9dyi) · Towards Data Science · 0 upvotes · 0 comments - [EverC: Regaining Control of Data Platform Costs on AWS with EMR on EKS](https://daily.dev/posts/everc-regaining-control-of-data-platform-costs-on-aws-with-emr-on-eks-x1fldlsgc) · BigData Boutique blog · 1 upvotes · 0 comments - [\[AWS-AI\] Amazon SageMaker Explained: From Data Preparation to Model Deployment](https://daily.dev/posts/aws-ai-amazon-sagemaker-explained-from-data-preparation-to-model-deployment-izl4ihjhw) · Medium · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#aws](https://daily.dev/tags/aws), [#apache-spark](https://daily.dev/tags/apache-spark) [View this post on daily.dev](https://daily.dev/posts/building-a-more-efficient-data-infrastructure-for-machine-learning-with-open-source-using-delta-lake-phdxn9dlh) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Building a more efficient data infrastructure for machine learning with Open Source using Delta Lake, Amazon SageMaker, and EMR","url":"https://daily.dev/posts/building-a-more-efficient-data-infrastructure-for-machine-learning-with-open-source-using-delta-lake-phdxn9dlh","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/building-a-more-efficient-data-infrastructure-for-machine-learning-with-open-source-using-delta-lake-phdxn9dlh"},"datePublished":"2026-08-16T08:53:36.408Z","dateModified":"2026-08-16T08:55:44.778Z","description":"Explains how to combine Delta Lake, Amazon SageMaker Studio, and Amazon EMR to build an end-to-end data infrastructure for machine learning workflows. Covers...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/6ce3561f74ff5314e593610e1ee4ce1b?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/6ce3561f74ff5314e593610e1ee4ce1b?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Delta Lake Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Delta Lake Blog","logo":"https://media.daily.dev/image/upload/s--7f2q9Asf--/f_auto,q_auto/v1786870393/logos/delta-lake?_a=BAMAMicg0","url":"https://daily.dev/sources/delta-lake"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/building-a-more-efficient-data-infrastructure-for-machine-learning-with-open-source-using-delta-lake-phdxn9dlh","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,aws,apache-spark","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Delta Lake Blog","item":"https://daily.dev/sources/delta-lake"},{"@type":"ListItem","position":3,"name":"Building a more efficient data infrastructure for machine learning with Open Source using Delta Lake, Amazon SageMaker, and EMR"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/building-a-more-efficient-data-infrastructure-for-machine-learning-with-open-source-using-delta-lake-phdxn9dlh#faq","mainEntity":[{"@type":"Question","name":"How can I read Delta Lake tables from Amazon SageMaker Studio?","acceptedAnswer":{"@type":"Answer","text":"Use the delta-spark PyPI library, which provides Python APIs for reading and writing Delta Lake tables with Apache Spark inside a SageMaker Studio JupyterLab notebook. You can run Spark commands locally on the studio instance for prototyping on smaller datasets, or connect to a distributed Amazon EMR cluster for larger workloads once the delta-spark library is installed there. daily.dev surfaces practical guides for teams wiring Delta Lake into their SageMaker ML pipelines."}},{"@type":"Question","name":"What is the difference between single-node mode and cluster mode when using SageMaker Studio with Delta Lake?","acceptedAnswer":{"@type":"Answer","text":"Single-node mode runs Spark locally on the SageMaker Studio instance, ideal for prototyping on smaller datasets to save cost. Cluster mode connects SageMaker Studio to a distributed Amazon EMR cluster running Spark, using tools like SparkMagic and Livy or PyHive, for scaling to larger Delta Lake workloads after prototyping is complete. Compare data pipeline scaling options like this on daily.dev before choosing your ML infrastructure setup."}},{"@type":"Question","name":"Why use Delta Lake instead of raw Parquet files for machine learning data pipelines?","acceptedAnswer":{"@type":"Answer","text":"Delta Lake adds ACID transactions, scalable metadata handling, and unified streaming/batch processing on top of the open-source Parquet file format, plus a transaction log with granular per-file metadata that speeds up metadata searches compared to a plain files-in-object-store approach. It also supports time travel to view or revert to previous data versions. daily.dev helps engineers weighing storage layer choices like Delta Lake for lakehouse architectures."}}]} ```