--- title: Best ETL posts of 2024 | daily.dev description: The most upvoted ETL posts of 2024, curated by the daily.dev community. canonical: https://daily.dev/tags/etl/best-of/2024 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/tags/etl/best-of/2024 og:type: website og:site_name: daily.dev og:title: Best ETL posts of 2024 | daily.dev og:description: The most upvoted ETL posts of 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of ETL — 2024 1. 1 [](/posts/python-based-etl-95lcukt4k "Python-based ETL") Article Hacker News · 2y Python-based ETL Amphi is a Python-based ETL tool designed for efficient data extraction, transformation, and loading with a low-code approach. It features a graphical user interface for designing data pipelines, generating deployable native Python code, and supporting various data formats like CSV and JSON. Amphi ensures flexibility, ease of sharing pipeline definitions, and guarantees data privacy as processing is done locally. The platform is aimed at fostering community collaboration among data practitioners of all levels. 117 4 2. 2 [](/posts/10-built-in-python-modules-every-data-engineer-should-know-hlqhj7vgl "10 Built-In Python Modules Every Data Engineer Should Know") Article KDnuggets · 2y 10 Built-In Python Modules Every Data Engineer Should Know Python's standard library includes built-in modules that are essential for data engineering tasks. Key modules like os, pathlib, shutil, csv, json, pickle, sqlite3, datetime, re, and subprocess enable efficient file and directory management, data handling and serialization, database interaction, text processing, and more. Utilizing these modules can streamline your data engineering workflows, providing essential functionality without relying on external libraries. 110 3. 3 [](/posts/building-a-serverless-data-pipeline-a-step-by-step-guide-apjwxpd51 "Building a Serverless Data Pipeline: A Step-by-Step Guide") Article Towards Dev · 2y Building a Serverless Data Pipeline: A Step-by-Step Guide The guide provides step-by-step instructions to build a serverless data pipeline using AWS services. Key components include AWS Lambda for data extraction from the Colombo Stock Market Index API, Amazon Kinesis Data Firehose for data ingestion, Amazon S3 for storage, and AWS Glue for ETL orchestration with Athena for querying data. The pipeline uses event-driven architectures with SQS notifications and Glue crawlers for efficient data processing. 81 1 4. 4 [](/posts/the-great-database-migration-g3j8d9szh "The Great Database Migration") Article Hacker News · 2y The Great Database Migration Shepherd successfully migrated its pricing engine database from SQLite to Postgres with zero downtime. The new architecture improves scalability, performance, and developer experience. The migration included converting synchronous functions to asynchronous, leveraging a serverless architecture with Neon, and automating ETL processes. The project highlighted performance optimizations, including caching strategies and connection pooling, resulting in significantly improved response times. 56 5. 5 [](/posts/top-data-engineering-courses-in-2024-xkkj0amci "Top Data Engineering Courses in 2024") Article Machine Learning News · 2y Top Data Engineering Courses in 2024 Data engineering is crucial for organizations relying on data-driven insights. This post lists top courses for mastering data engineering skills such as building scalable data solutions, ETL processes, and leveraging technologies like Apache Spark and cloud platforms. Courses include IBM’s Data Engineering Foundations, Meta Database Engineer Professional Certificate, and Google Cloud Database Engineer Specialization, among others. 46 6. 6 [](/posts/etl-and-elt-eto5kaieg "ETL and ELT") Article Data Engineer Things · 2y ETL and ELT The author reflects on their journey from chasing the latest data engineering tools to focusing on foundational concepts, emphasizing the shift from ETL (Extract, Transform, Load) to ELT (Extract, Load, Transform). The traditional ETL process, necessitated by the high costs and limitations of early data warehouses, is contrasted with the modern ELT approach, facilitated by advancements in cloud data warehousing. ELT offers greater flexibility and efficiency by loading raw data into the warehouse and handling transformations within the warehouse, aligning better with agile development practices. 45 4 7. 7 [](/posts/stripe-data-vs-open-source-alternatives-a-mrr-example-tdhhhocr9 "Stripe Data vs Open‐Source Alternatives: a MRR example") Article Hacker News · 2y Stripe Data vs Open‐Source Alternatives: a MRR example Stripe's API lacks straightforward methods for calculating MRR and necessitates the use of additional costly tools like Stripe Sigma and Stripe Data Pipeline. These tools are ideal for large companies with substantial transactions but it's impractical for smaller transactions due to high costs. Open-source alternatives, such as Lago, provide more flexibility and control over financial data, avoiding dependence on expensive third-party solutions. 31 8. 8 [](/posts/data-pipelines-and-scds-wct0vovp9 "Data pipelines and SCDs") Article Substack · 2y Data pipelines and SCDs Designing backfillable data pipelines using idempotent transformation code avoids the complications of ad-hoc SQL. When handling Slowly Changing Dimensions (SCDs), SCD Type 2 is preferred for its immutability and compressive qualities, though it involves complex surrogate key lookups. Alternatively, snapshot tables offer a simpler, reproducible model at the cost of higher data replication, making them ideal in cloud environments where storage is cheaper than engineering time. 27 9. 9 [](/posts/data-modeling-techniques-for-data-warehouse-ngrkadi92 "Data Modeling Techniques For Data Warehouse") Article Towards Data Science · 2y Data Modeling Techniques For Data Warehouse Data modeling is a key process in creating conceptual representations of organizational data and its relationships. Focusing on various methodologies like Kimball's, Inmon's, and Data Vault, this guide provides insights into dimensional modeling, including benefits like simplicity, improved query performance, and scalability. It also covers different schema types (star and snowflake), and strategies for data loading. Special attention is given to innovative approaches like using one big table (OBT) for modern data warehouses. 19 10. 10 [](/posts/golang-weekly-issue-514-july-9-2024-m237kytvw "Golang Weekly Issue 514: July 9, 2024") Article Golang Weekly · 2y Golang Weekly Issue 514: July 9, 2024 Learn how to locally patch dependencies in Go using \`go mod\` and discover tools for building scalable Go applications on AWS. This issue also covers the release of Ergo 2.14, a modern IRC server, and highlights CoreDNS for DNS and service discovery, as well as OmniParser for ETL tasks. Additional updates include releases like gocron 2.8, Go Micro 5.3, River 0.9, and go-arg 1.5. 17 11. 11 [](/posts/performant-dbt-pipelines-with-motherduck-3zxfr5dam "Performant dbt pipelines with MotherDuck") Article MotherDuck · 2y Performant dbt pipelines with MotherDuck This post recaps learnings from the dbt+MotherDuck workshop and delves into building performant data pipelines using DuckDB and MotherDuck. Key steps include utilizing the read\_blob() function, leveraging pre\_hooks and variables in DuckDB, implementing incremental models with read\_csv(), and handling data de-duplication using unnest() and arg\_max(). These techniques aim to optimize data workflows and enhance data transformation and analysis efficiency. 16 1 12. 12 [](/posts/it-s-time-to-merge-analytics-and-data-engineering-again--itzhzbuty "It's Time to Merge Analytics and Data Engineering (Again)") Article Materialized View · 2y It's Time to Merge Analytics and Data Engineering (Again) The post argues for merging analytics and data engineering roles, citing the commoditization of data pipelines and the limited value provided by distinct analytics engineers. With advancements like LLMs, data integration tools, and data pipeline vendors, there's a push for a consolidated data team handling extraction, transformation, and loading (ETL) processes. The author notes emerging tools that facilitate this transition and predicts a convergence of these roles in the coming years. 14 13. 13 [](/posts/end-to-end-etl-and-sales-dashboard-project-in-microsoft-fabric-jqys36uxs "End-to-End ETL and Sales Dashboard Project in Microsoft Fabric") Article Data Engineer Things · 2y End-to-End ETL and Sales Dashboard Project in Microsoft Fabric A step-by-step guide on creating a sales dashboard using Microsoft Fabric and PowerBI Desktop for the WideWorldImporters sample database. Key goals include creating a dynamic and user-friendly interface for monitoring sales performance across various dimensions like customer, product, and region. The post covers data gathering, ETL processes, creating views and tables, setting up a semantic model, and building various visuals to support different user stories. 13 14. 14 [](/posts/reverse-etl-in-healthcare--datasciencecentral-com-ylbj7o5hl "Reverse ETL in Healthcare- DataScienceCentral.com") Article Data Science Central · 2y Reverse ETL in Healthcare- DataScienceCentral.com Managing patient data is a significant challenge in healthcare. Reverse ETL is a data integration method that ensures the smooth flow of data from data warehouses to operational systems like CRMs and ERPs. This real-time data synchronization improves patient care, enhances decision-making, maintains data consistency, ensures regulatory compliance, and enhances operational efficiency. Key components for successful implementation include a centralized data warehouse, robust ETL tools, seamless integration with operational systems, stringent data governance measures, and proper training for healthcare staff. 13 15. 15 [](/posts/fundamentals-of-data-engineering-masterclass-eekpecujz "Fundamentals Of Data Engineering Masterclass") Video YouTube · 2y Fundamentals Of Data Engineering Masterclass This Data Engineering masterclass covers the fundamentals of Data Engineering, including the life cycle, data generation, storage, database management, data modeling, and the distinction between SQL and NoSQL. It delves into data processing systems like OLTP and OLAP, ETL processes, and building data architecture from scratch. The session also explores data warehousing, dimensional modeling, data marts, data lakes, big data, cloud services (AWS, GCP, Azure), and key tools for data engineering such as Python, SQL, Apache Spark, Databricks, Apache Airflow, and Apache Kafka. Real-world architecture case studies on AWS and GCP are discussed as well. 13 16. 16 [](/posts/aws-glue-visual-etl-for-your-data-in-mongodb-atlas-hsq32dju3 "AWS Glue Visual ETL for Your Data in MongoDB Atlas") Article MongoDB\_Official · 2y AWS Glue Visual ETL for Your Data in MongoDB Atlas Learn how to use AWS Glue's visual ETL capabilities to transfer data between MongoDB Atlas and AWS S3\. AWS Glue Studio allows developers to create ETL pipelines without needing knowledge of Spark or SQL, facilitating seamless data transformation and integration with other AWS services. AWS S3 is utilized for scalable, durable, and cost-effective data storage, making it suitable for data lakes, warehousing, machine learning, media streaming, backup, and web hosting. 10 17. 17 [](/posts/accelerate-feature-engineering-with-photon-rctd2fdii "Accelerate Feature Engineering With Photon") Article databricks · 2y Accelerate Feature Engineering With Photon Training high-quality machine learning models involves careful data preparation, which can be time-consuming for large datasets. The Photon Engine, now available in Databricks Machine Learning Runtime, significantly speeds up Spark SQL and Spark DataFrame workloads, achieving speed improvements of 2x-4x. The Photon Engine enhances ETL processes and feature engineering, especially for time series data, using a new point-in-time join implementation. Users can enable Photon in Databricks ML Runtime 15.2 and above for better query performance. 10 18. 18 [](/posts/how-to-pivot-tables-in-sql-jey2i90lg "How to Pivot Tables in SQL") Article Towards Data Science · 2y How to Pivot Tables in SQL A comprehensive guide on creating pivot tables in SQL for enhanced data analysis using DECODE() and PIVOT() functions. 10 [See all ETL archives](/tags/etl/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/tags/etl/best-of/2024#page","url":"https://daily.dev/tags/etl/best-of/2024","name":"Best ETL Posts of 2024","description":"The most upvoted ETL posts of 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/tags/etl/best-of/2024#items","numberOfItems":18,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/python-based-etl-95lcukt4k","name":"Python-based ETL"},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/10-built-in-python-modules-every-data-engineer-should-know-hlqhj7vgl","name":"10 Built-In Python Modules Every Data Engineer Should Know"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/building-a-serverless-data-pipeline-a-step-by-step-guide-apjwxpd51","name":"Building a Serverless Data Pipeline: A Step-by-Step Guide"},{"@type":"ListItem","position":4,"url":"https://daily.dev/posts/the-great-database-migration-g3j8d9szh","name":"The Great Database Migration"},{"@type":"ListItem","position":5,"url":"https://daily.dev/posts/top-data-engineering-courses-in-2024-xkkj0amci","name":"Top Data Engineering Courses in 2024"},{"@type":"ListItem","position":6,"url":"https://daily.dev/posts/etl-and-elt-eto5kaieg","name":"ETL and ELT"},{"@type":"ListItem","position":7,"url":"https://daily.dev/posts/stripe-data-vs-open-source-alternatives-a-mrr-example-tdhhhocr9","name":"Stripe Data vs Open‐Source Alternatives: a MRR example"},{"@type":"ListItem","position":8,"url":"https://daily.dev/posts/data-pipelines-and-scds-wct0vovp9","name":"Data pipelines and SCDs"},{"@type":"ListItem","position":9,"url":"https://daily.dev/posts/data-modeling-techniques-for-data-warehouse-ngrkadi92","name":"Data Modeling Techniques For Data Warehouse"},{"@type":"ListItem","position":10,"url":"https://daily.dev/posts/golang-weekly-issue-514-july-9-2024-m237kytvw","name":"Golang Weekly Issue 514: July 9, 2024"},{"@type":"ListItem","position":11,"url":"https://daily.dev/posts/performant-dbt-pipelines-with-motherduck-3zxfr5dam","name":"Performant dbt pipelines with MotherDuck"},{"@type":"ListItem","position":12,"url":"https://daily.dev/posts/it-s-time-to-merge-analytics-and-data-engineering-again--itzhzbuty","name":"It's Time to Merge Analytics and Data Engineering (Again)"},{"@type":"ListItem","position":13,"url":"https://daily.dev/posts/end-to-end-etl-and-sales-dashboard-project-in-microsoft-fabric-jqys36uxs","name":"End-to-End ETL and Sales Dashboard Project in Microsoft Fabric"},{"@type":"ListItem","position":14,"url":"https://daily.dev/posts/reverse-etl-in-healthcare--datasciencecentral-com-ylbj7o5hl","name":"Reverse ETL in Healthcare- DataScienceCentral.com"},{"@type":"ListItem","position":15,"url":"https://daily.dev/posts/fundamentals-of-data-engineering-masterclass-eekpecujz","name":"Fundamentals Of Data Engineering Masterclass"},{"@type":"ListItem","position":16,"url":"https://daily.dev/posts/aws-glue-visual-etl-for-your-data-in-mongodb-atlas-hsq32dju3","name":"AWS Glue Visual ETL for Your Data in MongoDB Atlas"},{"@type":"ListItem","position":17,"url":"https://daily.dev/posts/accelerate-feature-engineering-with-photon-rctd2fdii","name":"Accelerate Feature Engineering With Photon"},{"@type":"ListItem","position":18,"url":"https://daily.dev/posts/how-to-pivot-tables-in-sql-jey2i90lg","name":"How to Pivot Tables in SQL"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Tags","item":"https://daily.dev/tags"},{"@type":"ListItem","position":3,"name":"ETL","item":"https://daily.dev/tags/etl"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/tags/etl/best-of"},{"@type":"ListItem","position":5,"name":"2024"}]}]} ```