--- title: Best Data Engineering posts of 2024 | daily.dev description: The most upvoted Data Engineering posts of 2024, curated by the daily.dev community. canonical: https://daily.dev/tags/data-engineering/best-of/2024 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/tags/data-engineering/best-of/2024 og:type: website og:site_name: daily.dev og:title: Best Data Engineering posts of 2024 | daily.dev og:description: The most upvoted Data Engineering posts of 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Data Engineering — 2024 1. 1 [](/posts/how-did-linkedin-handle-7-trillion-messages-daily-with-apache-kafka--jf20b1xpw "How Did LinkedIn Handle 7 Trillion Messages Daily With Apache Kafka?") Article Medium · 2y How Did LinkedIn Handle 7 Trillion Messages Daily With Apache Kafka? LinkedIn uses Apache Kafka to manage and process up to 7 trillion messages daily. They achieve reliability and scalability through a multi-tiered Kafka deployment across multiple data centers, leveraging local and aggregate clusters. LinkedIn ensures message completeness with an internal auditing tool that tracks sent and consumed messages. They maintain a close relationship with the open-source Kafka community by regularly integrating features and patches from their internal branches into the upstream Kafka branch. 175 4 2. 2 [](/posts/roadmap-to-learn-data-engineering-how-i-would-start-again-tq6trnc6y "Roadmap to Learn Data Engineering: How I Would Start Again") Article Medium · 2y Roadmap to Learn Data Engineering: How I Would Start Again A roadmap for learning data engineering, covering Python, SQL, command line, data warehouse, data modeling, data storage, data processing, data transformation, data orchestration, advanced topics, and staying updated. 159 6 3. 3 [](/posts/how-to-choose-the-right-type-of-database-4olcmsr9t "How to choose the right type of database") Article Tinybird · 3y How to choose the right type of database Understanding the different types of databases, factors to consider when choosing a database, and the implications of the CAP theorem on database selection. 159 12 4. 4 [](/posts/5-tips-for-improving-sql-query-performance-mkht1ssaz "5 Tips for Improving SQL Query Performance") Article KDnuggets · 2y 5 Tips for Improving SQL Query Performance Strong SQL skills are crucial in data roles, where optimizing query performance can significantly impact application efficiency. Key tips include avoiding SELECT \* by specifying columns, using GROUP BY instead of SELECT DISTINCT, limiting query results, and employing indexes with caution. Balancing these techniques can improve query performance and ensure efficient database operations. 149 3 5. 5 [](/posts/the-state-of-data-engineering-2024-t1ywu2dho "The State of Data Engineering 2024") Article Community Picks · 2y The State of Data Engineering 2024 The 2024 State of Data Engineering report discusses the influence of GenAI on software infrastructure, the expansion of product offerings due to the economic downturn, and the impact of open table formats and their catalogs in the data lake industry. It also highlights the importance of data version control and observability in AI/ML systems. 144 3 6. 6 [](/posts/building-a-python-based-data-lake-bu41iyreu "Building a Python-Based Data Lake") Article Dev Genius · 2y Building a Python-Based Data Lake Data lakes are vital for modern data ecosystems, allowing organizations to store and analyze large volumes of varied data without requiring a predefined schema. This guide details setting up a Python-based data lake using MinIO, PyIceberg, PyArrow, and Postgres, ideal for small to medium setups due to its simplicity. The step-by-step instructions cover installation of libraries, configuring SQL catalogs, data transformation using Pandas and PyArrow, and querying data. Advanced operations using DuckDB are also explored, showcasing robust data handling with flexibility and scalability. 134 3 7. 7 [](/posts/10-built-in-python-modules-every-data-engineer-should-know-hlqhj7vgl "10 Built-In Python Modules Every Data Engineer Should Know") Article KDnuggets · 2y 10 Built-In Python Modules Every Data Engineer Should Know Python's standard library includes built-in modules that are essential for data engineering tasks. Key modules like os, pathlib, shutil, csv, json, pickle, sqlite3, datetime, re, and subprocess enable efficient file and directory management, data handling and serialization, database interaction, text processing, and more. Utilizing these modules can streamline your data engineering workflows, providing essential functionality without relying on external libraries. 110 8. 8 [](/posts/10-github-repositories-to-master-data-engineering-x2qqtjlwq "10 GitHub Repositories to Master Data Engineering") Article KDnuggets · 2y 10 GitHub Repositories to Master Data Engineering Learn data engineering through free courses, tutorials, books, tools, guides, roadmaps, practice exercises, projects, and other resources. 106 4 9. 9 [](/posts/building-a-serverless-data-pipeline-a-step-by-step-guide-apjwxpd51 "Building a Serverless Data Pipeline: A Step-by-Step Guide") Article Towards Dev · 2y Building a Serverless Data Pipeline: A Step-by-Step Guide The guide provides step-by-step instructions to build a serverless data pipeline using AWS services. Key components include AWS Lambda for data extraction from the Colombo Stock Market Index API, Amazon Kinesis Data Firehose for data ingestion, Amazon S3 for storage, and AWS Glue for ETL orchestration with Athena for querying data. The pipeline uses event-driven architectures with SQS notifications and Glue crawlers for efficient data processing. 81 1 10. 10 [](/posts/what-is-ai-engineering--veb6hnmzc "What is AI Engineering?") Article SwirlAI · 2y What is AI Engineering? AI Engineering is a rapidly evolving role focused on developing and deploying AI systems that utilize Large Language Models (LLMs) to solve business problems. AI Engineers differ from Software Engineers and Machine Learning Engineers in that they deal extensively with non-deterministic systems and require skills in prompt engineering, infrastructure, and data integration. The field is witnessing the rise of Agentic systems, which are advanced AI systems capable of performing complex tasks with a degree of autonomy. AI Engineering is poised to become one of the most in-demand roles in the tech industry with high salaries and growing opportunities. 78 2 11. 11 [](/posts/project-ideas-to-master-data-engineering-oupn9e6gc "Project Ideas to Master Data Engineering") Article KDnuggets · 2y Project Ideas to Master Data Engineering To effectively learn data engineering, working on projects is essential. Key skills to focus on include data transformation, data visualization, building data pipelines, and implementing data storage solutions like data lakes and data warehouses. The post suggests six project ideas to cover these aspects: building an end-to-end data pipeline, transforming data sets, implementing a data lake, creating a data warehouse, processing real-time data, and visualizing data with dashboards. 68 12. 12 [](/posts/meet-megaparse-an-open-source-ai-tool-for-parsing-various-types-of-documents-for-llm-ingestion-1ua70xvbi "Meet MegaParse: An Open-Source AI Tool for Parsing Various Types of Documents for LLM Ingestion") Article Machine Learning News · 2y Meet MegaParse: An Open-Source AI Tool for Parsing Various Types of Documents for LLM Ingestion MegaParse is an open-source tool designed to efficiently parse various types of documents (PDF, Word, Excel, CSV, etc.) for ingestion into large language models (LLMs). It saves users significant time and effort by automating the conversion process while retaining information integrity. The tool is highly versatile, handling different document elements such as tables and images, and supports customizable output formats. Installation is straightforward via pip, with additional setups for dependencies like Poppler, Tesseract, and libmagic. MegaParse also provides advanced usage options and benchmarking capabilities, making it a reliable choice for developers and enterprises looking to streamline their AI data pipeline. 56 5 13. 13 [](/posts/trillions-of-indexes-how-uber-s-ledgerstore-supports-such-massive-scale-fi9o1kmth "Trillions of Indexes: How Uber’s LedgerStore Supports Such Massive Scale") Article ByteByteGo · 2y Trillions of Indexes: How Uber’s LedgerStore Supports Such Massive Scale Uber's LedgerStore is a custom-built solution to manage trillions of financial transaction records efficiently. It ensures data immutability and supports various types of indexes including strongly consistent, eventually consistent, and time-range indexes. The migration from DynamoDB to LedgerStore for Uber's payment data was driven by the need for cost savings, simplified architecture, improved performance, and tailored features for financial data management. This transition involved handling 1.2 PB of compressed data with zero data inconsistencies detected over six months. 55 3 14. 14 [](/posts/7-python-libraries-every-data-engineer-should-know-3klrdr7pf "7 Python Libraries Every Data Engineer Should Know") Article KDnuggets · 2y 7 Python Libraries Every Data Engineer Should Know Discover some essential Python libraries for data engineers, including Requests for API data extraction, BeautifulSoup for web scraping, Pandas for data manipulation, SQLAlchemy for database work, Airflow for workflow orchestration, PySpark for big data processing, and Kafka-Python for real-time data processing. 52 15. 15 [](/posts/the-most-important-soft-skill-in-tech-ynbmflylz "The Most Important Soft Skill in Tech") Article Medium · 3y The Most Important Soft Skill in Tech Interviewing is the most important soft skill in tech. Building rapport, demonstrating value, and projecting professional growth are key objectives in an interview. 52 16. 16 [](/posts/top-5-data-engineering-projects-for-beginners-2024-o1lrocakd "Top 5 Data Engineering Projects for Beginners 2024") Article Community Picks · 2y Top 5 Data Engineering Projects for Beginners 2024 Get hands-on experience with five beginner-friendly data engineering projects that enhance skills in data processing, analytics, and visualization using tools like Apache Kafka, Python, and cloud platforms. These projects prepare you for real-world challenges and make your resume stand out. 48 17. 17 [](/posts/omniparse-an-ai-platform-that-ingests-parses-any-unstructured-data-into-structured-actionable-data-kp1jrtlkj "OmniParse: An AI Platform that Ingests/Parses Any Unstructured Data into Structured, Actionable Data Optimized for GenAI (LLM) Applications") Article Machine Learning News · 2y OmniParse: An AI Platform that Ingests/Parses Any Unstructured Data into Structured, Actionable Data Optimized for GenAI (LLM) Applications OmniParse is an AI platform designed to convert various unstructured data types, including documents, images, audio, video, and web content, into structured, actionable data. It supports around 20 different file types and operates entirely locally, ensuring data privacy. OmniParse deploys easily using Docker and Skypilot and works with platforms like Colab. It uses advanced models such as Surya OCR and Whisper, achieving high accuracy and efficiency in data conversion, optimizing it for Generative AI applications. 48 18. 18 [](/posts/5-free-online-courses-to-learn-data-engineering-fundamentals-0bew3r3f1 "5 Free Online Courses to Learn Data Engineering Fundamentals") Article KDnuggets · 2y 5 Free Online Courses to Learn Data Engineering Fundamentals Explore five free online courses designed to teach the fundamentals of data engineering. These courses range from beginner-friendly introductions to comprehensive professional certificates. Key areas covered include data pipelines, databases, Python and Pandas, cloud computing, and big data tools like Hadoop and Apache Spark. 46 1 19. 19 [](/posts/top-data-engineering-courses-in-2024-xkkj0amci "Top Data Engineering Courses in 2024") Article Machine Learning News · 2y Top Data Engineering Courses in 2024 Data engineering is crucial for organizations relying on data-driven insights. This post lists top courses for mastering data engineering skills such as building scalable data solutions, ETL processes, and leveraging technologies like Apache Spark and cloud platforms. Courses include IBM’s Data Engineering Foundations, Meta Database Engineer Professional Certificate, and Google Cloud Database Engineer Specialization, among others. 46 20. 20 [](/posts/etl-and-elt-eto5kaieg "ETL and ELT") Article Data Engineer Things · 2y ETL and ELT The author reflects on their journey from chasing the latest data engineering tools to focusing on foundational concepts, emphasizing the shift from ETL (Extract, Transform, Load) to ELT (Extract, Load, Transform). The traditional ETL process, necessitated by the high costs and limitations of early data warehouses, is contrasted with the modern ELT approach, facilitated by advancements in cloud data warehousing. ELT offers greater flexibility and efficiency by loading raw data into the warehouse and handling transformations within the warehouse, aligning better with agile development practices. 45 4 21. 21 [](/posts/poc-data-platform-project-utilizing-modern-data-stack-airflow-spark-dbt-trino-hive-metastore-l-cgmnyhhkv "PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake)") Article Data Engineer Things · 2y PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake) The PoC Data Platform demonstrates extracting, loading, and transforming data using modern data technologies like Airflow, Spark, DBT, Trino, Hive Metastore, Lightdash, and Delta Lake. It utilizes AdventureWorks data within a data lake environment and offers insights into configuring these tools for data engineering and system design. The platform provides a comprehensive Docker setup with detailed instructions, making it a valuable resource for both beginners and professionals in data systems. 40 1 22. 22 [](/posts/landing-a-data-engineer-role-free-courses-and-certifications-ge6as3bvj "Landing a Data Engineer Role: Free Courses and Certifications") Article KDnuggets · 2y Landing a Data Engineer Role: Free Courses and Certifications Training for a data engineer role doesn't have to be expensive. A curated list of 10 free data engineering courses offers quality education at no cost. Courses cover key areas such as SQL, Python, cloud data engineering, ETL and data pipelines, data warehousing, and Apache Spark. Many courses are provided by edX, and some require prior knowledge in SQL and relational databases. The article encourages that with dedication and persistence, one can achieve their data engineering goals through these free resources. 40 23. 23 [](/posts/sql-interview-problem-solution--8kzvvr9wl "SQL Interview Problem — Solution.") Article Towards AI · 2y SQL Interview Problem — Solution. The post provides a step-by-step solution to an SQL interview problem where the task is to determine the second highest employee-manager pair average salary. It details how to observe the expected output, identify conditions like the Employee-Manager pair, use self-join to fetch necessary data, calculate average salaries, and assign rankings to filter for the needed result. 39 2 24. 24 [](/posts/bridging-backend-and-data-engineering-communicating-through-events-bec6u7ldk "Bridging Backend and Data Engineering: Communicating Through Events") Article Community Picks · 2y Bridging Backend and Data Engineering: Communicating Through Events In modern software development, seamless communication between backend services and data engineering pipelines is crucial. Traditional methods like REST APIs and batch processing often fall short for real-time demands. An event-driven architecture (EDA) offers a solution by using asynchronous event communication, enabling integration of diverse systems. A practical approach is setting up a Pub/Sub system where services broadcast and consume events via standardized formats. This method allows for selective event subscription and facilitates efficient asynchronous communication without overhauling the infrastructure. 38 25. 25 [](/posts/5-python-libraries-every-data-engineer-should-know-ew9pm7ztn "5 Python Libraries Every Data Engineer Should Know") Article The New Stack · 2y 5 Python Libraries Every Data Engineer Should Know Python is a powerful language for data engineering, enhanced by essential third-party libraries. For beginners, Beautiful Soup 4 and Requests are ideal for web scraping and sending HTTP requests. Intermediate users may benefit from Apache Airflow for workflow automation and Boto3 for integrating AWS services. Advanced users can leverage Pandas for comprehensive data manipulation and analysis. 36 [See all Data Engineering archives](/tags/data-engineering/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/tags/data-engineering/best-of/2024#page","url":"https://daily.dev/tags/data-engineering/best-of/2024","name":"Best Data Engineering Posts of 2024","description":"The most upvoted Data Engineering posts of 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/tags/data-engineering/best-of/2024#items","numberOfItems":25,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/how-did-linkedin-handle-7-trillion-messages-daily-with-apache-kafka--jf20b1xpw","name":"How Did LinkedIn Handle 7 Trillion Messages Daily With Apache Kafka?"},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/roadmap-to-learn-data-engineering-how-i-would-start-again-tq6trnc6y","name":"Roadmap to Learn Data Engineering: How I Would Start Again"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/how-to-choose-the-right-type-of-database-4olcmsr9t","name":"How to choose the right type of database"},{"@type":"ListItem","position":4,"url":"https://daily.dev/posts/5-tips-for-improving-sql-query-performance-mkht1ssaz","name":"5 Tips for Improving SQL Query Performance"},{"@type":"ListItem","position":5,"url":"https://daily.dev/posts/the-state-of-data-engineering-2024-t1ywu2dho","name":"The State of Data Engineering 2024"},{"@type":"ListItem","position":6,"url":"https://daily.dev/posts/building-a-python-based-data-lake-bu41iyreu","name":"Building a Python-Based Data Lake"},{"@type":"ListItem","position":7,"url":"https://daily.dev/posts/10-built-in-python-modules-every-data-engineer-should-know-hlqhj7vgl","name":"10 Built-In Python Modules Every Data Engineer Should Know"},{"@type":"ListItem","position":8,"url":"https://daily.dev/posts/10-github-repositories-to-master-data-engineering-x2qqtjlwq","name":"10 GitHub Repositories to Master Data Engineering"},{"@type":"ListItem","position":9,"url":"https://daily.dev/posts/building-a-serverless-data-pipeline-a-step-by-step-guide-apjwxpd51","name":"Building a Serverless Data Pipeline: A Step-by-Step Guide"},{"@type":"ListItem","position":10,"url":"https://daily.dev/posts/what-is-ai-engineering--veb6hnmzc","name":"What is AI Engineering?"},{"@type":"ListItem","position":11,"url":"https://daily.dev/posts/project-ideas-to-master-data-engineering-oupn9e6gc","name":"Project Ideas to Master Data Engineering"},{"@type":"ListItem","position":12,"url":"https://daily.dev/posts/meet-megaparse-an-open-source-ai-tool-for-parsing-various-types-of-documents-for-llm-ingestion-1ua70xvbi","name":"Meet MegaParse: An Open-Source AI Tool for Parsing Various Types of Documents for LLM Ingestion"},{"@type":"ListItem","position":13,"url":"https://daily.dev/posts/trillions-of-indexes-how-uber-s-ledgerstore-supports-such-massive-scale-fi9o1kmth","name":"Trillions of Indexes: How Uber’s LedgerStore Supports Such Massive Scale"},{"@type":"ListItem","position":14,"url":"https://daily.dev/posts/7-python-libraries-every-data-engineer-should-know-3klrdr7pf","name":"7 Python Libraries Every Data Engineer Should Know"},{"@type":"ListItem","position":15,"url":"https://daily.dev/posts/the-most-important-soft-skill-in-tech-ynbmflylz","name":"The Most Important Soft Skill in Tech"},{"@type":"ListItem","position":16,"url":"https://daily.dev/posts/top-5-data-engineering-projects-for-beginners-2024-o1lrocakd","name":"Top 5 Data Engineering Projects for Beginners 2024"},{"@type":"ListItem","position":17,"url":"https://daily.dev/posts/omniparse-an-ai-platform-that-ingests-parses-any-unstructured-data-into-structured-actionable-data-kp1jrtlkj","name":"OmniParse: An AI Platform that Ingests/Parses Any Unstructured Data into Structured, Actionable Data Optimized for GenAI (LLM) Applications"},{"@type":"ListItem","position":18,"url":"https://daily.dev/posts/5-free-online-courses-to-learn-data-engineering-fundamentals-0bew3r3f1","name":"5 Free Online Courses to Learn Data Engineering Fundamentals"},{"@type":"ListItem","position":19,"url":"https://daily.dev/posts/top-data-engineering-courses-in-2024-xkkj0amci","name":"Top Data Engineering Courses in 2024"},{"@type":"ListItem","position":20,"url":"https://daily.dev/posts/etl-and-elt-eto5kaieg","name":"ETL and ELT"},{"@type":"ListItem","position":21,"url":"https://daily.dev/posts/poc-data-platform-project-utilizing-modern-data-stack-airflow-spark-dbt-trino-hive-metastore-l-cgmnyhhkv","name":"PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake)"},{"@type":"ListItem","position":22,"url":"https://daily.dev/posts/landing-a-data-engineer-role-free-courses-and-certifications-ge6as3bvj","name":"Landing a Data Engineer Role: Free Courses and Certifications"},{"@type":"ListItem","position":23,"url":"https://daily.dev/posts/sql-interview-problem-solution--8kzvvr9wl","name":"SQL Interview Problem — Solution."},{"@type":"ListItem","position":24,"url":"https://daily.dev/posts/bridging-backend-and-data-engineering-communicating-through-events-bec6u7ldk","name":"Bridging Backend and Data Engineering: Communicating Through Events"},{"@type":"ListItem","position":25,"url":"https://daily.dev/posts/5-python-libraries-every-data-engineer-should-know-ew9pm7ztn","name":"5 Python Libraries Every Data Engineer Should Know"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Tags","item":"https://daily.dev/tags"},{"@type":"ListItem","position":3,"name":"Data Engineering","item":"https://daily.dev/tags/data-engineering"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/tags/data-engineering/best-of"},{"@type":"ListItem","position":5,"name":"2024"}]}]} ```