--- title: Best Data Engineer Things posts — November 2024 | daily.dev description: The most upvoted Data Engineer Things posts from November 2024, curated by the daily.dev community. canonical: https://daily.dev/sources/detlife/best-of/2024/11 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/sources/detlife/best-of/2024/11 og:type: website og:site_name: daily.dev og:title: Best Data Engineer Things posts — November 2024 | daily.dev og:description: The most upvoted Data Engineer Things posts from November 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Data Engineer Things — November 2024 1. 1 [](https://daily.dev/posts/i-spent-3-hours-learning-how-uber-manages-data-quality--xsplb1rhk "I spent 3 hours learning how Uber manages data quality.") Article Data Engineer Things·2y I spent 3 hours learning how Uber manages data quality. Uber leverages a comprehensive data quality platform that utilizes automatic detection and management to maintain high data standards across over 2,000 datasets. The platform includes components such as Test Execution Engine, Test Generator, and Alert Generator to ensure operational excellence. The platform automates various tasks, such as generating tests and alerts, and rerunning failed tests to verify incidents. Uber also integrates its data quality tools with other platforms to provide a seamless experience for its internal teams. 131 2 2. 2 [](https://daily.dev/posts/i-spent-4-hours-learning-how-netflix-operates-apache-iceberg-at-scale--kil5pkufh "I spent 4 hours learning how Netflix operates Apache Iceberg at scale.") Article Data Engineer Things·2y I spent 4 hours learning how Netflix operates Apache Iceberg at scale. Netflix has developed a sophisticated data platform to handle extensive data pipelines and analytics, using Apache Iceberg to overcome the limitations of their previous Hive-based system. Key components include Polaris, a custom metastore for Iceberg, and Janitors, a cleanup service. They also implemented Autotune for optimizing data layout and Autolift for localizing data files. Moreover, secure access controls were established for Iceberg tables. Netflix's migration tool for transitioning from Hive to Iceberg minimizes data movement and business interruptions. 36 3. 3 [](https://daily.dev/posts/i-spent-8-hours-learning-the-clickhouse-mergetree-table-engine-wkubcgk2u "I spent 8 hours learning the ClickHouse MergeTree Table Engine") Article Data Engineer Things·2y I spent 8 hours learning the ClickHouse MergeTree Table Engine The post provides an in-depth exploration of the ClickHouse MergeTree table engine, covering its data organization, write/read processes, merging and mutation mechanisms, and data replication methods. Key concepts such as wide and compact formats for column storage, primary and mark files, and idempotent inserts are discussed. The author explains how rows are inserted and managed within the MergeTree engine, including the handling of background merges to optimize performance. Additionally, the post addresses how data replication is utilized for high availability and increased read throughput. 32 4. 4 [](https://daily.dev/posts/i-spent-3-hours-learning-the-overview-of-clickhouse-4ckkbzl0n "I spent 3 hours learning the overview of ClickHouse") Article Data Engineer Things·2y I spent 3 hours learning the overview of ClickHouse ClickHouse is a high-performance, column-oriented SQL OLAP system developed initially for Yandex Metrica. It supports high ingestion rates, low-latency queries, and is adaptable for various data architectures. The system's architecture includes a query processing layer with vectorized execution, a storage layer with diverse table engines, and an integration layer for extensive external connectivity. ClickHouse uses sharding and replication to handle large-scale data efficiently. 30 5. 5 [](https://daily.dev/posts/how-does-netflix-ensure-the-data-quality-for-thousands-of-apache-iceberg-tables--j1ol86yuv "How does Netflix ensure the data quality for thousands of Apache Iceberg tables?") Article Data Engineer Things·2y How does Netflix ensure the data quality for thousands of Apache Iceberg tables? Netflix employs the Write-Audit-Publish (WAP) pattern using Apache Iceberg to maintain high data quality across thousands of tables. The WAP pattern involves writing data to a hidden snapshot, auditing it, and publishing it only if it passes quality checks. This approach is analogous to CI/CD workflows, ensuring validated data is exposed to downstream consumers. Apache Iceberg's structure, including manifest files, metadata files, and catalog, supports efficient snapshot management and branching, similar to version control in Git. 23 6. 6 [](https://daily.dev/posts/excel-isn-t-going-anywhere-so-let-s-automate-parsing-it-7een2bfav "Excel Isn’t Going Anywhere, So Let’s Automate Parsing It") Article Data Engineer Things·2y Excel Isn’t Going Anywhere, So Let’s Automate Parsing It Automating Excel file parsing with Python and Pandas can significantly improve efficiency, consistency, and scalability in handling messy, manually filled Excel files. This guide provides a step-by-step process to read and extract specific table data, handle issues, and alert stakeholders about any problems encountered during parsing. 21 7. 7 [](https://daily.dev/posts/change-data-capture-cdc-comprehensive-guide-postgresql-to-s3-minio-using-nifi-ex05xjw0x "Change Data Capture (CDC): Comprehensive Guide-PostgreSQL To S3(MinIO) Using NiFi") Article Data Engineer Things·2y Change Data Capture (CDC): Comprehensive Guide-PostgreSQL To S3(MinIO) Using NiFi Change Data Capture (CDC) technology is essential for real-time database updates, ensuring data integrity and quick access. This guide provides a step-by-step approach to using CDC with PostgreSQL, Debezium, Apache NiFi, and storing data in MinIO. The process involves setting up Docker Compose, configuring Debezium to monitor PostgreSQL, using Kafka as a message broker, and employing NiFi for data flow management to transfer data to MinIO for real-time analysis. 19 [See all Data Engineer Things archives](/sources/detlife/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/sources/detlife/best-of/2024/11#page","url":"https://daily.dev/sources/detlife/best-of/2024/11","name":"Best Data Engineer Things Posts — November 2024","description":"The most upvoted Data Engineer Things posts from November 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/sources/detlife/best-of/2024/11#items","numberOfItems":7,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/i-spent-3-hours-learning-how-uber-manages-data-quality--xsplb1rhk","name":"I spent 3 hours learning how Uber manages data quality."},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/i-spent-4-hours-learning-how-netflix-operates-apache-iceberg-at-scale--kil5pkufh","name":"I spent 4 hours learning how Netflix operates Apache Iceberg at scale."},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/i-spent-8-hours-learning-the-clickhouse-mergetree-table-engine-wkubcgk2u","name":"I spent 8 hours learning the ClickHouse MergeTree Table Engine"},{"@type":"ListItem","position":4,"url":"https://daily.dev/posts/i-spent-3-hours-learning-the-overview-of-clickhouse-4ckkbzl0n","name":"I spent 3 hours learning the overview of ClickHouse"},{"@type":"ListItem","position":5,"url":"https://daily.dev/posts/how-does-netflix-ensure-the-data-quality-for-thousands-of-apache-iceberg-tables--j1ol86yuv","name":"How does Netflix ensure the data quality for thousands of Apache Iceberg tables?"},{"@type":"ListItem","position":6,"url":"https://daily.dev/posts/excel-isn-t-going-anywhere-so-let-s-automate-parsing-it-7een2bfav","name":"Excel Isn’t Going Anywhere, So Let’s Automate Parsing It"},{"@type":"ListItem","position":7,"url":"https://daily.dev/posts/change-data-capture-cdc-comprehensive-guide-postgresql-to-s3-minio-using-nifi-ex05xjw0x","name":"Change Data Capture (CDC): Comprehensive Guide-PostgreSQL To S3(MinIO) Using NiFi"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Sources","item":"https://daily.dev/sources"},{"@type":"ListItem","position":3,"name":"Data Engineer Things","item":"https://daily.dev/sources/detlife"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/sources/detlife/best-of"},{"@type":"ListItem","position":5,"name":"November 2024"}]}]} ```