--- title: Best Data Engineer Things posts — September 2024 | daily.dev description: The most upvoted Data Engineer Things posts from September 2024, curated by the daily.dev community. canonical: https://daily.dev/sources/detlife/best-of/2024/09 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/sources/detlife/best-of/2024/09 og:type: website og:site_name: daily.dev og:title: Best Data Engineer Things posts — September 2024 | daily.dev og:description: The most upvoted Data Engineer Things posts from September 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Data Engineer Things — September 2024 1. 1 [](https://daily.dev/posts/i-spent-5-hours-learning-how-google-manages-terabytes-of-metadata-for-bigquery--q1etmyej9 "I spent 5 hours learning how Google manages terabytes of metadata for BigQuery.") Article Data Engineer Things · 2y I spent 5 hours learning how Google manages terabytes of metadata for BigQuery. Google BigQuery uses innovative techniques to manage massive amounts of metadata efficiently, treating it as crucial as the data itself. BigQuery's architecture includes Colossus for storage, Dremel for querying, and a dedicated shuffle service, all coordinated by Borg. Metadata is handled in a distributed manner using a unique columnar storage format called CMETA, improving efficiency and performance. Real-time data ensures physical query plans adapt dynamically for optimized results, while integrated metadata scans enhance query processing. 51 2. 2 [](https://daily.dev/posts/i-spent-5-hours-learning-how-clickhouse-built-their-internal-data-warehouse--ykgna2cg3 "I spent 5 hours learning how ClickHouse built their internal data warehouse.") Article Data Engineer Things · 2y I spent 5 hours learning how ClickHouse built their internal data warehouse. ClickHouse has built an internal data warehouse to handle 50 TB of data daily, incorporating multiple internal sources like AWS, GCP, and Salesforce. They use Airflow for scheduling, AWS S3 as the intermediate data layer, and Superset for BI tools. Key features include data consistency, idempotency, and real-time analytics. They have also adopted dbt to streamline data transformations and introduced new tools for improved user access to data. 42 [See all Data Engineer Things archives](/sources/detlife/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/sources/detlife/best-of/2024/09#page","url":"https://daily.dev/sources/detlife/best-of/2024/09","name":"Best Data Engineer Things Posts — September 2024","description":"The most upvoted Data Engineer Things posts from September 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/sources/detlife/best-of/2024/09#items","numberOfItems":2,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/i-spent-5-hours-learning-how-google-manages-terabytes-of-metadata-for-bigquery--q1etmyej9","name":"I spent 5 hours learning how Google manages terabytes of metadata for BigQuery."},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/i-spent-5-hours-learning-how-clickhouse-built-their-internal-data-warehouse--ykgna2cg3","name":"I spent 5 hours learning how ClickHouse built their internal data warehouse."}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Sources","item":"https://daily.dev/sources"},{"@type":"ListItem","position":3,"name":"Data Engineer Things","item":"https://daily.dev/sources/detlife"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/sources/detlife/best-of"},{"@type":"ListItem","position":5,"name":"September 2024"}]}]} ```