--- title: Best Data Engineer Things posts — December 2024 | daily.dev description: The most upvoted Data Engineer Things posts from December 2024, curated by the daily.dev community. canonical: https://daily.dev/sources/detlife/best-of/2024/12 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/sources/detlife/best-of/2024/12 og:type: website og:site_name: daily.dev og:title: Best Data Engineer Things posts — December 2024 | daily.dev og:description: The most upvoted Data Engineer Things posts from December 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Data Engineer Things — December 2024 1. 1 [](https://daily.dev/posts/etl-and-elt-eto5kaieg "ETL and ELT") Article Data Engineer Things · 2y ETL and ELT The author reflects on their journey from chasing the latest data engineering tools to focusing on foundational concepts, emphasizing the shift from ETL (Extract, Transform, Load) to ELT (Extract, Load, Transform). The traditional ETL process, necessitated by the high costs and limitations of early data warehouses, is contrasted with the modern ELT approach, facilitated by advancements in cloud data warehousing. ELT offers greater flexibility and efficiency by loading raw data into the warehouse and handling transformations within the warehouse, aligning better with agile development practices. 45 4 2. 2 [](https://daily.dev/posts/apache-flink-overview-tjlcrxz28 "Apache Flink Overview") Article Data Engineer Things · 2y Apache Flink Overview Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. It excels in real-time processing with a model centered on streams, using components such as Dispatcher, JobManager, ResourceManager, and TaskManager. Flink differentiates between event-time and processing-time semantics to manage complexities in data flows. It also offers robust state management and checkpointing to ensure fault tolerance. Its architecture supports scalable, high-throughput, and low-latency processing environments, making it suitable for applications involving complex event data. 42 2 3. 3 [](https://daily.dev/posts/the-data-lake-warehouse-and-lakehouse-mdea4qaxv "The Data Lake, Warehouse and Lakehouse") Article Data Engineer Things · 2y The Data Lake, Warehouse and Lakehouse The post explores the evolution of data architecture, beginning with traditional data warehouses, followed by the introduction of data lakes, and culminating in the emergence of the Lakehouse paradigm. It highlights the limitations of data warehouses and data lakes, such as challenges with unstructured data and data staleness. The Lakehouse architecture aims to combine the best features of both by utilizing low-cost storage and enhancing management features such as ACID transactions and query optimization. The post also mentions various technologies like Delta Lake, Apache Hudi, and Apache Iceberg that facilitate efficient data management in Lakehouse architectures. 29 4. 4 [](https://daily.dev/posts/your-machine-your-ai-the-ultimate-local-productivity-stack-with-ollama-ibwqjzg4p "Your Machine, Your AI — The Ultimate Local Productivity Stack with Ollama") Article Data Engineer Things · 2y Your Machine, Your AI — The Ultimate Local Productivity Stack with Ollama Discover how to boost your productivity using local AI tools like Ollama with the Gemma 2 and LLaVA models. Learn to set up these tools, execute various tasks such as code reviews, summarizing text, creating mind maps, and analyzing images, all while keeping data secure and private on your local machine. Integrate scripted workflows with Raycast for enhanced efficiency, and explore using Open WebUI for a local AI chat setup. The guide provides practical examples and encourages the creation of personalized AI-powered helper scripts to automate repetitive tasks. 19 5. 5 [](https://daily.dev/posts/talk-to-airflow-build-an-ai-agent-using-pydanticai-and-gemini-2-0-hgrygrcwe "Talk to Airflow — Build an AI Agent Using PydanticAI and Gemini 2.0") Article Data Engineer Things · 2y Talk to Airflow — Build an AI Agent Using PydanticAI and Gemini 2.0 Developing AI applications can be challenging, but tools like PydanticAI and models like Gemini 2.0 are making it easier. This post guides you through building an AI agent that interacts with Apache Airflow using PydanticAI, highlighting its simplicity and production-readiness. The AI agent can understand natural language queries, fetch real-time status updates, and provide structured data, similar to how modern cockpits give pilots full control over their aircraft. Examples and detailed steps showcase how to set up the environment, create sample DAGs, and implement an AI agent from scratch. 18 6. 6 [](https://daily.dev/posts/building-machine-learning-pipelines-with-the-fti-architecture-a-practical-step-by-step-guide-c9utevqdp "Building Machine Learning Pipelines with the FTI Architecture: A Practical Step-by-Step Guide") Article Data Engineer Things · 2y Building Machine Learning Pipelines with the FTI Architecture: A Practical Step-by-Step Guide FTI (Feature, Training, Inference) architecture offers a modular and scalable framework for building machine learning pipelines. It divides the workflow into three independent stages: Feature Pipeline, Training Pipeline, and Inference Pipeline. This approach ensures modularity, reusability, consistency, scalability, and reproducibility. The Feature Pipeline transforms raw data into engineered features, the Training Pipeline manages the model's lifecycle, and the Inference Pipeline serves real-time or batch predictions using the trained model. 14 [See all Data Engineer Things archives](/sources/detlife/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/sources/detlife/best-of/2024/12#page","url":"https://daily.dev/sources/detlife/best-of/2024/12","name":"Best Data Engineer Things Posts — December 2024","description":"The most upvoted Data Engineer Things posts from December 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/sources/detlife/best-of/2024/12#items","numberOfItems":6,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/etl-and-elt-eto5kaieg","name":"ETL and ELT"},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/apache-flink-overview-tjlcrxz28","name":"Apache Flink Overview"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/the-data-lake-warehouse-and-lakehouse-mdea4qaxv","name":"The Data Lake, Warehouse and Lakehouse"},{"@type":"ListItem","position":4,"url":"https://daily.dev/posts/your-machine-your-ai-the-ultimate-local-productivity-stack-with-ollama-ibwqjzg4p","name":"Your Machine, Your AI — The Ultimate Local Productivity Stack with Ollama"},{"@type":"ListItem","position":5,"url":"https://daily.dev/posts/talk-to-airflow-build-an-ai-agent-using-pydanticai-and-gemini-2-0-hgrygrcwe","name":"Talk to Airflow — Build an AI Agent Using PydanticAI and Gemini 2.0"},{"@type":"ListItem","position":6,"url":"https://daily.dev/posts/building-machine-learning-pipelines-with-the-fti-architecture-a-practical-step-by-step-guide-c9utevqdp","name":"Building Machine Learning Pipelines with the FTI Architecture: A Practical Step-by-Step Guide"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Sources","item":"https://daily.dev/sources"},{"@type":"ListItem","position":3,"name":"Data Engineer Things","item":"https://daily.dev/sources/detlife"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/sources/detlife/best-of"},{"@type":"ListItem","position":5,"name":"December 2024"}]}]} ```