--- title: Best Databricks posts of 2024 | daily.dev description: The most upvoted Databricks posts of 2024, curated by the daily.dev community. canonical: https://daily.dev/tags/databricks/best-of/2024 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/tags/databricks/best-of/2024 og:type: website og:site_name: daily.dev og:title: Best Databricks posts of 2024 | daily.dev og:description: The most upvoted Databricks posts of 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Databricks — 2024 1. 1 [](/posts/medallion-architecture-hype-or-useful--itqe1nj9q "Medallion Architecture Hype or Useful?") Article Data Engineering · 2y Medallion Architecture Hype or Useful? Medallion Architecture is a term coined by Databricks that aims to simplify data architecture for business and domain experts. However, it may be confusing for data professionals who are accustomed to classical data architecture models such as stage, cleansing, core, and mart, where marts are typically persisted in cubes for faster responses. 27 2 2. 2 [](/posts/building-a-cost-optimized-chatbot-with-semantic-caching-phmdo5buk "Building a Cost-Optimized Chatbot with Semantic Caching") Article databricks · 2y Building a Cost-Optimized Chatbot with Semantic Caching Chatbots are useful tools for businesses, enhancing efficiency and supporting employees by providing informed responses. However, high-performing models can be expensive to query at scale. A cost-saving strategy, semantic caching, reuses responses for similar questions, reducing redundant computations. Databricks offers an optimal platform for implementing this approach, providing necessary components like Vector Search and MLflow. While semantic caching can reduce costs and latency, slight declines in response quality must be weighed against these benefits. Databricks Mosaic AI efficiently supports these implementations with robust governance and model evaluation tools. 20 3. 3 [](/posts/data-ai-use-cases-from-the-world-s-leading-companies-1dbdurs06 "Data + AI Use Cases from the World’s Leading Companies") Article databricks · 2y Data + AI Use Cases from the World’s Leading Companies Leading companies such as GM, McDonald’s, and Unilever are using Databricks to enhance business outcomes through data and AI applications. Examples include optimizing player mechanics for the Texas Rangers, reducing processing times for Minecraft, and powering autonomous tractors for Blue River Technology. Other notable use cases involve Ahold Delhaize USA's self-service data platform and Block's AI-powered infrastructure improvements. 19 4. 4 [](/posts/an-introduction-to-time-series-forecasting-with-generative-ai-zzlfp5zes "An Introduction to Time Series Forecasting with Generative AI") Article databricks · 2y An Introduction to Time Series Forecasting with Generative AI Time series forecasting is essential for business decisions and has traditionally faced accuracy limitations. Generative AI and time series transformers provide new capabilities for improved predictions by identifying patterns in vast datasets. Popular models such as Chronos, TimesFM, Moirai, and TimeGPT offer diverse features and configurations for different forecasting needs. Organizations can leverage these models within Databricks to enhance their forecasting processes, with available notebooks to facilitate the integration of these advanced tools. 18 5. 5 [](/posts/i-spent-8-hours-relearning-the-delta-lake-table-format-b45y0i9ix "I spent 8 hours relearning the Delta Lake table format") Article Data Engineer Things · 2y I spent 8 hours relearning the Delta Lake table format Delta Lake is an ACID table storage layer built on cloud object storage, designed to address the challenges of using cloud storage for data lakes. It uses a transaction log to maintain data consistency and supports features like Time Travel, UPDATES, and DELETES. Concurrency is managed through optimistic concurrency control, and data mutation strategies include both copy-on-write and merge-on-read. The system is optimized for read/write performance and supports various data management features like layout optimizations and audit logging. 17 6. 6 [](/posts/the-ultimate-guide-to-ci-cd-for-data-engineering-in-databricks-fsmnklt6q "The Ultimate Guide to CI/CD for Data Engineering in Databricks") Article Data Engineer Things · 2y The Ultimate Guide to CI/CD for Data Engineering in Databricks Implementing Continuous Integration and Continuous Deployment (CI/CD) for data engineering in Databricks involves unique challenges, such as the interdependence of code, data, and compute resources. Solutions include using Databricks' Git integration, Asset Bundles, and other tools for automating builds, tests, and deployments. Setting up CI/CD requires managing environments, code, data assets, and complex system integrations. Proper testing and handling of data state management are crucial for effective CI/CD pipelines in data engineering. 16 7. 7 [](/posts/the-secret-to-success-in-large-scale-data-engineering-projects-n6edhalgp "The Secret to Success in Large-Scale Data Engineering Projects") Article Medium · 2y The Secret to Success in Large-Scale Data Engineering Projects This post explores how Databricks Asset Bundles (DABs) can be used for workflow implementation and automation in Databricks. It highlights the advantages of DABs in managing large-scale data projects and provides a step-by-step guide for deploying DABs. 16 8. 8 [](/posts/introducing-the-new-sql-editor-ajil2spl4 "Introducing the New SQL Editor") Article databricks · 2y Introducing the New SQL Editor Databricks announces the public preview of its new SQL editor, designed to enhance productivity with features like multiple statement results, real-time collaboration, improved assistant integrations, and enhanced editor ergonomics. New functionalities such as Quick Fix, AI-generated filters, and Git support for queries are also introduced, aiming to streamline SQL development and collaboration. 14 9. 9 [](/posts/supernovas-black-holes-and-streaming-data-inzdaepeq "Supernovas, Black Holes and Streaming Data") Article databricks · 2y Supernovas, Black Holes and Streaming Data Learn how publicly available NASA satellite data on supernovas and black holes can be consumed and processed with Apache Kafka and Databricks. This guide covers the setup and use of Databricks Data Intelligence Platform, Delta Live Tables, and AI/BI Genie to simplify the ingestion, transformation, and visualization of streaming data. Detailed steps for using open-source Apache Spark and Kafka, along with Databricks enhancements for serverless compute and natural language querying, make this complex data accessible to data scientists and engineers. 11 10. 10 [](/posts/accelerate-feature-engineering-with-photon-rctd2fdii "Accelerate Feature Engineering With Photon") Article databricks · 2y Accelerate Feature Engineering With Photon Training high-quality machine learning models involves careful data preparation, which can be time-consuming for large datasets. The Photon Engine, now available in Databricks Machine Learning Runtime, significantly speeds up Spark SQL and Spark DataFrame workloads, achieving speed improvements of 2x-4x. The Photon Engine enhances ETL processes and feature engineering, especially for time series data, using a new point-in-time join implementation. Users can enable Photon in Databricks ML Runtime 15.2 and above for better query performance. 10 [See all Databricks archives](/tags/databricks/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/tags/databricks/best-of/2024#page","url":"https://daily.dev/tags/databricks/best-of/2024","name":"Best Databricks Posts of 2024","description":"The most upvoted Databricks posts of 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/tags/databricks/best-of/2024#items","numberOfItems":10,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/medallion-architecture-hype-or-useful--itqe1nj9q","name":"Medallion Architecture Hype or Useful?"},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/building-a-cost-optimized-chatbot-with-semantic-caching-phmdo5buk","name":"Building a Cost-Optimized Chatbot with Semantic Caching"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/data-ai-use-cases-from-the-world-s-leading-companies-1dbdurs06","name":"Data + AI Use Cases from the World’s Leading Companies"},{"@type":"ListItem","position":4,"url":"https://daily.dev/posts/an-introduction-to-time-series-forecasting-with-generative-ai-zzlfp5zes","name":"An Introduction to Time Series Forecasting with Generative AI"},{"@type":"ListItem","position":5,"url":"https://daily.dev/posts/i-spent-8-hours-relearning-the-delta-lake-table-format-b45y0i9ix","name":"I spent 8 hours relearning the Delta Lake table format"},{"@type":"ListItem","position":6,"url":"https://daily.dev/posts/the-ultimate-guide-to-ci-cd-for-data-engineering-in-databricks-fsmnklt6q","name":"The Ultimate Guide to CI/CD for Data Engineering in Databricks"},{"@type":"ListItem","position":7,"url":"https://daily.dev/posts/the-secret-to-success-in-large-scale-data-engineering-projects-n6edhalgp","name":"The Secret to Success in Large-Scale Data Engineering Projects"},{"@type":"ListItem","position":8,"url":"https://daily.dev/posts/introducing-the-new-sql-editor-ajil2spl4","name":"Introducing the New SQL Editor"},{"@type":"ListItem","position":9,"url":"https://daily.dev/posts/supernovas-black-holes-and-streaming-data-inzdaepeq","name":"Supernovas, Black Holes and Streaming Data"},{"@type":"ListItem","position":10,"url":"https://daily.dev/posts/accelerate-feature-engineering-with-photon-rctd2fdii","name":"Accelerate Feature Engineering With Photon"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Tags","item":"https://daily.dev/tags"},{"@type":"ListItem","position":3,"name":"Databricks","item":"https://daily.dev/tags/databricks"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/tags/databricks/best-of"},{"@type":"ListItem","position":5,"name":"2024"}]}]} ```