--- title: MiniLake Is a Local Databricks API Emulator with SQL,... description: MiniLake is a local emulator that implements the Databricks API, including Unity Catalog, SQL statements and Delta Lake, for running entirely on localhost. It... canonical: https://daily.dev/posts/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark-eavtazych twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark | daily.dev og:description: MiniLake is a local emulator that implements the Databricks API, including Unity Catalog, SQL statements and Delta Lake, for running entirely on localhost. It... og:url: https://daily.dev/posts/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark-eavtazych og:image: https://api.daily.dev/og/posts/EavTAzycH.png og:image:alt: MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark **[Hacker News](https://daily.dev/sources/hn)** · 2 min read · 0 upvotes · 0 comments ## Summary MiniLake is a local emulator that implements the Databricks API, including Unity Catalog, SQL statements and Delta Lake, for running entirely on localhost. It lets you point the databricks-sdk, Terraform, Databricks Asset Bundles or Spark at http://localhost:8000 instead of a real workspace. Catalogs, schemas and tables are created and queried through the same API calls used in production, backed by real DuckDB databases and real Delta files with parquet and a _delta_log directory. Spark jobs run in a sibling container and return actual output. Unsupported APIs fail loudly with a 501 instead of pretending to succeed, and a reset endpoint gives a clean workspace without restarting. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Is there a way to run the Databricks API locally without a real workspace for testing? MiniLake is a local emulator that implements the Databricks API surface, including Unity Catalog, SQL statement execution and Delta Lake, running at localhost:8000. You can point the databricks-sdk, Terraform's databricks provider, Databricks Asset Bundles, or Spark directly at it, with catalogs and tables backed by real DuckDB databases and real Delta files. _daily.dev surfaces tools like this for teams building local Databricks testing workflows._ ### What happens if MiniLake doesn't support a particular Databricks API I'm calling? Resources backed by APIs MiniLake does not emulate fail loudly with an HTTP 501 error, rather than silently pretending the call succeeded. This makes gaps in emulation coverage visible immediately instead of causing confusing downstream failures in Terraform applies or SDK calls. _developers evaluating API emulators weigh failure behavior like this before trusting them in CI, a comparison daily.dev helps track._ ### Can Terraform and Databricks Asset Bundles be tested against a local Databricks-compatible backend? Yes, the databricks Terraform provider can target http://localhost:8000 so 'terraform apply' creates real catalogs and real DuckDB databases, and 'terraform destroy' removes them. Databricks Asset Bundles work the same way: 'databricks bundle deploy' and 'bundle run' execute jobs on real Spark in a sibling container and return actual output. _engineers wiring Terraform and bundle deployments into CI can track tooling like this on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 22 comments across hackernews (as of 2026-10-29). **TL;DR:** Developers who struggle to test against real cloud data warehouses see strong appeal in a local Databricks emulator for integration testing, though some question whether it can faithfully replicate Databricks' quirks, renames, and feature restrictions closely enough to be trustworthy. **Sentiment:** 40% positive · 40% mixed · 20% skeptical **The case for** - Running expensive, hard-to-provision services like Databricks, Snowflake, Redshift locally for integration testing would be a huge time/cost saver. - It could be great for CI pipelines and homelab setups. - Having a reset endpoint and real backing files (DuckDB/Delta/parquet) rather than a fake is seen as a meaningful design choice. **The pushback** - Some doubt the value of integration tests against real providers at all, calling them slow and brittle. - Databricks constantly renames products and has inconsistent feature support across its own products, so an emulator may struggle to keep up or replicate those nuances faithfully. - The landing page and docs were criticized as generic, AI-generated-looking content that signals low effort even if the tool itself is useful. **By community** - hackernews (mixed): Many see real value for integration testing against otherwise hard-to-run cloud warehouses, but others question the testing philosophy, the emulator's fidelity to Databricks' quirks, and the generic AI-style presentation of the project. **Hottest debate:** Whether integration tests against real/emulated cloud providers are actually valuable or just slow and brittle with little payoff. **Open questions** - Can the emulator keep pace with Databricks' frequent product renames and inconsistent feature support across its various warehouse types? - Will it be extended further, and in what directions, per requests from people who wish it existed sooner? - Are there risks or limitations when trying to use similar local-emulation approaches on GCP or Azure equivalents? **Highlights** > If this actually works, this is such a huge boon. Let me share why: I work on connectors to various datasources, mostly SQL DB's The most useful tests are the integration tests, sending actual queries to the source For most DB's we have a giant Docker Compose which runs a local instance But things like Snowflake, Databricks, Redshift, Athena suck to test because we need an external hosted instance I would kill to be able to run test versions of all of these locally > — [gavinray on hackernews · 2 comments](https://news.ycombinator.com/item?id=50287407) > I'm tempted to use this but the rub with Databricks is that there are always a bunch of different restrictions on what features work in what products (for example RT//Lakehouse looks great but runs a more limited regex parser than other warehouses), and so I'm basically forced to rely heavily on integration tests to catch weird behavior. This would need to copy it closely enough to catch those. The other problem is that Databricks renames everything like every 6 months so keeping up with them is non-trivial. Their DLT product alone has gone on a jazz odyssey of renames. > — [viccis on hackernews](https://news.ycombinator.com/item?id=50292246) > Databricks is a furnace. It burns money. > — [NDizzle on hackernews](https://news.ycombinator.com/item?id=50289010) > I would say go straight to their GitHub but there again, same thing. With a list of "Known gaps" and implemented engines listed with status like "Real", and of course the Quick start first, as Claude loves to do. It's not "wrong", it just shows low effort. But then again it's just something someone needed and shared it for free so let's not complain too much. All the products and saas launching with obvious AI landing pages, that the authors did not even care to trim down, that's another story. > — [smashed on hackernews](https://news.ycombinator.com/item?id=50287865) > Why are you writing tests on the actual provider? IME these types of tests are slow, brittle and don't give you much value. Even if you get a true positive, what can you do about it? > — [skeeter2020 on hackernews · 3 comments](https://news.ycombinator.com/item?id=50289054) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50259697) · 84 points · 22 comments ## Similar posts on daily.dev - [Develop and deploy Serverless JARs](https://daily.dev/posts/develop-and-deploy-serverless-jars-spy7rgvds) · databricks · 0 upvotes · 0 comments - [DuckLake to MotherDuck: Validate locally, deploy to cloud in minutes](https://daily.dev/posts/ducklake-to-motherduck-validate-locally-deploy-to-cloud-in-minutes-qsfki85tt) · dltHub · 0 upvotes · 0 comments - [How to Build Production-Ready Data and AI Apps with Databricks Apps and Lakebase](https://daily.dev/posts/how-to-build-production-ready-data-and-ai-apps-with-databricks-apps-and-lakebase-u6tqwxua6) · databricks · 1 upvotes · 0 comments - [DuckLake 1.0: Data Lake Format with SQL Catalog Metadata](https://daily.dev/posts/ducklake-1-0-data-lake-format-with-sql-catalog-metadata-i1k4mw1ne) · InfoQ · 0 upvotes · 0 comments --- Tags: [#terraform](https://daily.dev/tags/terraform), [#apache-spark](https://daily.dev/tags/apache-spark), [#databricks](https://daily.dev/tags/databricks), [#unity-catalog](https://daily.dev/tags/unity-catalog) [View this post on daily.dev](https://daily.dev/posts/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark-eavtazych) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark","url":"https://daily.dev/posts/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark-eavtazych","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark-eavtazych"},"datePublished":"2026-10-29T13:15:33.988Z","dateModified":"2026-10-29T18:31:39.268Z","description":"MiniLake is a local emulator that implements the Databricks API, including Unity Catalog, SQL statements and Delta Lake, for running entirely on localhost. It...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/10cc9a578d7983cab8e0ad8cc300ad53?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/10cc9a578d7983cab8e0ad8cc300ad53?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark-eavtazych","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"terraform,apache-spark,databricks,unity-catalog","timeRequired":"PT2M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark-eavtazych#faq","mainEntity":[{"@type":"Question","name":"Is there a way to run the Databricks API locally without a real workspace for testing?","acceptedAnswer":{"@type":"Answer","text":"MiniLake is a local emulator that implements the Databricks API surface, including Unity Catalog, SQL statement execution and Delta Lake, running at localhost:8000. You can point the databricks-sdk, Terraform's databricks provider, Databricks Asset Bundles, or Spark directly at it, with catalogs and tables backed by real DuckDB databases and real Delta files. daily.dev surfaces tools like this for teams building local Databricks testing workflows."}},{"@type":"Question","name":"What happens if MiniLake doesn't support a particular Databricks API I'm calling?","acceptedAnswer":{"@type":"Answer","text":"Resources backed by APIs MiniLake does not emulate fail loudly with an HTTP 501 error, rather than silently pretending the call succeeded. This makes gaps in emulation coverage visible immediately instead of causing confusing downstream failures in Terraform applies or SDK calls. developers evaluating API emulators weigh failure behavior like this before trusting them in CI, a comparison daily.dev helps track."}},{"@type":"Question","name":"Can Terraform and Databricks Asset Bundles be tested against a local Databricks-compatible backend?","acceptedAnswer":{"@type":"Answer","text":"Yes, the databricks Terraform provider can target http://localhost:8000 so 'terraform apply' creates real catalogs and real DuckDB databases, and 'terraform destroy' removes them. Databricks Asset Bundles work the same way: 'databricks bundle deploy' and 'bundle run' execute jobs on real Spark in a sibling container and return actual output. engineers wiring Terraform and bundle deployments into CI can track tooling like this on daily.dev."}}]} ```