--- title: Moving our warehouse off DuckDB | daily.dev description: PostHog is moving its data warehouse off DuckDB and DuckLake onto a custom Trino setup and a new open table format called HogLake. DuckDB worked great as a... canonical: https://daily.dev/posts/moving-our-warehouse-off-duckdb-ns6yl1bka twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Moving our warehouse off DuckDB | daily.dev og:description: PostHog is moving its data warehouse off DuckDB and DuckLake onto a custom Trino setup and a new open table format called HogLake. DuckDB worked great as a... og:url: https://daily.dev/posts/moving-our-warehouse-off-duckdb-ns6yl1bka og:image: https://api.daily.dev/og/posts/nS6yL1BKa.png og:image:alt: Moving our warehouse off DuckDB og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Moving our warehouse off DuckDB **[Hacker News](https://daily.dev/sources/hn)** · 6 min read · 1 upvotes · 0 comments ## Summary PostHog is moving its data warehouse off DuckDB and DuckLake onto a custom Trino setup and a new open table format called HogLake. DuckDB worked great as a single-node engine but was hard to size safely in the cloud, since resource needs per query were unpredictable. DuckLake's catalog had to stay compatible with its existing schema, which left fewer optimization options. When PostHog moved off it, DuckLake held about 2 PB, roughly 1 trillion rows and 15.3 million snapshots. Direct Postgres connections over port 5432 also proved harder to scale than HTTP due to client compatibility and connection management issues. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did PostHog move its data warehouse off DuckDB and DuckLake? PostHog moved off DuckDB and DuckLake because single-node sizing in the cloud was unpredictable and DuckLake's catalog couldn't scale to its needs. At about 2 PB and 1 trillion rows, a single-table write could take over three minutes since every commit re-read statistics for the entire catalog, and over 99% of catalog rows belonged to deleted tables. _Engineers weighing warehouse architecture tradeoffs can follow real-world scaling writeups like this on daily.dev._ ### What is HogLake? HogLake is PostHog's own open table format, built as part of a rearchitected warehouse running on a custom flavor of Trino. It replaces DuckDB and the DuckLake extension PostHog previously used, and is being tested first at PostHog's own petabyte-scale data before being offered to design partners. _Teams evaluating new open table formats can track developments like HogLake on daily.dev._ ### Why is serving Postgres wire protocol (port 5432) connections harder to scale than HTTP? Direct Postgres connections over port 5432 are harder to scale than HTTP because long-running connections are costly, reconnects and idle TTLs are hard to manage, and there's far less load-balancing tooling for pgwire than HTTP, unlike the mature options for AWS ALBs versus NLBs. Each Postgres client also expects its own niche behavior, creating a long compatibility tail. _Developers deciding between pgwire and HTTP interfaces can find firsthand scaling lessons like this on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 101 comments across hackernews (as of 2026-11-02). **TL;DR:** The discussion is largely skeptical, viewing the rapid churn from ClickHouse to DuckDB to Trino/HogLake within months as a sign of poor upfront planning rather than solid engineering, though the author actively engaged to clarify context and some technical points were debated. **Sentiment:** 10% positive · 25% mixed · 65% skeptical **The case for** - Some see rapid, agent-assisted migrations as a natural and even positive consequence of how engineering is now done, letting teams learn fast from real workloads. - A few found the author's willingness to answer questions directly in the thread valuable and rare. - Some agreed the move to a distributed engine like Trino makes sense for unpredictable, resource-heavy multi-tenant query workloads. **The pushback** - Many felt the post lacked concrete measurements, benchmarks, or methodology to justify the rewrite. - Several questioned why Iceberg/Trino or staying on ClickHouse with Iceberg support wasn't tried from the start instead of cycling through three stacks in under a year. - Commenters worried this reflects a broader pattern of trigger-happy rewrites enabled by LLM-assisted development rather than careful engineering. - Some saw the post as thinly-veiled recruiting or self-promotion rather than a genuine technical postmortem. - Criticism of the company's website/cookie banner and general engineering credibility came up repeatedly, unrelated to the technical merits. **By community** - hackernews (skeptical): Most commenters doubted the rigor behind the rapid stack changes, though the author's live responses added useful clarifying context and softened some criticism. **Hottest debate:** Whether this rapid succession of database migrations reflects sound, iterative engineering learning from real workloads, or reckless churn driven by inadequate upfront planning and LLM-accelerated rewrites. **Open questions** - What specific measurements or query-pattern studies were done before and after each migration? - Why wasn't Iceberg+Trino (or ClickHouse-on-Iceberg) adopted from the start given it's a well-known pattern? - Did the new Trino/HogLake setup actually solve the scaling and cost problems, since the post doesn't report outcomes? - What upstream contributions or collaboration attempts were made with the DuckDB/DuckLake community before forking? **Highlights** > With DuckDB and the queries we got, we had to provision such huge pods/nodes that we made it work but could never get to an economical viability. In the end, most of our customers had pods running 46 vCPUs / 360GB and asked for more. DuckDB is amazing, it really does everything right on a single node to make your query run successfully to the end almost always but it just doesn't work in our cloud env. Imagine when a query is running for 12h with this since and then another 10 from the same customer come in :) Trino on the other hand has a much more convenient distributed query concept, so you can much more dynamically assign resources. > — [roboben on hackernews](https://news.ycombinator.com/item?id=50342216) > Hi! Author here. We engineered, then we measured and then we tried to make it work, probably for a too long of a time before we decided to completely scrap our current architecture and go back to the drawing board. Happy to answer more questions. > — [roboben on hackernews · 2 comments](https://news.ycombinator.com/item?id=50341652) > > Five months ago, we rebuilt our warehouse on DuckDB instead of ClickHoue They switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative? The article then ends before saying if the new setup solved their problems. It’s great that we can now make these migrations with little downtime (I assume?) and try different things, but I wish we could have read more about what testing they did before jumping. The fact that they jumped to DuckDB without realizing it wasn’t going to work at their scale and spent only a few months with it before jumping again is not convincing. I hoe there’s more to this story that just didn’t make it into the blog posts. As written it’s giving me flashbacks to some teams I worked with who changed tech stacks every six months because they liked migrating more than tuning and maintaining. > — [Aurornis on hackernews · 6 comments](https://news.ycombinator.com/item?id=50339465) > Unironically, I think we will see a lot more of this given how engineering is being done currently. I have been involved in 2 green field projects in the last 2 years. Both have been rewritten or had massive "ship of theseus" type changes multiple times over that time window. Engineers working on new products, thanks to LLMs, have very little understanding of the technical details of the product. They have the same level of understanding a traditional high level manager had of products they owned. If you ever worked in an engineering org with an M2 or M3 who always attended your outage review, and in every outage review something has to involve some issue with Postgres or your Kafka queue, that disconnected M3 would conclude: "Almost every outage involves some problem with Postgres. My conclusion, observed over the last 6-12 months, Postgres is bad. I see this new FancyDB or DynamoDB thing is popular on Twitter. We should try that" I have lost count of the number of idiotic takes like that from M2s or M3s who have no idea how things work but instead focus on keywords they can latch onto that seems to be in the intersection of all issues. Then the logical solution is just to remove that and see what happens. This used to be very expensive, but if you can throw an LLM at a code base and say "Replace Postgres with FancyDB, split it to a list of logical checkpoints adding regression testing when missing to each checkpoint and open stacked PRs for all checkpoints", let it churn for an after noon, and 46k-line change PR (and $800 token bill) later you are ready to try FancyDB, why would you not give it a try? Then if you have issues with FancyDB, are you gonna spend the time to understand what's going on or just say "lets try DynamoDB next"? > — [eddythompson80 on hackernews · 3 comments](https://news.ycombinator.com/item?id=50341283) > Founder here -- this is about our warehouse product which is separate from our main analytics DB. Our DuckDB warehouse was used by a handful of pilot customers. Our clickhouse database is serving all of our analytical queries. > — [timgl on hackernews · 1 comments](https://news.ycombinator.com/item?id=50341562) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50338747) · 131 points · 101 comments ## Similar posts on daily.dev - [Self-hosting DuckDB: the road to production](https://daily.dev/posts/self-hosting-duckdb-the-road-to-production-crsdobrlt) · MotherDuck · 1 upvotes · 0 comments - [DuckLake: The Lakehouse That's Just SQL and Parquet](https://daily.dev/posts/ducklake-the-lakehouse-that-s-just-sql-and-parquet-vyaa3jmht) · Planet Python · 1 upvotes · 0 comments - [From Spreadsheets to DuckLake: A Data Engineer's Road to DuckDB](https://daily.dev/posts/from-spreadsheets-to-ducklake-a-data-engineer-s-road-to-duckdb-ucbstyyau) · DuckDB · 0 upvotes · 0 comments --- Tags: [#postgresql](https://daily.dev/tags/postgresql), [#data-warehouse](https://daily.dev/tags/data-warehouse), [#duckdb](https://daily.dev/tags/duckdb) [View this post on daily.dev](https://daily.dev/posts/moving-our-warehouse-off-duckdb-ns6yl1bka) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Moving our warehouse off DuckDB","url":"https://daily.dev/posts/moving-our-warehouse-off-duckdb-ns6yl1bka","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/moving-our-warehouse-off-duckdb-ns6yl1bka"},"datePublished":"2026-11-02T17:07:20.913Z","dateModified":"2026-11-02T21:18:30.268Z","description":"PostHog is moving its data warehouse off DuckDB and DuckLake onto a custom Trino setup and a new open table format called HogLake. DuckDB worked great as a...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/539faaa8d47db90a1b641ac092bee060?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/539faaa8d47db90a1b641ac092bee060?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/moving-our-warehouse-off-duckdb-ns6yl1bka","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"postgresql,data-warehouse,duckdb","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Moving our warehouse off DuckDB"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/moving-our-warehouse-off-duckdb-ns6yl1bka#faq","mainEntity":[{"@type":"Question","name":"Why did PostHog move its data warehouse off DuckDB and DuckLake?","acceptedAnswer":{"@type":"Answer","text":"PostHog moved off DuckDB and DuckLake because single-node sizing in the cloud was unpredictable and DuckLake's catalog couldn't scale to its needs. At about 2 PB and 1 trillion rows, a single-table write could take over three minutes since every commit re-read statistics for the entire catalog, and over 99% of catalog rows belonged to deleted tables. Engineers weighing warehouse architecture tradeoffs can follow real-world scaling writeups like this on daily.dev."}},{"@type":"Question","name":"What is HogLake?","acceptedAnswer":{"@type":"Answer","text":"HogLake is PostHog's own open table format, built as part of a rearchitected warehouse running on a custom flavor of Trino. It replaces DuckDB and the DuckLake extension PostHog previously used, and is being tested first at PostHog's own petabyte-scale data before being offered to design partners. Teams evaluating new open table formats can track developments like HogLake on daily.dev."}},{"@type":"Question","name":"Why is serving Postgres wire protocol (port 5432) connections harder to scale than HTTP?","acceptedAnswer":{"@type":"Answer","text":"Direct Postgres connections over port 5432 are harder to scale than HTTP because long-running connections are costly, reconnects and idle TTLs are hard to manage, and there's far less load-balancing tooling for pgwire than HTTP, unlike the mature options for AWS ALBs versus NLBs. Each Postgres client also expects its own niche behavior, creating a long compatibility tail. Developers deciding between pgwire and HTTP interfaces can find firsthand scaling lessons like this on daily.dev."}}]} ```