--- title: Building an open data pipeline in 2024 | daily.dev description: Building an open data pipeline in 2024 involves understanding requirements for data scale, latency, governance and access controls, and cost. By using Iceberg... canonical: https://daily.dev/posts/building-an-open-data-pipeline-in-2024-5htoiyned twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Building an open data pipeline in 2024 | daily.dev og:description: Building an open data pipeline in 2024 involves understanding requirements for data scale, latency, governance and access controls, and cost. By using Iceberg... og:url: https://daily.dev/posts/building-an-open-data-pipeline-in-2024-5htoiyned og:image: https://api.daily.dev/og/posts/5HTOiyNEd.png og:image:alt: Building an open data pipeline in 2024 og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Building an open data pipeline in 2024 **[Hacker News](https://daily.dev/sources/hn)** · 5 min read · 11 upvotes · 0 comments ## Summary Building an open data pipeline in 2024 involves understanding requirements for data scale, latency, governance and access controls, and cost. By using Iceberg as the core data storage layer, you can leverage different compute environments and achieve flexibility and cost-effectiveness. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Don’t just attend KubeCon \+ CloudNativeCon, Merge Forward your experience\!](https://daily.dev/posts/don-t-just-attend-kubecon-cloudnativecon-merge-forward-your-experience--l0rpp73x8) · CNCF · 1 upvotes · 0 comments - [Announcing H2 2026 KCDs](https://daily.dev/posts/announcing-h2-2026-kcds-m96goajm1) · CNCF · 1 upvotes · 0 comments - [Two months of Open Community Groups](https://daily.dev/posts/two-months-of-open-community-groups-asf52zhbs) · CNCF · 0 upvotes · 0 comments - [CNCF Unveils Schedule for KubeCon \+ CloudNativeCon Europe 2026](https://daily.dev/posts/cncf-unveils-schedule-for-kubecon-cloudnativecon-europe-2026-ikhcoa5cb) · CNCF · 2 upvotes · 0 comments - [CNCF Debuts KubeCon \+ CloudNativeCon Japan 2026 Schedule](https://daily.dev/posts/cncf-debuts-kubecon-cloudnativecon-japan-2026-schedule-xp5pyudub) · CNCF · 1 upvotes · 0 comments --- Tags: [#backend](https://daily.dev/tags/backend), [#big-data](https://daily.dev/tags/big-data) [View this post on daily.dev](https://daily.dev/posts/building-an-open-data-pipeline-in-2024-5htoiyned) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Building an open data pipeline in 2024","url":"https://daily.dev/posts/building-an-open-data-pipeline-in-2024-5htoiyned","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/building-an-open-data-pipeline-in-2024-5htoiyned"},"datePublished":"2024-04-27T02:26:37.472Z","dateModified":"2024-05-09T09:30:38.138Z","description":"Building an open data pipeline in 2024 involves understanding requirements for data scale, latency, governance and access controls, and cost. By using Iceberg...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/38d4a5b7455502d2813342403fd24dcd?_a=AQAEuiZ","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/38d4a5b7455502d2813342403fd24dcd?_a=AQAEuiZ","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/building-an-open-data-pipeline-in-2024-5htoiyned","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":11},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"backend,big-data","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Building an open data pipeline in 2024"}]} ```