--- title: How ngrok uses Dagster to run our data platform | daily.dev description: ngrok describes how it orchestrates its data platform with Dagster, treating data as assets rather than abstract tasks. The stack combines Apache Iceberg for... canonical: https://daily.dev/posts/how-ngrok-uses-dagster-to-run-our-data-platform-qelr3utuc twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How ngrok uses Dagster to run our data platform | daily.dev og:description: ngrok describes how it orchestrates its data platform with Dagster, treating data as assets rather than abstract tasks. The stack combines Apache Iceberg for... og:url: https://daily.dev/posts/how-ngrok-uses-dagster-to-run-our-data-platform-qelr3utuc og:image: https://api.daily.dev/og/posts/QelR3UtUc.png og:image:alt: How ngrok uses Dagster to run our data platform og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How ngrok uses Dagster to run our data platform **[ngrok Blog](https://daily.dev/sources/ngrok)** · 9 min read · 0 upvotes · 0 comments ## Summary ngrok describes how it orchestrates its data platform with Dagster, treating data as assets rather than abstract tasks. The stack combines Apache Iceberg for storage, AWS Athena for queries, dbt for modeling, Spark and AWS Glue for heavy workloads, and Kafka/Kinesis for streaming, managing about 20TB of new data monthly across 400+ Dagster assets and 170 dbt models. The post details the dagster-dbt and dagster-airbyte integrations, how asset keys let Dagster unify lineage across dbt sources and non-dbt jobs like Spark, and how Dagster is deployed on Kubernetes behind ngrok's own Ingress Controller with OAuth protection. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does the dagster-dbt integration handle a dbt source() that actually points to a non-dbt asset like a Spark job? Dagster's dbt integration detects when a dbt source() function shares an AssetKey with an existing Dagster asset created outside dbt, such as a table produced by a Spark job, and automatically replaces the bare source reference with the richer asset definition. This surfaces full metadata, descriptions, and upstream lineage instead of an empty source stub, and Dagster will skip dependent dbt models if that upstream Spark job fails. _Teams wiring Dagster and dbt lineage together can track integration deep dives like this on daily.dev._ ### What does a typical Dagster-based ETL pipeline for pulling Salesforce data into a data warehouse look like? One pattern extracts Salesforce data via Airbyte, writes it to S3 as nested JSON, processes it with Spark into Apache Iceberg tables, and then exposes the result through dbt models, all orchestrated end to end by Dagster using the dagster-airbyte and dagster-aws integrations. This chains three components (Airbyte, Spark, dbt) into a single monitored dependency graph rather than separate disconnected jobs. _Engineers assembling multi-tool ETL pipelines can follow real-world orchestration patterns like this on daily.dev._ ## Similar posts on daily.dev - [The Dagster Almanack: Operationalizing Data Orchestration](https://daily.dev/posts/the-dagster-almanack-operationalizing-data-orchestration-hmpjorjvu) · Dagster Blog · 4 upvotes · 0 comments - [Creative Dagster Projects from the Community](https://daily.dev/posts/creative-dagster-projects-from-the-community-d3reeugog) · Dagster Blog · 0 upvotes · 0 comments - [Building an Enterprise Data Warehouse on Heroku: From Complex ETL to Seamless Salesforce Integration](https://daily.dev/posts/building-an-enterprise-data-warehouse-on-heroku-from-complex-etl-to-seamless-salesforce-integration-s43jz7u9t) · Heroku · 1 upvotes · 0 comments - [Real-Time AI Pipelines: Building a Kafka \+ Kubernetes \+ Databricks Stack for Agentic AI Systems](https://daily.dev/posts/real-time-ai-pipelines-building-a-kafka-kubernetes-databricks-stack-for-agentic-ai-systems-qoytpduhq) · Medium · 0 upvotes · 0 comments - [Open Platform, Unified Pipelines: Why dbt on Databricks is Accelerating](https://daily.dev/posts/open-platform-unified-pipelines-why-dbt-on-databricks-is-accelerating-jrmprogl5) · databricks · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-iceberg](https://daily.dev/tags/apache-iceberg), [#dagster](https://daily.dev/tags/dagster) [View this post on daily.dev](https://daily.dev/posts/how-ngrok-uses-dagster-to-run-our-data-platform-qelr3utuc) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How ngrok uses Dagster to run our data platform","url":"https://daily.dev/posts/how-ngrok-uses-dagster-to-run-our-data-platform-qelr3utuc","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-ngrok-uses-dagster-to-run-our-data-platform-qelr3utuc"},"datePublished":"2026-08-23T07:21:30.900Z","dateModified":"2026-08-23T07:28:30.240Z","description":"ngrok describes how it orchestrates its data platform with Dagster, treating data as assets rather than abstract tasks. The stack combines Apache Iceberg for...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/11bc5af882b60b0e903cf997a3975f06?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/11bc5af882b60b0e903cf997a3975f06?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ngrok Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ngrok Blog","logo":"https://media.daily.dev/image/upload/s--718XXxvE--/f_auto,q_auto/v1787469610/logos/ngrok?_a=BAMAMicg0","url":"https://daily.dev/sources/ngrok"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-ngrok-uses-dagster-to-run-our-data-platform-qelr3utuc","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,data-engineering,apache-iceberg,dagster","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ngrok Blog","item":"https://daily.dev/sources/ngrok"},{"@type":"ListItem","position":3,"name":"How ngrok uses Dagster to run our data platform"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-ngrok-uses-dagster-to-run-our-data-platform-qelr3utuc#faq","mainEntity":[{"@type":"Question","name":"How does the dagster-dbt integration handle a dbt source() that actually points to a non-dbt asset like a Spark job?","acceptedAnswer":{"@type":"Answer","text":"Dagster's dbt integration detects when a dbt source() function shares an AssetKey with an existing Dagster asset created outside dbt, such as a table produced by a Spark job, and automatically replaces the bare source reference with the richer asset definition. This surfaces full metadata, descriptions, and upstream lineage instead of an empty source stub, and Dagster will skip dependent dbt models if that upstream Spark job fails. Teams wiring Dagster and dbt lineage together can track integration deep dives like this on daily.dev."}},{"@type":"Question","name":"What does a typical Dagster-based ETL pipeline for pulling Salesforce data into a data warehouse look like?","acceptedAnswer":{"@type":"Answer","text":"One pattern extracts Salesforce data via Airbyte, writes it to S3 as nested JSON, processes it with Spark into Apache Iceberg tables, and then exposes the result through dbt models, all orchestrated end to end by Dagster using the dagster-airbyte and dagster-aws integrations. This chains three components (Airbyte, Spark, dbt) into a single monitored dependency graph rather than separate disconnected jobs. Engineers assembling multi-tool ETL pipelines can follow real-world orchestration patterns like this on daily.dev."}}]} ```