A deep-dive into operationalizing data orchestration with Dagster, covering DevOps and GitOps best practices, code location architecture, deployment options (Kubernetes/Helm, Docker Compose, AWS ECS, Dagster+), and governance. Key themes include separating business logic from infrastructure code, using workspaces/code locations for multi-tenancy, partitioned backfills for cost efficiency, and asset-level data quality checks. Includes a reference GitHub blueprint for Kubernetes-based GitOps data platforms.
Table of contents
DevOps and GitOps: Why It's a Given?How to Operationalize Data Orchestration?How to Deploy Dagster: Kubernetes, Helm, DockerGuardrails and PitfallsOrchestration as the Operating System of Your Data Platform4K Impressions