Daikin Applied Americas (DAA) scaled its data engineering practice by combining a structured operating model with Databricks Genie Code, an AI-assisted pipeline development tool. Rather than relying on ad hoc prompts, the team built a MECE (Mutually Exclusive, Collectively Exhaustive) skill framework where each skill encodes a specific data engineering competency — covering medallion architecture design, transformation patterns, canonical alignment and governance standards. These skills are loaded at runtime by Genie Code, replacing inconsistent prompt engineering with a governed execution model. The medallion architecture (Bronze/Silver/Gold) was reinforced with explicit checkpoints between layers, enforcing source grain definition, join validation and data stability checks before data advances. Pipelines are also anchored in stable business entities rather than technical structures, reducing ambiguity across teams. The result: pipeline prototyping that previously took days now takes minutes, outputs are more consistent across teams, and governance guardrails are embedded directly into the development workflow rather than enforced through downstream review.

7m read timeFrom databricks.com
Post cover image
Table of contents
Agentic data engineering is changing how pipelines are builtScaling data engineering through reusable skillsUsing the medallion architecture to guide pipeline developmentConnecting pipelines to business conceptsWhat changed for the teamThe payoff: what this unlocked at scaleConclusion
542 Impressions