dlthub offers two approaches to running data transformations: native dlthub transformations and dbt integration. Native transformations use a single decorator (@dlt.hub.transformation), support Ibis, raw SQL, pandas, Polars, and Arrow, and run in the same runtime as ingestion — sharing credentials, scheduling, and metadata. This unified context makes them agent-friendly, with schemas, lineage, and PII annotations flowing end-to-end. The scheduling system uses Python decorators with event-based triggers (success/fail/completed) to eliminate race conditions common in time-based scheduling. For teams with existing dbt projects, dlthub can trigger dbt Core or dbt Cloud jobs after ingestion completes, removing the need for a separate orchestrator and eliminating the risk of transforming stale data. The post frames native transformations as an 'AI harness' that captures tribal knowledge and enables less senior engineers to operate at a higher level.

8m read timeFrom dlthub.com
Post cover image
Table of contents
The evolution of the data dev harness Link iconTransformations on dltHub: one place for the meaning, two ways to run them Link iconMethod 1: Run dltHub transformations natively Link iconHow dltHub handles scheduling Link iconMethod 2: Run dbt from dltHub Link iconWhere to start Link icon
182 Impressions