Enabling Data Intelligence: Data Profiling Framework at Halodoc

Halodoc built a scalable, config-driven Data Profiling Framework on top of Apache Airflow to bring automated visibility and trust to their data pipelines. The system has three components: a Column-Level Profiler (computing null rates, distinct counts, distributions, and sample records), a Join Intelligence profiler (parsing production SQL files with SQLGlot AST traversal to extract real join relationships), and a Source Table Analyser (using information_schema metadata and Hudi metrics to generate partition and compaction recommendations). Key design choices include Kubernetes pod fan-out for parallel isolated execution, pushdown compute to Redshift/Athena, a staging table safety pattern for atomic writes, and a config-driven DAG trigger requiring no code changes to profile new tables. The framework reduced column profiling time from ~1 hour to ~10 minutes and join discovery from 30–60 minutes to under 10 minutes, while enabling proactive detection of data drift and partition explosion issues.