Yelp's Core ML Team built Training Orchestrator to unify fragmented model training workflows across applied ML teams. Previously, each team maintained monolithic Spark scripts with duplicated boilerplate, scattered validation, and no local testability. The new system introduces a declarative, configuration-driven approach using Pydantic for type-safe configs, a DAG-based execution engine for step ordering, automatic MLflow integration for experiment tracking and reproducibility, and built-in Slack notifications. Steps are decoupled from infrastructure, enabling local runs, unit testing, and environment parity. Key outcomes include faster feedback loops, centralized configuration as a single source of truth, and reproducible runs via versioned MLflow artifacts. Future plans include first-class declarative steps for model evaluation and lineage tracking.
