Callstack describes how it automated its React Native Evals benchmark suite using Apache Airflow to orchestrate solver/judge model pairs and MLflow Tracking to keep run configurations, metrics, and artifacts together. The workflow moves from manual, ad-hoc benchmark runs to a six-stage pipeline: an operator selects models and eval scope, Airflow spins up isolated containerized workers per solver/judge pair, results are tracked in MLflow experiments, and human reviewers gate publication of results to the public benchmark site. The team reports that adding a new supported model and completing its evals now takes about a day instead of about a week, though this excludes review and publication time.
Table of contents
Stage 1: choose the runStage 2: solve and judge each taskStage 3: keep the run record and files togetherStage 4: review and publish the resultWhat changed for the operatorQuestions this post answers
How can I automate running AI coding model benchmarks with a solver and judge setup?
Use a workflow orchestrator like Apache Airflow to schedule a separate containerized worker for each solver/judge model pair, invoking both models through an AI coding agent such as OpenCode. Each worker keeps its own workspace and logs, and the orchestrator marks the run successful only after all required pairs and their configured repeats finish. daily.dev surfaces engineering writeups like this for teams designing their own AI eval pipelines.
What is MLflow Tracking used for in an AI benchmark pipeline?
It keeps each model pair's run configuration, status, metrics, generated files, scores, and summaries together for later comparison and review, since benchmark workers are short-lived and need a durable record after execution ends. Each solver/judge pair gets its own MLflow experiment, and it can be disabled in favor of a separate file export. Track how teams pair tools like MLflow with orchestration on daily.dev when building eval infrastructure.
How much faster is it to add a new model to an automated evaluation benchmark versus a manual setup?
Adding a supported model and completing its evals takes about a day with an automated Airflow-based pipeline, compared to about a week with manual setup, according to one team's practice. This comparison covers only the completed evals step, not review or publication time. daily.dev follows workflow automation wins like this for engineers weighing manual versus automated benchmarking.