HeyGen replaced a brittle job-queue system (MySQL + RabbitMQ + Celery + callback service) with Temporal to orchestrate AI video generation pipelines. The post details how video generation is modeled as Temporal workflows with ~90 activity types, how scene-level concurrency is managed with semaphores inside a single workflow, and how a shared platform layer centralizes identity propagation, error classification, metrics, and heartbeating. It also covers a global autoscaling coordinator (itself a Temporal workflow) that manages a heterogeneous multi-cloud GPU fleet, and a custom SlotSupplier that overlaps input preparation with GPU inference to improve utilization without double-booking the GPU.
Table of contents
The system we outgrew #Modeling video generation as a Temporal Workflow #Keeping scene orchestration in one workflow #Applying fairness at multiple layers #The shared workflow platform #Scaling a heterogeneous GPU fleet #Overlapping input preparation with GPU inference #What Temporal changed for us #73 Impressions