Engineers from Temporal's cloud team share production lessons from running Temporal Cloud's control plane on Temporal itself. Key patterns covered include: the entity workflow pattern for pessimistic concurrency on shared resources, modeling long-running async requests as workflows for idempotency and status tracking, payload encryption with codec servers for secure debugging, activity timeout tuning based on measured execution times, fire-and-forget abandoned child workflows for non-critical operations, and cross-namespace communication using Temporal Nexus. The system handles 1.8 billion billable actions per month with 600+ registered activity and workflow types, deployed on Kubernetes/AWS with Go.
•33m watch time
158 Impressions