The Secrets Behind Plumber: Managing Thousands of Pipelines at Semaphore
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Semaphore's Plumber engine orchestrates CI/CD pipelines using a finite state machine architecture built in Elixir. The system manages pipeline execution through distinct states (initializing, pending, running, done) with dedicated Looper processes handling each state. Key components include PostgreSQL for transactional integrity, RabbitMQ for service communication, and Kubernetes for deployment. The architecture provides fault tolerance through supervision trees, recovery mechanisms for stuck pipelines, and horizontal scalability across distributed agents.
Table of contents
Understanding Semaphore’s Pipeline ModelFrom Git Push to Pipeline ExecutionWhy a State Machine?The Tech StackHandling Failures GracefullyExtending Pipeline BehaviorStopping Pipelines ManuallyQueuing PipelinesRecap: Why It WorksLearn More20 Impressions