One Thread to Poll Them All: How a Single Pipe Made WaterDrop 50% Faster
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
WaterDrop, the Ruby Kafka producer library, introduces a new file descriptor-based polling mode that delivers 39–54% throughput improvements. Instead of spawning a dedicated background thread per producer (each competing for Ruby's GVL), a single Poller thread uses `IO.select` to multiplex OS pipe file descriptors across all producers. librdkafka's `rd_kafka_queue_io_event_enable()` API writes a byte to a pipe whenever its event queue transitions from empty to non-empty, waking the Poller only when real work exists. Combined with the non-blocking `poll_nb` variant that avoids GVL overhead, 25 producers now share one thread instead of 25. The feature is opt-in in WaterDrop 2.8 (`config.polling.mode = :fd`), becomes the default in 2.9, and replaces thread mode entirely in 2.10. Trade-offs include shared callback execution on the Poller thread, with dedicated Poller instances available for isolation.
