Neon's engineering team traced a severe cold start latency regression (up to 10s p99) to their Pageserver's walredo process lifecycle changes. When idle walredo sidecar processes were shut down after 200s to save DRAM, restarting them became expensive. The root cause turned out not to be fork+exec vs posix_spawn, but rather the close_fds Rust crate falling back to iterating over hundreds of thousands of open file descriptors with individual fcntl calls — because the CLOSE_RANGE_CLOEXEC flag requires Linux 5.11 but production ran 5.10. The fix moved file descriptor closing into the walredo subprocess itself using a direct close_range syscall with flags=0, eliminating the need for pre_exec and allowing posix_spawn to be used, reducing cold start latency from ~2s mean / ~10s p99 to low hundreds of milliseconds.

14m read timeFrom neon.com
Post cover image
Table of contents
Background: Neon PageserverBackground: Role Of WAL in PostgresBackground: walredo in PageserverThe Regression: Changes To The walredo Process LifecycleDiscoveryThe Root CauseThe FixFix ImpactWhy Is It Faster, Though?Summary & Lessons LearnedAppendix