Ubuntu
Read post

Tracing a memory leak bug in PID 1 and contributing an upstream fix: a Linux support story

A global retail company's Ceph cluster nodes were experiencing out-of-memory kills, with PID 1 (systemd) consuming ~70 GB of RAM instead of the expected 8 GB. Canonical Support engineers traced the root cause across three system layers: a cephadm orchestration misconfiguration was repeatedly attempting to 'zap' a non-existent OSD, causing a ceph-volume Docker container to spin up and tear down in a continuous loop. This generated millions of mount table entries, triggering a kernel race condition that caused systemd to continuously reparse the mount table while glibc's allocator retained the memory without returning it to the kernel. The fix involved correcting the cephadm loop as a workaround, then isolating a minimal reproducer (mounting/unmounting tmpfs in a tight loop) to characterize the kernel race condition precisely. A patch was merged into the mainline Linux kernel and subsequently landed in several Ubuntu kernel releases via the Stable Release Update process.

    #linux
Jul 17•6m read time•From ubuntu.com
Post cover image
Table of contents
The symptom that made no senseStarting the investigation: getting the right artifactThe diagnostic leap: from memory to mountsThe second leap: a silent failure nobody knew aboutFrom diagnosis to upstreamWhy this case stands out
3.5K Impressions
Ubuntu's image
Ubuntu

The Ubuntu Blog provides updates, tutorials, and insights on the Ubuntu operating system and related...

461 Followers

•

641 Upvotes

Would you recommend this post?

Copy link
WhatsApp
Facebook
X
New Squad
  • © 2026 Daily Dev Ltd.
  • Guidelines
  • Explore
  • Tags
  • Sources
  • Squads
  • Leaderboard