--- title: Excessive IO caused by systemd-journald · Issue #40262 ·... description: A GitHub issue documents excessive disk I/O caused by systemd-journald, where writing a single short log message results in tens of kilobytes of physical disk... canonical: https://daily.dev/posts/excessive-io-caused-by-systemd-journald-issue-40262-systemd-systemd-og4rnygfu twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Excessive IO caused by systemd-journald · Issue #40262 · systemd/systemd | daily.dev og:description: A GitHub issue documents excessive disk I/O caused by systemd-journald, where writing a single short log message results in tens of kilobytes of physical disk... og:url: https://daily.dev/posts/excessive-io-caused-by-systemd-journald-issue-40262-systemd-systemd-og4rnygfu og:image: https://api.daily.dev/og/posts/Og4rNygfu.png og:image:alt: Excessive IO caused by systemd-journald · Issue #40262 · systemd/systemd og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Excessive IO caused by systemd-journald · Issue #40262 · systemd/systemd **[Hacker News](https://daily.dev/sources/hn)** · 13 min read · 0 upvotes · 0 comments ## Summary A GitHub issue documents excessive disk I/O caused by systemd-journald, where writing a single short log message results in tens of kilobytes of physical disk writes, far more than a comparable syslog write to a text file. Users report VMs and desktops experiencing IO spikes up to 100%, with journald logging 'Under memory pressure, flushing caches' and one machine writing nearly 7GB in 15 minutes. Detailed testing with block-stat and cgroup io.stat instrumentation shows journald's mmap-based on-disk format is verbose, unindexed, and writes to persistent storage on every log line rather than batching, with SyncIntervalSec only controlling fsync timing, not the write itself. Commenters, including a self-described kernel developer, argue the fundamental design choice of using mmapped writes instead of pwrite is flawed and question why journald relies on it. Workarounds discussed include disabling compression, switching to volatile storage, and enabling ext4 lazytime/noatime. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does systemd-journald cause so much more disk IO than plain syslog for the same log messages? Journald writes to its mmap-backed persistent journal file on every log entry, not just periodically, because SyncIntervalSec only controls the fdatasync/fsync delay, not when data is actually written to disk. Testing on ext4 showed a single short log entry (about 752 bytes of journal data) produced over 55 KB of physical block writes, versus about 21 KB total for 3 similar syslog appends using fdatasync+fsync, because the on-disk format is verbose, duplicates fields like boot ID per entry, and is not compactly indexed. _Developers tuning logging pipelines can track journald IO debates like this one on daily.dev._ ### How can I reduce disk writes caused by systemd-journald on a system where I don't need persistent logs? Set Storage=volatile in the [Journal] section of journald.conf so logs are kept in memory instead of being written to persistent disk files, eliminating the excessive writes tied to journald's mmap-based persistent storage. Other partial mitigations discussed include disabling compression, though testers found it made no meaningful difference for short log entries. _daily.dev helps sysadmins keep up with practical fixes for logging-related IO issues like this._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 83 comments across hackernews (as of 2026-08-14). **TL;DR:** Commenters largely agree journald's on-disk format causes real write amplification and question the mmap-based design, while a smaller but heated side-thread argues bug reporters shouldn't be expected to submit fixes. **Sentiment:** 5% positive · 25% mixed · 70% skeptical **The case for** - A former journald contributor explains the historical performance and stability motivations behind its design. - Some argue journald's indexed query features via journalctl provide real value over plain text logs for certain use cases. **The pushback** - Multiple commenters call the on-disk format badly designed, citing severe write amplification from small writes crossing block boundaries. - Several argue journald's indexing rarely matters in practice since fast tools like ripgrep make plain-text log scanning perform just as well. - Commenters note journald has historically corrupted logs and scales poorly with many journal files. - Critics say journald should have reused an existing storage engine (SQLite, LMDB, DuckDB, LSM-style) instead of inventing a new format. - Some say journald is hard to filter or scope logging per-service/identifier, causing runaway disk usage from chatty subsystems. - A few argue mmap-based writes plus buffered IO explains the amplification and question why pwrite wasn't used instead. **By community** - hackernews (heated): Strong technical criticism of journald's mmap-based format and write amplification is intertwined with a separate, contentious argument about whether bug reporters owe maintainers a fix. **Hottest debate:** Whether someone filing a performance bug report is obligated to also propose or implement a fix, versus that being the maintainers' job. **Open questions** - Would switching to an LSM-style, append-only, or WAL-based format actually eliminate the write amplification without losing indexing benefits? - Why was mmap chosen for writes instead of pwrite, and would replacing it meaningfully reduce IO? - Is there a viable path to reusing an existing embedded database (SQLite, DuckDB, LMDB) for journald without a costly rewrite? **Highlights** > I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size. Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written. The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem. > — [pengaru on hackernews · 4 comments](https://news.ycombinator.com/item?id=49291376) > Something must have happened along the way, because this was not the original design intent of the database (emphasis mine): """ The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that. """ See https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG... > — [otterley on hackernews · 1 comments](https://news.ycombinator.com/item?id=49292109) > I did, and I still don't get why you would want this monstrosity over a structured append-only log file. If you want to index the data, you can do that when you roll over the file. That way you get the exact same robustness guarantees, without the insane architecture. Like ultimately it isn't even fast, journalctl is so bad at rendering text that it's approximately still as slow as seeking in a 400 MB .log-file using less. Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space. > — [marginalia\_nu on hackernews · 1 comments](https://news.ycombinator.com/item?id=49293041) > If it ever worked like that, then gradual accretion of (mis)features and misguided enhancements pretty clearly broke it. Based on my years and years and years of reading about and using the output of the Systemd Project, there's really clearly no Linus Torvalds on the project to hold the line on software quality. > — [simoncion on hackernews · 1 comments](https://news.ycombinator.com/item?id=49292552) > Because you become responsible for feeding and taking said care of horse and dealing with any technical debt associated with it. If someone submits code to a project that I maintain that's going to make my life difficult in the future, I'm not going to accept it. > — [ericpruitt on hackernews](https://news.ycombinator.com/item?id=49292614) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49290215) · 129 points · 83 comments ## Similar posts on daily.dev - [Your Proxmox boot drive is probably dying faster than you think — here's what's actually happening](https://daily.dev/posts/your-proxmox-boot-drive-is-probably-dying-faster-than-you-think-here-s-what-s-actually-happening-zezi3zo6t) · XDA Developers · 0 upvotes · 0 comments - [Linux 7.3 To Overcome "Significant Bottleneck" For Small I/O With PCIe Gen5 NVMe SSDs](https://daily.dev/posts/linux-7-3-to-overcome-significant-bottleneck-for-small-i-o-with-pcie-gen5-nvme-ssds-sazbtubwn) · Phoronix · 2 upvotes · 0 comments - [Common Performance Pitfalls of Modern Storage I/O](https://daily.dev/posts/common-performance-pitfalls-of-modern-storage-i-o-ucpqaclej) · P99 Conf · 8 upvotes · 0 comments - [Netflix Uncovers Kernel-Level Bottlenecks While Scaling Containers on Modern CPUs](https://daily.dev/posts/netflix-uncovers-kernel-level-bottlenecks-while-scaling-containers-on-modern-cpus-bfs33g6uj) · InfoQ · 3 upvotes · 0 comments --- Tags: [#linux](https://daily.dev/tags/linux), [#performance](https://daily.dev/tags/performance), [#filesystems](https://daily.dev/tags/filesystems) [View this post on daily.dev](https://daily.dev/posts/excessive-io-caused-by-systemd-journald-issue-40262-systemd-systemd-og4rnygfu) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Excessive IO caused by systemd-journald · Issue #40262 · systemd/systemd","url":"https://daily.dev/posts/excessive-io-caused-by-systemd-journald-issue-40262-systemd-systemd-og4rnygfu","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/excessive-io-caused-by-systemd-journald-issue-40262-systemd-systemd-og4rnygfu"},"datePublished":"2026-08-13T20:34:09.371Z","dateModified":"2026-08-14T00:08:55.731Z","description":"A GitHub issue documents excessive disk I/O caused by systemd-journald, where writing a single short log message results in tens of kilobytes of physical disk...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9f3d260fed2534d55c1b49bcab822a21?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9f3d260fed2534d55c1b49bcab822a21?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/excessive-io-caused-by-systemd-journald-issue-40262-systemd-systemd-og4rnygfu","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"linux,performance,filesystems","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Excessive IO caused by systemd-journald · Issue #40262 · systemd/systemd"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/excessive-io-caused-by-systemd-journald-issue-40262-systemd-systemd-og4rnygfu#faq","mainEntity":[{"@type":"Question","name":"Why does systemd-journald cause so much more disk IO than plain syslog for the same log messages?","acceptedAnswer":{"@type":"Answer","text":"Journald writes to its mmap-backed persistent journal file on every log entry, not just periodically, because SyncIntervalSec only controls the fdatasync/fsync delay, not when data is actually written to disk. Testing on ext4 showed a single short log entry (about 752 bytes of journal data) produced over 55 KB of physical block writes, versus about 21 KB total for 3 similar syslog appends using fdatasync+fsync, because the on-disk format is verbose, duplicates fields like boot ID per entry, and is not compactly indexed. Developers tuning logging pipelines can track journald IO debates like this one on daily.dev."}},{"@type":"Question","name":"How can I reduce disk writes caused by systemd-journald on a system where I don't need persistent logs?","acceptedAnswer":{"@type":"Answer","text":"Set Storage=volatile in the [Journal] section of journald.conf so logs are kept in memory instead of being written to persistent disk files, eliminating the excessive writes tied to journald's mmap-based persistent storage. Other partial mitigations discussed include disabling compression, though testers found it made no meaningful difference for short log entries. daily.dev helps sysadmins keep up with practical fixes for logging-related IO issues like this."}}]} ```