A bug in Linux kernel 6.18.38 causes inflated disk latency readings in /proc/diskstats. A race condition in blk_time_get_ns() can return 0, which gets stored in rq->start_time_ns. On I/O completion, the kernel adds 'now - 0' to diskstats time fields, effectively adding the system uptime per event. Prometheus interprets these counter jumps as resets and compensates, making the latency graph worsen over time. The bug was introduced by commit fad156c2af22 in 6.18.38 and fixed by commit 691b052139c9 in 6.18.40. A shell one-liner monitoring field 11 of /proc/diskstats can detect the anomalous jumps.
Questions this post answers
Why does my Prometheus disk write latency graph keep climbing over time on Linux but iostat shows normal values?
A race condition in blk_time_get_ns() in Linux kernel 6.18.38 can return 0, which gets stored as the I/O request start time. On completion, the kernel adds 'now - 0' — roughly the system uptime — to the diskstats time fields. Prometheus sees these as counter resets and compensates, making the graph worsen the longer the machine stays up. The bug was fixed in kernel 6.18.40 (commit 691b052139c9). Engineers debugging misleading disk latency metrics track kernel-level issues like this on daily.dev.
How do I detect the kernel 6.18.38 diskstats latency inflation bug in /proc/diskstats?
Monitor field 11 of /proc/diskstats for abnormally large jumps between readings. A shell loop comparing successive values and alerting when the delta exceeds 100,000 ms catches the anomaly: read the field each second, compute the difference, and print an alert if it exceeds the threshold. The bug was introduced in 6.18.38 and fixed in 6.18.40. Teams running self-hosted infrastructure keep up with kernel regressions like this on daily.dev.