A hands-on incident report walks through diagnosing and fixing a Loki disk space crisis in Kubernetes after a persistent volume hit 98% utilization due to missing log retention policies. The author scales down the Loki statefulset, uses a debug pod to inspect storage, discovers the chunks directory consuming nearly all disk space, and hits an 'Argument list too long' error when trying to clean up. A restart then reveals corrupted BoltDB index files causing CrashLoopBackOff, which are removed along with the chunk files using find instead of rm. Loki is brought back online successfully, with a promise to cover retention policy configuration in a follow-up.
Table of contents
Get Vamsi Koduri ’s stories in your inboxGetting Hands-On with Debug PodsIdentifying the CulpritUnderstanding Loki’s Storage ArchitectureThe Failed Cleanup AttemptSummary of what was fixed:Questions this post answers
What causes a Loki pod to enter CrashLoopBackOff with a 'file size too small' error?
Corrupted BoltDB index files in the boltdb-shipper-active directory cause this error and put the Loki pod into CrashLoopBackOff. The fix is deleting the corrupted files in boltdb-shipper-active, boltdb-shipper-cache, and boltdb-shipper-compactor directories, then restarting the statefulset, after which Loki starts successfully again. daily.dev surfaces real-world Kubernetes incident fixes like this for engineers troubleshooting Loki crashes.
How do I delete a huge number of files in a directory when rm gives 'Argument list too long' in Kubernetes?
Use find with -delete instead of rm -rf, since find handles large file counts without hitting the shell's argument list limit. For example, run find /data/loki/chunks -type f -delete followed by find /data/loki/chunks -type d -empty -delete to clean up both files and now-empty directories. Engineers debugging disk space issues in production track practical fixes like this through daily.dev.
Why does a Loki persistent volume fill up disk space over time in Kubernetes?
Without a configured retention policy, Loki ingests logs into the chunks directory indefinitely and never deletes old data, eventually filling the persistent volume. In one case the chunks directory consumed 48.1GB of a 50GB volume, pushing disk usage to 98% and triggering a crisis requiring manual cleanup. Teams running Loki in Kubernetes can follow storage and retention gotchas like this via daily.dev.