--- title: How I Resolved a Critical Loki Disk Space Crisis in... description: A hands-on incident report walks through diagnosing and fixing a Loki disk space crisis in Kubernetes after a persistent volume hit 98% utilization due to... canonical: https://daily.dev/posts/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-enimjfl4i twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How I Resolved a Critical Loki Disk Space Crisis in Kubernetes | daily.dev og:description: A hands-on incident report walks through diagnosing and fixing a Loki disk space crisis in Kubernetes after a persistent volume hit 98% utilization due to... og:url: https://daily.dev/posts/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-enimjfl4i og:image: https://api.daily.dev/og/posts/eNiMJFL4I.png og:image:alt: How I Resolved a Critical Loki Disk Space Crisis in Kubernetes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How I Resolved a Critical Loki Disk Space Crisis in Kubernetes **[Towards Dev](https://daily.dev/sources/towardsdev)** · 3 min read · 2 upvotes · 0 comments ## Summary A hands-on incident report walks through diagnosing and fixing a Loki disk space crisis in Kubernetes after a persistent volume hit 98% utilization due to missing log retention policies. The author scales down the Loki statefulset, uses a debug pod to inspect storage, discovers the chunks directory consuming nearly all disk space, and hits an 'Argument list too long' error when trying to clean up. A restart then reveals corrupted BoltDB index files causing CrashLoopBackOff, which are removed along with the chunk files using find instead of rm. Loki is brought back online successfully, with a promise to cover retention policy configuration in a follow-up. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I fix Loki crashing with CrashLoopBackOff due to corrupted BoltDB index files? Remove the corrupted BoltDB shipper files and restart Loki. Scale the Loki statefulset to zero replicas, use a debug pod with access to the same persistent volume claim, delete the contents of boltdb-shipper-active, boltdb-shipper-cache, and boltdb-shipper-compactor directories, then scale the statefulset back to one replica so Loki can rebuild its index. _Teams running Loki in production can find similar Kubernetes incident writeups on daily.dev before their own disk fills up._ ### Why does rm -rf fail with 'Argument list too long' when clearing Loki chunk files? The chunks directory can contain so many individual files that the shell expands the wildcard into an argument list exceeding the operating system limit, causing rm -rf to fail. The workaround is using find /data/loki/chunks -type f -delete followed by find /data/loki/chunks -type d -empty -delete, which processes files one at a time instead of expanding them all as arguments. _daily.dev surfaces practical Kubernetes storage fixes like this one for engineers debugging full volumes under pressure._ ### Why did Loki fill up its disk to 98% with no retention policy configured? Loki stores compressed log data indefinitely in its chunks directory when no retention policy is set, so ingested logs are never deleted and storage grows unbounded. In one case the chunks directory consumed 48.1GB of a 50GB persistent volume, versus only 25.9MB for the write-ahead log and 4.6MB for the active BoltDB index, eventually triggering a disk space crisis. _Anyone configuring Loki retention policies can track hands-on operational lessons like this via daily.dev._ ## Similar posts on daily.dev - [Your Pod Died of “Out of Ephemeral Storage” but the Disk Is Half Empty\!\!](https://daily.dev/posts/your-pod-died-of-out-of-ephemeral-storage-but-the-disk-is-half-empty--jvrjsmjmg) · ITNEXT · 9 upvotes · 1 comments - [Fixing AKS NodePressure Memory Eviction with Azure Disk CSI: Root Cause & Recovery Guide](https://daily.dev/posts/fixing-aks-nodepressure-memory-eviction-with-azure-disk-csi-root-cause-recovery-guide-jylppuc9r) · Medium · 1 upvotes · 0 comments - [How Do I Fix CrashLoopBackOff in Kubernetes \(Step‑by‑Step\)?](https://daily.dev/posts/how-do-i-fix-crashloopbackoff-in-kubernetes-step-by-step--bemn5k6tc) · Security Boulevard · 1 upvotes · 0 comments - [CrashLoopBackOff in Kubernetes: The Real Causes and How We Fix It](https://daily.dev/posts/crashloopbackoff-in-kubernetes-the-real-causes-and-how-we-fix-it-odapgy48g) · Cast AI · 3 upvotes · 1 comments - [Running out of Disk Space in Production](https://daily.dev/posts/running-out-of-disk-space-in-production-oy7o8gbtw) · Lobsters · 5 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#grafana](https://daily.dev/tags/grafana) [View this post on daily.dev](https://daily.dev/posts/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-enimjfl4i) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How I Resolved a Critical Loki Disk Space Crisis in Kubernetes","url":"https://daily.dev/posts/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-enimjfl4i","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-enimjfl4i"},"datePublished":"2026-08-24T09:36:23.436Z","dateModified":"2026-09-14T07:18:23.247Z","description":"A hands-on incident report walks through diagnosing and fixing a Loki disk space crisis in Kubernetes after a persistent volume hit 98% utilization due to...","image":"https://media.daily.dev/image/upload/s--58gMhC4P--/f_auto/v1722860399/public/Placeholder%2012","thumbnailUrl":"https://media.daily.dev/image/upload/s--58gMhC4P--/f_auto/v1722860399/public/Placeholder%2012","isAccessibleForFree":true,"articleSection":"Towards Dev","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Towards Dev","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/2c710db2750a4a70835862534cb7c9d4","url":"https://daily.dev/sources/towardsdev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-enimjfl4i","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,observability,grafana","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Towards Dev","item":"https://daily.dev/sources/towardsdev"},{"@type":"ListItem","position":3,"name":"How I Resolved a Critical Loki Disk Space Crisis in Kubernetes"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-enimjfl4i#faq","mainEntity":[{"@type":"Question","name":"How do I fix Loki crashing with CrashLoopBackOff due to corrupted BoltDB index files?","acceptedAnswer":{"@type":"Answer","text":"Remove the corrupted BoltDB shipper files and restart Loki. Scale the Loki statefulset to zero replicas, use a debug pod with access to the same persistent volume claim, delete the contents of boltdb-shipper-active, boltdb-shipper-cache, and boltdb-shipper-compactor directories, then scale the statefulset back to one replica so Loki can rebuild its index. Teams running Loki in production can find similar Kubernetes incident writeups on daily.dev before their own disk fills up."}},{"@type":"Question","name":"Why does rm -rf fail with 'Argument list too long' when clearing Loki chunk files?","acceptedAnswer":{"@type":"Answer","text":"The chunks directory can contain so many individual files that the shell expands the wildcard into an argument list exceeding the operating system limit, causing rm -rf to fail. The workaround is using find /data/loki/chunks -type f -delete followed by find /data/loki/chunks -type d -empty -delete, which processes files one at a time instead of expanding them all as arguments. daily.dev surfaces practical Kubernetes storage fixes like this one for engineers debugging full volumes under pressure."}},{"@type":"Question","name":"Why did Loki fill up its disk to 98% with no retention policy configured?","acceptedAnswer":{"@type":"Answer","text":"Loki stores compressed log data indefinitely in its chunks directory when no retention policy is set, so ingested logs are never deleted and storage grows unbounded. In one case the chunks directory consumed 48.1GB of a 50GB persistent volume, versus only 25.9MB for the write-ahead log and 4.6MB for the active BoltDB index, eventually triggering a disk space crisis. Anyone configuring Loki retention policies can track hands-on operational lessons like this via daily.dev."}}]} ```