---
title: "How I Resolved a Critical Loki Disk Space Crisis in Kubernetes"
url: https://daily.dev/posts/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-enimjfl4i
source_url: https://towardsdev.com/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-809f2d0b0acd
type: article
source: "Towards Dev"
published: 2026-08-24T09:36:23.436Z
updated: 2026-08-24T09:36:44.901Z
tags: ["kubernetes", "observability", "grafana"]
reading_time: 3
upvotes: 2
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# How I Resolved a Critical Loki Disk Space Crisis in Kubernetes

**[Towards Dev](https://daily.dev/sources/towardsdev)** · 3 min read · 2 upvotes · 0 comments

## Summary

A hands-on incident report walks through diagnosing and fixing a Loki disk space crisis in Kubernetes after a persistent volume hit 98% utilization due to missing log retention policies. The author scales down the Loki statefulset, uses a debug pod to inspect storage, discovers the chunks directory consuming nearly all disk space, and hits an 'Argument list too long' error when trying to clean up. A restart then reveals corrupted BoltDB index files causing CrashLoopBackOff, which are removed along with the chunk files using find instead of rm. Loki is brought back online successfully, with a promise to cover retention policy configuration in a follow-up.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://towardsdev.com/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-809f2d0b0acd>

## Questions this post answers

### What causes a Loki pod to enter CrashLoopBackOff with a 'file size too small' error?

Corrupted BoltDB index files in the boltdb-shipper-active directory cause this error and put the Loki pod into CrashLoopBackOff. The fix is deleting the corrupted files in boltdb-shipper-active, boltdb-shipper-cache, and boltdb-shipper-compactor directories, then restarting the statefulset, after which Loki starts successfully again.

_daily.dev surfaces real-world Kubernetes incident fixes like this for engineers troubleshooting Loki crashes._

### How do I delete a huge number of files in a directory when rm gives 'Argument list too long' in Kubernetes?

Use find with -delete instead of rm -rf, since find handles large file counts without hitting the shell's argument list limit. For example, run find /data/loki/chunks -type f -delete followed by find /data/loki/chunks -type d -empty -delete to clean up both files and now-empty directories.

_Engineers debugging disk space issues in production track practical fixes like this through daily.dev._

### Why does a Loki persistent volume fill up disk space over time in Kubernetes?

Without a configured retention policy, Loki ingests logs into the chunks directory indefinitely and never deletes old data, eventually filling the persistent volume. In one case the chunks directory consumed 48.1GB of a 50GB volume, pushing disk usage to 98% and triggering a crisis requiring manual cleanup.

_Teams running Loki in Kubernetes can follow storage and retention gotchas like this via daily.dev._

## Similar posts on daily.dev

- [Your Pod Died of “Out of Ephemeral Storage” but the Disk Is Half Empty\!\!](https://daily.dev/posts/your-pod-died-of-out-of-ephemeral-storage-but-the-disk-is-half-empty--jvrjsmjmg) · ITNEXT · 9 upvotes · 1 comments
- [Fixing AKS NodePressure Memory Eviction with Azure Disk CSI: Root Cause & Recovery Guide](https://daily.dev/posts/fixing-aks-nodepressure-memory-eviction-with-azure-disk-csi-root-cause-recovery-guide-jylppuc9r) · Medium · 1 upvotes · 0 comments

---

Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#grafana](https://daily.dev/tags/grafana)

[View this post on daily.dev](https://daily.dev/posts/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-enimjfl4i)