--- title: Your Pod Died of “Out of Ephemeral Storage” but the Disk... description: Kubernetes pods can be evicted for 'out of ephemeral storage' even when the node disk appears half empty. Ephemeral storage is an accounting category (writable... canonical: https://daily.dev/posts/your-pod-died-of-out-of-ephemeral-storage-but-the-disk-is-half-empty--jvrjsmjmg twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Your Pod Died of “Out of Ephemeral Storage” but the Disk Is Half Empty!! | daily.dev og:description: Kubernetes pods can be evicted for 'out of ephemeral storage' even when the node disk appears half empty. Ephemeral storage is an accounting category (writable... og:url: https://daily.dev/posts/your-pod-died-of-out-of-ephemeral-storage-but-the-disk-is-half-empty--jvrjsmjmg og:image: https://api.daily.dev/og/posts/jVrJSmJmg.png og:image:alt: Your Pod Died of “Out of Ephemeral Storage” but the Disk Is Half Empty!! og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Your Pod Died of “Out of Ephemeral Storage” but the Disk Is Half Empty!! **[ITNEXT](https://daily.dev/sources/itnext)** · 13 min read · 9 upvotes · 1 comments ## Summary Kubernetes pods can be evicted for 'out of ephemeral storage' even when the node disk appears half empty. Ephemeral storage is an accounting category (writable layer + logs + emptyDir), not a physical wall, and the Kubelet enforces it via a lagging ~10-second scan rather than real-time cgroups. Two root causes explain the paradox: a pod exceeded its own logical limit (unrelated to actual disk fullness), or the node hit a DiskPressure threshold that may have already been cleared by the time you check. The noisy-neighbor problem is especially counterintuitive: Best-Effort pods with no resource requests are evicted first under disk pressure, while pods that reserved large ephemeral-storage requests survive even if they caused the pressure. Practical fixes include setting requests and limits on ephemeral storage, enforcing them cluster-wide with Kyverno or OPA Gatekeeper, enabling log rotation, and using hard-wall solutions like TopoLVM for scratch space. A reproducible minikube lab demonstrates the behavior. The remaining gap is graceful in-place recovery that keeps expensive jobs (AI training, Spark) alive as they approach their budget rather than killing them. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did my Kubernetes pod get evicted for ephemeral storage when df shows plenty of free disk space? Ephemeral storage is an accounting category tracked by the kubelet, not a real disk partition, so df on the node is irrelevant to it. Eviction happens either because the pod exceeded its own limits.ephemeral-storage (a logical budget decoupled from the physical disk) or because the node crossed its DiskPressure threshold, which by default triggers at less than 10% free space, well before the disk looks full. _Debugging these eviction mysteries gets easier when you can quickly surface deep k8s internals explainers like this one on daily.dev._ ### Why do Best-Effort Kubernetes pods with no resource requests get evicted before the pod actually causing disk pressure? Under node DiskPressure, the kubelet evicts pods in order of those exceeding their ephemeral-storage request first, then by priority, then by how far over their request they are. A Best-Effort pod with no requests has an ephemeral-storage request of zero, so any usage puts it in the first eviction bucket, while a pod that reserved a large request (even if it caused the pressure) stays protected as long as its usage is under that reservation. _Understanding this eviction ranking helps teams sizing requests correctly, a topic worth tracking on daily.dev alongside other kubernetes reliability deep dives._ ### How can I stop noisy neighbor pods from causing disk pressure evictions on Kubernetes nodes? Set requests.ephemeral-storage on important pods so they're ranked within budget and protected, and set limits.ephemeral-storage on risky pods so they get evicted for exceeding their own budget before filling the node. Enforce these settings cluster-wide with Kyverno or OPA Gatekeeper, enable log rotation via containerLogMaxSize or Fluent Bit, and use a capacity-enforcing CSI driver like TopoLVM for a true hard wall, since emptyDir.sizeLimit is still eviction-based rather than a hard cap. _Teams hardening GPU or AI training clusters against wasted compute can keep these mitigation patterns bookmarked on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@wolfsrudel** · 0 upvotes > inodes can be monsters. ## Similar posts on daily.dev - [Fixing AKS NodePressure Memory Eviction with Azure Disk CSI: Root Cause & Recovery Guide](https://daily.dev/posts/fixing-aks-nodepressure-memory-eviction-with-azure-disk-csi-root-cause-recovery-guide-jylppuc9r) · Medium · 1 upvotes · 0 comments - [Designing for Ephemeral Storage: Why Kubernetes Uses Disk-Based Merge Sort to Explain emptyDir](https://daily.dev/posts/designing-for-ephemeral-storage-why-kubernetes-uses-disk-based-merge-sort-to-explain-emptydir-kohqaaoko) · Medium · 1 upvotes · 0 comments - [Kubelet watches inodes. Just not until it’s an emergency.](https://daily.dev/posts/kubelet-watches-inodes-just-not-until-it-s-an-emergency--zsh9rqokg) · CNCF · 2 upvotes · 0 comments - [How I Resolved a Critical Loki Disk Space Crisis in Kubernetes](https://daily.dev/posts/how-i-resolved-a-critical-loki-disk-space-crisis-in-kubernetes-enimjfl4i) · Towards Dev · 2 upvotes · 0 comments - [OOMKilled and Exit Code 137: Why Kubernetes Kills Your Pods and How to Stop It](https://daily.dev/posts/oomkilled-and-exit-code-137-why-kubernetes-kills-your-pods-and-how-to-stop-it-y4q54mlwx) · Cast AI · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes) [View this post on daily.dev](https://daily.dev/posts/your-pod-died-of-out-of-ephemeral-storage-but-the-disk-is-half-empty--jvrjsmjmg) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Your Pod Died of “Out of Ephemeral Storage” but the Disk Is Half Empty!!","url":"https://daily.dev/posts/your-pod-died-of-out-of-ephemeral-storage-but-the-disk-is-half-empty--jvrjsmjmg","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/your-pod-died-of-out-of-ephemeral-storage-but-the-disk-is-half-empty--jvrjsmjmg"},"datePublished":"2026-07-30T17:46:18.414Z","dateModified":"2026-09-14T08:22:20.268Z","description":"Kubernetes pods can be evicted for 'out of ephemeral storage' even when the node disk appears half empty. Ephemeral storage is an accounting category (writable...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fb31e5441aee5bdb4b0981b95558cc06?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fb31e5441aee5bdb4b0981b95558cc06?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ITNEXT","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ITNEXT","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/itnext2","url":"https://daily.dev/sources/itnext"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/your-pod-died-of-out-of-ephemeral-storage-but-the-disk-is-half-empty--jvrjsmjmg","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":9},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"kubernetes","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ITNEXT","item":"https://daily.dev/sources/itnext"},{"@type":"ListItem","position":3,"name":"Your Pod Died of “Out of Ephemeral Storage” but the Disk Is Half Empty!!"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/your-pod-died-of-out-of-ephemeral-storage-but-the-disk-is-half-empty--jvrjsmjmg","comment":[{"@type":"Comment","text":"inodes can be monsters.","datePublished":"2026-08-01T09:07:28.842Z","url":"https://daily.dev/posts/jVrJSmJmg#c-8ZelvPzNo","author":{"@type":"Person","name":"Tobias Wolf","url":"https://daily.dev/wolfsrudel","image":"https://media.daily.dev/image/upload/f_auto/v1653726290/avatars/RGsblgUil29AlCPKEbd0g"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/your-pod-died-of-out-of-ephemeral-storage-but-the-disk-is-half-empty--jvrjsmjmg#faq","mainEntity":[{"@type":"Question","name":"Why did my Kubernetes pod get evicted for ephemeral storage when df shows plenty of free disk space?","acceptedAnswer":{"@type":"Answer","text":"Ephemeral storage is an accounting category tracked by the kubelet, not a real disk partition, so df on the node is irrelevant to it. Eviction happens either because the pod exceeded its own limits.ephemeral-storage (a logical budget decoupled from the physical disk) or because the node crossed its DiskPressure threshold, which by default triggers at less than 10% free space, well before the disk looks full. Debugging these eviction mysteries gets easier when you can quickly surface deep k8s internals explainers like this one on daily.dev."}},{"@type":"Question","name":"Why do Best-Effort Kubernetes pods with no resource requests get evicted before the pod actually causing disk pressure?","acceptedAnswer":{"@type":"Answer","text":"Under node DiskPressure, the kubelet evicts pods in order of those exceeding their ephemeral-storage request first, then by priority, then by how far over their request they are. A Best-Effort pod with no requests has an ephemeral-storage request of zero, so any usage puts it in the first eviction bucket, while a pod that reserved a large request (even if it caused the pressure) stays protected as long as its usage is under that reservation. Understanding this eviction ranking helps teams sizing requests correctly, a topic worth tracking on daily.dev alongside other kubernetes reliability deep dives."}},{"@type":"Question","name":"How can I stop noisy neighbor pods from causing disk pressure evictions on Kubernetes nodes?","acceptedAnswer":{"@type":"Answer","text":"Set requests.ephemeral-storage on important pods so they're ranked within budget and protected, and set limits.ephemeral-storage on risky pods so they get evicted for exceeding their own budget before filling the node. Enforce these settings cluster-wide with Kyverno or OPA Gatekeeper, enable log rotation via containerLogMaxSize or Fluent Bit, and use a capacity-enforcing CSI driver like TopoLVM for a true hard wall, since emptyDir.sizeLimit is still eviction-based rather than a hard cap. Teams hardening GPU or AI training clusters against wasted compute can keep these mitigation patterns bookmarked on daily.dev."}}]} ```