OOMKilled (exit code 137) occurs when the Linux kernel's OOM killer terminates a container for exceeding its cgroup memory limit — Kubernetes only reports it, not enforces it. Three distinct scenarios produce OOMKilled: container limit breach, kubelet eviction under node memory pressure, and node-level kernel OOM. Diagnosis involves kubectl describe, previous container logs, and Prometheus queries using container_memory_working_set_bytes (not container_memory_usage_bytes, which includes reclaimable page cache). The correct fix is per-container rightsizing: set requests at p95 of working set over 7 days, limits at p99 or 1.5–2x requests. QoS class affects kill priority — Guaranteed pods are last to be killed. JVM containers need -Xmx capped at 75% of the container limit to account for off-heap allocation. At scale, automated rightsizing tools that track PSI metrics can proactively adjust limits before OOM kills occur.

10m read timeFrom cast.ai
Post cover image
Table of contents
What does OOMKilled (exit code 137) mean?Why Kubernetes kills a pod with exit code 137How to diagnose OOMKilled (kubectl describe, events, logs)How to fix OOMKilledHow to prevent OOM kills at scale (automated rightsizing)FAQ
454 Impressions