--- title: "5 Failures Your Service Should Survive (Test Each in 5 Minutes)" url: https://daily.dev/posts/5-failures-your-service-should-survive-test-each-in-5-minutes--u2fiq1o4q source_url: https://metalbear.com/blog/chaos-testing-failure-modes type: article source: "MetalBear" published: 2026-08-26T12:15:31.307Z updated: 2026-08-26T12:15:59.975Z tags: ["kubernetes", "distributed-systems"] reading_time: 9 upvotes: 2 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # 5 Failures Your Service Should Survive (Test Each in 5 Minutes) **[MetalBear](https://daily.dev/sources/metalbear)** · 9 min read · 2 upvotes · 0 comments ## Summary Five common dependency failure modes that most services never get properly tested against are broken down: a slow (not down) cache defeating fallback logic, a flaky dependency amplified by retries or circuit breakers, a client timeout that fires before the server finishes work (causing duplicate side effects), a dependency that slows down only under sustained load and silently backs up a queue, and a fully-down non-critical dependency that isn't properly isolated. Each failure includes a mirrord Chaos Testing JSON rule to simulate it in minutes, plus guidance on what to check for. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why is a slow cache sometimes worse than a completely down cache? A slow cache can be worse than a down cache because with a timeout configured, a request pays the full timeout wait and then still falls back to the database, instead of failing fast and going straight to the database as happens when the cache is simply unreachable. Without any timeout at all, a slow cache makes every request as slow as the cache itself, defeating the purpose of caching. _Teams debugging cache latency issues can find deeper resilience patterns like this on daily.dev._ ### How can a failed request still cause duplicate side effects like double inventory reservations? This happens when a client's timeout fires before the server finishes processing, so the client sees an error and retries, but the original request actually completes and commits on the server side anyway. For example, an inventory reservation that takes longer than the client's timeout can succeed twice, reserving the same stock item twice while the customer only sees one error. The fix is an idempotency key, not a longer timeout. _Anyone hardening checkout or payment flows against duplicate writes can track patterns like this on daily.dev._ ### Why can a message queue backlog grow silently even when checkout still looks healthy? A queue backlog can grow invisibly because publishing to a queue like RabbitMQ stays fast regardless of how backed up it is, so the producing service (like checkout) never waits on the slow part. If the consumer's database writes slow down, say from 100ms to a full second, the consumer can't keep up with incoming messages, the backlog keeps growing, and if unbounded, it can eventually exhaust memory and crash even though checkout still looks fine. _Engineers designing queue-based systems can follow throughput failure patterns like this on daily.dev._ ## Similar posts on daily.dev - [3 Resilience Patterns That Keep a Backend Service Up When Its Dependencies Aren't](https://daily.dev/posts/3-resilience-patterns-that-keep-a-backend-service-up-when-its-dependencies-aren-t-by1axueza) · The T-Shaped Dev · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/5-failures-your-service-should-survive-test-each-in-5-minutes--u2fiq1o4q)