5 Failures Your Service Should Survive (Test Each in 5 Minutes)
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Five common dependency failure modes that most services never get properly tested against are broken down: a slow (not down) cache defeating fallback logic, a flaky dependency amplified by retries or circuit breakers, a client timeout that fires before the server finishes work (causing duplicate side effects), a dependency that slows down only under sustained load and silently backs up a queue, and a fully-down non-critical dependency that isn't properly isolated. Each failure includes a mirrord Chaos Testing JSON rule to simulate it in minutes, plus guidance on what to check for.
Table of contents
1. Your cache is slow but not down #2. Your dependency is flaky, not fully down #3. Your request failed and the work still happened #4. Your dependency slows down under sustained load #5. Your dependency is completely down #Frequently asked chaos testing questions #None of this requires much setup #Questions this post answers
Why is a slow cache sometimes worse than a completely down cache?
A slow cache can be worse than a down cache because with a timeout configured, a request pays the full timeout wait and then still falls back to the database, instead of failing fast and going straight to the database as happens when the cache is simply unreachable. Without any timeout at all, a slow cache makes every request as slow as the cache itself, defeating the purpose of caching. Teams debugging cache latency issues can find deeper resilience patterns like this on daily.dev.
How can a failed request still cause duplicate side effects like double inventory reservations?
This happens when a client's timeout fires before the server finishes processing, so the client sees an error and retries, but the original request actually completes and commits on the server side anyway. For example, an inventory reservation that takes longer than the client's timeout can succeed twice, reserving the same stock item twice while the customer only sees one error. The fix is an idempotency key, not a longer timeout. Anyone hardening checkout or payment flows against duplicate writes can track patterns like this on daily.dev.
Why can a message queue backlog grow silently even when checkout still looks healthy?
A queue backlog can grow invisibly because publishing to a queue like RabbitMQ stays fast regardless of how backed up it is, so the producing service (like checkout) never waits on the slow part. If the consumer's database writes slow down, say from 100ms to a full second, the consumer can't keep up with incoming messages, the backlog keeps growing, and if unbounded, it can eventually exhaust memory and crash even though checkout still looks fine. Engineers designing queue-based systems can follow throughput failure patterns like this on daily.dev.