A hands-on walkthrough shows how to use mirrord's chaos testing feature to inject latency and connection errors into a single developer's session against a shared staging cluster, without affecting anyone else on the cluster. Using a small loan-application demo app with two outgoing calls (one with a timeout and retry, one without), the guide demonstrates how faults like injected latency, connection resets, and partial (percentage-based) faults expose a missing timeout bug in one service. It then shows fixing the bug (adding a timeout and a manual-review fallback for unknown fraud risk) while the fault stays applied, and shows how faults can be moved one hop deeper onto a dependency's own dependency (e.g., a Postgres call). Requires mirrord 3.250.0 or newer.
Table of contents
The App, and the Difference Between Its Two Outgoing Calls #Running One Service Against the Cluster #A Dependency That Slows Down, But Not Past the Timeout #What Everyone Else Sees #The Call With No Timeout #Fixing It, With the Fault Still Applied #Moving the Fault One Hop Deeper #Trying It #Questions this post answers
How do I use mirrord to test what happens when a dependency service goes down or gets slow, without affecting other developers on the shared cluster?
mirrord chaos scopes fault injection to a single session, so you add a rule (via mirrord chaos add -s $SESSION_ID -f rule.json) targeting an upstream selector, and only requests carrying your session's baggage header hit the fault. Effects include latency (read_ms/write_ms) and connection_error (e.g., reset), with a percentage field to apply the fault to only some connections. Requires mirrord 3.250.0 or newer. See how daily.dev surfaces practical walkthroughs like this for evaluating chaos testing tools.
What is the difference between adding a timeout to an outgoing fetch call and leaving it unbounded, in terms of how the app behaves when a dependency hangs?
An unbounded fetch (no AbortSignal.timeout) waits as long as the fault or dependency takes, in one demonstrated case 40 seconds, with no fallback behavior. Adding AbortSignal.timeout with a bound (e.g., 2 seconds) plus a defined fallback (reporting risk as 'unknown' and routing to manual review) means requests abort at the bound and retry, turning a multi-second hang into a bounded, predictable outcome around 2-4 seconds. Developers hardening service timeouts can track patterns like this on daily.dev before shipping.
Why does an intermittent fault affecting only 30% of requests to a dependency sometimes pass a smoke test even though the service has a real reliability bug?
Because most requests through a partial fault succeed by chance, a smoke test with only a few requests is likely to see all-passing results even when roughly a third of connections would hit multi-second latency. In one demonstrated run, 12 sequential requests to a fraud-check dependency with 30% latency injection showed 4 out of 12 hanging for 5-8 seconds, a pattern a single smoke test could easily miss. daily.dev helps engineers stay on top of testing techniques that catch intermittent failures early.