A deep dive into timeouts as a resilience pattern in distributed systems: what a timeout actually bounds, how to derive a sane value from real latency percentiles, the difference between per-attempt and overall budgets, how deadlines must propagate across service hops (unlike restarting timeouts), and why a timeout never proves work didn't happen — meaning any retry behind a timeout requires idempotency. Includes a real incident from a radiology imaging platform where an unconfigured 30-second default timeout caused a broad outage, and a .NET/Polly code example composing overall, retry, and per-attempt timeout layers.
Table of contents
What is a Timeout?ExamplesProblems addressed by TimeoutsCommon Use Case — The Call That Never Came BackChoosing the NumberPer-Attempt vs Overall TimeoutQuestions this post answers
What is the difference between a per-attempt timeout and an overall timeout in a retry policy?
A per-attempt timeout bounds a single call, while an overall timeout bounds the entire operation including all retries and pauses between them. They must both be set because the real worst-case latency is per-attempt timeout multiplied by the number of attempts, plus backoff delays — for example, a 2-second per-attempt timeout with 3 retries and exponential backoff of 1 and 2 seconds produces a 9-second worst case, not 2 seconds. daily.dev surfaces practical resilience patterns like this for engineers tuning retry and timeout policies.
What is the difference between a timeout and a deadline when a request passes through multiple services?
A timeout is a duration that restarts fresh at each hop ("two seconds from here"), while a deadline is one fixed instant the entire request chain shares ("stop at 12:00:02"). Without deadline propagation, a service three hops deep can start a multi-second query for a caller that already gave up, wasting CPU and holding connections for work nobody will read. engineers designing multi-hop service chains can track patterns like deadline propagation through daily.dev.
Why is it unsafe to automatically retry an HTTP call after it times out?
A timeout only tells you that no response arrived in time — it does not tell you whether the server actually completed the operation, such as charging a card or writing a database row. Retrying a timed-out call that actually succeeded means running the operation twice, so any retry behind a timeout requires the operation to be idempotent, typically via a unique request key the server uses to detect duplicates. daily.dev helps developers building idempotent, retry-safe APIs keep up with patterns like this.