Your job scheduler runs on 3 instances behind an ALB.
Every 5 minutes, a cron fires the same “generate-daily-report” job — and all 3 instances try to grab it.
You reach for the classic fix: a Redis SETNX lock.
SET job:daily-report locked NX EX 300 → only one instance wins. Done.
Works perfectly for a month. Until 2am on a Saturday.
Instance B wins the lock. Starts the job. 60 seconds in, the pod gets OOM-killed. The lock is still there. TTL says 300 seconds. The report runs once every 5 minutes.
Nothing processes that report for 4 more minutes. Then instance A grabs it — but instance B’s pod restarted and it’s ALSO trying to run it now. Both instances processing the same job.
Worse: a slow GC pause on instance A makes it hold the lock past TTL. Redis quietly hands the lock to instance C. Now A thinks it owns the lock. C also thinks it owns the lock. Both write to the same output.
What’s the right pattern here?
A) Redis SETNX + short TTL + fencing token — monotonically increasing token the downstream resource checks on every write.
B) Redlock — acquire the lock across 5 independent Redis nodes, majority wins.
C) DB-based pessimistic lock — SELECT … FOR UPDATE on a jobs table, transaction holds the lock.
D) Optimistic concurrency — no lock. Each instance tries to insert the result with a unique key; DB rejects duplicates.
All four are real patterns engineers genuinely ship. Only one of them actually survives the failure mode above — and one of them is what Martin Kleppmann publicly tore apart in a post that’s still cited 8 years later.
Pick one — A, B, C, or D — and tell me why. Full breakdown in the comments.
If your team has ever argued about Redlock vs SETNX at 3am, send this to them. The debate is the point.
Drop your answer 👇
#30DaysOfSystemDesign #SystemDesign #DistributedSystems #Redis

Heart

Joud Awad

A job scheduler running on 3 instances behind a load balancer uses Redis SETNX to prevent duplicate job execution, but fails under real-world conditions: OOM kills, GC pauses, and lock expiry cause multiple instances to process the same job simultaneously. Four distributed locking patterns are presented for comparison — Redis SETNX with fencing tokens, Redlock, DB-based pessimistic locking, and optimistic concurrency — with a nod to Martin Kleppmann's famous critique of Redlock. Readers are challenged to identify which pattern actually survives these failure modes.

6/30 Day System Design Questions

Answer: A — SETNX + short TTL + fencing token ✅
Here’s why, and why the other three trick smart engineers:
Why A wins (SETNX + TTL + fencing token):
A lock alone can’t protect a distributed resource. You need to assume the lock holder might be dead, paused, or partitioned — and the lock might get reassigned while they still think they own it. This is exactly the scenario above: GC pause, TTL expires, Redis gives the lock to someone else, original owner wakes up and keeps writing.
The fencing token fixes it. Every time a lock is acquired, Redis (or the lock service) hands out a monotonically increasing number — 42, 43, 44. The downstream resource (the DB, the storage system, the output queue) remembers the highest token it’s seen and rejects any write with a lower token.
So when instance A wakes up from its GC pause holding token 42, and instance C is already writing with token 43, A’s write gets rejected by the DB itself. The resource is the source of truth — not the lock service.
This is what Google’s Chubby does. It’s what Martin Kleppmann recommends in “How to do distributed locking.” It’s the only pattern on this list that’s actually safe against process pauses.
<img src="https://media.daily.dev/image/upload/s--5jh2_GDl--/f_auto/v1778519635/ugc/content_1459e0aa-3175-46d1-ac6c-d5221437ff0a?_a=BAMAMiWQ0" alt="ChatGPT Image May 11, 2026, 08_12_22 PM.png">

Why B is the trap answer (Redlock):
Redlock is the pattern most senior engineers reach for. It looks rigorous. Martin Kleppmann wrote a now-famous post (2016) showing Redlock is unsafe in the exact scenario above: any process can pause long enough to lose its lock without noticing.
More importantly — Redlock without a fencing token has the same fundamental flaw as SETNX. If your resource can’t verify a token on write, Redlock doesn’t save you. You’re maintaining 5 Redis clusters for a problem a fencing token solves on 1.

Why C is wrong (DB pessimistic lock):
SELECT … FOR UPDATE inside a transaction works — for tiny critical sections. The lock auto-releases when the transaction ends or the connection dies.
But the lock lives inside a single DB transaction. If your job is “generate a daily report” that takes 60 seconds of S3 writes and external API calls, you’re holding a row lock on your primary DB for 60 seconds per job. Connection pool starved. Every other query blocked behind it.

Why D is wrong here (optimistic concurrency):
Optimistic concurrency is beautiful when it fits — no lock service, no lock holder, no stale-lock problem. Each instance tries the operation; the DB unique constraint rejects losers.
It breaks when the work isn’t idempotent or isn’t a single write. Three instances racing to do all of that means triple the S3 writes, triple the Slack pings, triple the compute cost.

Hey Joud,
Building a hiring sim for devs who hate
LeetCode and whiteboard interviews.
10-min scenario: production DB crashes at 2am, 3,200 users locked out, board demo in 18 min.Real terminal. Real logs. Real pressure.
Would you try it and tell me if it
feels like actual on-call work?
Link: <a href="https://proof-of-chaos.lovable.app/" target="_blank" rel="noopener nofollow">https://proof-of-chaos.lovable.app/</a>