--- title: Building a Reliable PostgreSQL Queue: Concurrency,... description: A deep technical walkthrough builds a PostgreSQL-backed background job queue from scratch, exposing failure modes as they arise: race conditions solved with... canonical: https://daily.dev/posts/building-a-reliable-postgresql-queue-concurrency-crashes-retries-and-scale-master-dev-blog-0yuhzwdif twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Building a Reliable PostgreSQL Queue: Concurrency, Crashes, Retries, and Scale – Master.dev Blog | daily.dev og:description: A deep technical walkthrough builds a PostgreSQL-backed background job queue from scratch, exposing failure modes as they arise: race conditions solved with... og:url: https://daily.dev/posts/building-a-reliable-postgresql-queue-concurrency-crashes-retries-and-scale-master-dev-blog-0yuhzwdif og:image: https://api.daily.dev/og/posts/0yUHzWDIF.png og:image:alt: Building a Reliable PostgreSQL Queue: Concurrency, Crashes, Retries, and Scale – Master.dev Blog og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Building a Reliable PostgreSQL Queue: Concurrency, Crashes, Retries, and Scale – Master.dev Blog **[Frontend Masters](https://daily.dev/sources/frontendmasters)** · 14 min read · 1 upvotes · 0 comments ## Summary A deep technical walkthrough builds a PostgreSQL-backed background job queue from scratch, exposing failure modes as they arise: race conditions solved with FOR UPDATE SKIP LOCKED, worker crashes handled via leases and fencing tokens, idempotency requirements for external side effects, retry storms mitigated with exponential backoff plus jitter, and a three-table schema (jobs, state, attempts) to reduce hot-row bloat. A benchmarking deep dive reveals that using `queue = ANY(array)` in the claim query forces PostgreSQL into a bitmap scan and sort, making claims 9.8x slower than an equality predicate (`queue = $1`) at 100K pending jobs, leading to a redesign where each claim targets a single queue name. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why is my PostgreSQL job queue claim query slow when using queue = ANY(array) with multiple queue names Using queue = ANY(array) forces PostgreSQL to scan several index ranges instead of walking one contiguous ordered index range, so it collects a much larger candidate set and sorts it before applying LIMIT. In one benchmark against 1M total rows with 100K pending, this pushed execution time to 51.85ms versus 5.307ms (9.8x faster) when the predicate was changed to an equality check on a single queue value, which allowed an ordered top-N index scan. _daily.dev surfaces performance deep-dives like this for engineers tuning claim queries under real load._ ### how do I prevent duplicate job processing when multiple workers poll the same PostgreSQL queue table Use SELECT ... FOR UPDATE SKIP LOCKED when claiming pending rows so that locked rows are skipped rather than waited on, letting workers claim distinct jobs in parallel with no duplicate claims. A naive SELECT without locking let 4 workers grab the same row simultaneously, producing 24 duplicate executions across 8 jobs; a plain FOR UPDATE lock avoided duplicates but serialized workers behind the same row instead of moving to other available rows. _track concurrency-safe queue patterns like this on daily.dev while designing your own distributed job workers._ ### how do fencing tokens prevent a crashed worker from overwriting another worker's job status update in a lease-based queue A fencing token, implemented as a lease_version integer column, detects stale writes from a worker that wakes up after its lease has already expired and been reclaimed. When the original worker's stale update runs, its lease_version no longer matches the current ownership generation, so the conditional UPDATE matches zero rows and the stale write is rejected, protecting against last-write-wins corruption. _daily.dev keeps distributed-systems patterns like fencing tokens close at hand for engineers building crash-safe workers._ ## Similar posts on daily.dev - [I Tried the “Just Use Postgres” Queue for Six Weeks. SKIP LOCKED Paid Until It Didn’t](https://daily.dev/posts/i-tried-the-just-use-postgres-queue-for-six-weeks-skip-locked-paid-until-it-didn-t-sni9qhs1s) · Medium · 2 upvotes · 0 comments - [Potential Consequences of Using Postgres as a Job Queue](https://daily.dev/posts/potential-consequences-of-using-postgres-as-a-job-queue-kcv0c9q1v) · Lobsters · 70 upvotes · 0 comments - [Making Postgres queues scale](https://daily.dev/posts/making-postgres-queues-scale-otqspthk6) · Hacker News · 1 upvotes · 0 comments --- Tags: [#database](https://daily.dev/tags/database), [#postgresql](https://daily.dev/tags/postgresql), [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/building-a-reliable-postgresql-queue-concurrency-crashes-retries-and-scale-master-dev-blog-0yuhzwdif) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Building a Reliable PostgreSQL Queue: Concurrency, Crashes, Retries, and Scale – Master.dev Blog","url":"https://daily.dev/posts/building-a-reliable-postgresql-queue-concurrency-crashes-retries-and-scale-master-dev-blog-0yuhzwdif","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/building-a-reliable-postgresql-queue-concurrency-crashes-retries-and-scale-master-dev-blog-0yuhzwdif"},"datePublished":"2026-09-09T15:33:07.853Z","dateModified":"2026-09-14T06:08:54.526Z","description":"A deep technical walkthrough builds a PostgreSQL-backed background job queue from scratch, exposing failure modes as they arise: race conditions solved with...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/dec1f4ee0421a6dafeb0f877d56953fc?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/dec1f4ee0421a6dafeb0f877d56953fc?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Frontend Masters","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Frontend Masters","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/d34dd87208f24090bfb0917307fcf89f","url":"https://daily.dev/sources/frontendmasters"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/building-a-reliable-postgresql-queue-concurrency-crashes-retries-and-scale-master-dev-blog-0yuhzwdif","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"database,postgresql,distributed-systems","timeRequired":"PT14M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Frontend Masters","item":"https://daily.dev/sources/frontendmasters"},{"@type":"ListItem","position":3,"name":"Building a Reliable PostgreSQL Queue: Concurrency, Crashes, Retries, and Scale – Master.dev Blog"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/building-a-reliable-postgresql-queue-concurrency-crashes-retries-and-scale-master-dev-blog-0yuhzwdif#faq","mainEntity":[{"@type":"Question","name":"why is my PostgreSQL job queue claim query slow when using queue = ANY(array) with multiple queue names","acceptedAnswer":{"@type":"Answer","text":"Using queue = ANY(array) forces PostgreSQL to scan several index ranges instead of walking one contiguous ordered index range, so it collects a much larger candidate set and sorts it before applying LIMIT. In one benchmark against 1M total rows with 100K pending, this pushed execution time to 51.85ms versus 5.307ms (9.8x faster) when the predicate was changed to an equality check on a single queue value, which allowed an ordered top-N index scan. daily.dev surfaces performance deep-dives like this for engineers tuning claim queries under real load."}},{"@type":"Question","name":"how do I prevent duplicate job processing when multiple workers poll the same PostgreSQL queue table","acceptedAnswer":{"@type":"Answer","text":"Use SELECT ... FOR UPDATE SKIP LOCKED when claiming pending rows so that locked rows are skipped rather than waited on, letting workers claim distinct jobs in parallel with no duplicate claims. A naive SELECT without locking let 4 workers grab the same row simultaneously, producing 24 duplicate executions across 8 jobs; a plain FOR UPDATE lock avoided duplicates but serialized workers behind the same row instead of moving to other available rows. track concurrency-safe queue patterns like this on daily.dev while designing your own distributed job workers."}},{"@type":"Question","name":"how do fencing tokens prevent a crashed worker from overwriting another worker's job status update in a lease-based queue","acceptedAnswer":{"@type":"Answer","text":"A fencing token, implemented as a lease_version integer column, detects stale writes from a worker that wakes up after its lease has already expired and been reclaimed. When the original worker's stale update runs, its lease_version no longer matches the current ownership generation, so the conditional UPDATE matches zero rows and the stale write is rejected, protecting against last-write-wins corruption. daily.dev keeps distributed-systems patterns like fencing tokens close at hand for engineers building crash-safe workers."}}]} ```