--- title: Job queues are deceptively tricky | daily.dev description: Job queues appear simple but hide significant complexity in their scheduling semantics. Using a real-world example of git repository repacking (with 7-hour... canonical: https://daily.dev/posts/job-queues-are-deceptively-tricky-cj0ydgd0g twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Job queues are deceptively tricky | daily.dev og:description: Job queues appear simple but hide significant complexity in their scheduling semantics. Using a real-world example of git repository repacking (with 7-hour... og:url: https://daily.dev/posts/job-queues-are-deceptively-tricky-cj0ydgd0g og:image: https://api.daily.dev/og/posts/cj0yDGd0G.png og:image:alt: Job queues are deceptively tricky og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Job queues are deceptively tricky **[Lobsters](https://daily.dev/sources/lobsters)** · 16 min read · 3 upvotes · 0 comments ## Summary Job queues appear simple but hide significant complexity in their scheduling semantics. Using a real-world example of git repository repacking (with 7-hour wholesale vs 2-hour incremental jobs), the post explores four possible behaviors when a new job is triggered while a previous one is still running: Parallel Spawn, Prefer New, Wait, and Prefer Old. Counterintuitively, 'Prefer Old' (cancel the newly triggered job and let the running one finish) turns out to be the correct fit for workloads where jobs routinely exceed the scheduling interval. The post also covers how unbounded queues without proper limits can silently accumulate massive backlogs. Key design lenses discussed: being wary of queues, enforcing explicit limits, and reasoning about fault models. The takeaway is that explicitly documenting queue semantics, limits, and fault assumptions helps both system designers and consumers avoid subtle but costly misconfigurations. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What are the different ways a job queue can handle a new job being scheduled while the previous one is still running? There are four possible semantics: Parallel Spawn (run both if concurrency limit allows), Prefer New (cancel the old job and start the new one), Wait (queue the new job until the old one finishes), and Prefer Old (cancel the new job and let the old one continue). Each implies different assumptions about queue bounding, hard limits, and fault models, and the best choice depends on whether stale results are more harmful than dropped work. _Weighing these tradeoffs for a scheduling problem is easier with technical deep dives like this surfaced on daily.dev._ ### Why might a 'Prefer Old' job scheduling strategy that cancels the newest job be better than 'Prefer New' or 'Wait'? For a workload with 7-hour jobs on a 3-hour schedule interval over a 48-hour weekend, Prefer Old finishes nearly all pending work with only about 1 hour of spillover into Monday, canceling 9 of 16 triggered jobs. Prefer New wastes 48 hours of compute across 16 jobs that never complete, and Wait accumulates 64 hours of backlog that can't be drained before the next weekend. _Comparing scheduling strategies against real throughput numbers helps when tuning job queues, a case daily.dev keeps close at hand._ ### How much smaller is a wholesale git repack compared to an incremental repack? A wholesale repacked git repository can be roughly 50-60% smaller than an incrementally packed one, though wholesale repacking recomputes everything from scratch and can take around 7 hours on a large repo, versus about 2 hours for an incremental repack that only reprocesses recent changes. _Engineers tuning storage versus freshness tradeoffs in repo tooling can find grounded comparisons like this on daily.dev._ ## Similar posts on daily.dev - [Fair by design: orchestrating background jobs in Ruby—Martian Chronicles, Evil Martians’ team blog](https://daily.dev/posts/fair-by-design-orchestrating-background-jobs-in-ruby-martian-chronicles-evil-martians-team-blog-piyszlqky) · Martian Chronicles · 0 upvotes · 0 comments - [All your workflows are about to be long running. Build for it.](https://daily.dev/posts/all-your-workflows-are-about-to-be-long-running-build-for-it--ff8epi154) · Inngest Blog · 0 upvotes · 0 comments - [Why Queues Don’t Fix Overload \(And What To Do Instead\)](https://daily.dev/posts/why-queues-don-t-fix-overload-and-what-to-do-instead--z9coej1wx) · P99 Conf · 0 upvotes · 0 comments --- Tags: [#career](https://daily.dev/tags/career), [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/job-queues-are-deceptively-tricky-cj0ydgd0g) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Job queues are deceptively tricky","url":"https://daily.dev/posts/job-queues-are-deceptively-tricky-cj0ydgd0g","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/job-queues-are-deceptively-tricky-cj0ydgd0g"},"datePublished":"2026-07-15T06:09:01.143Z","dateModified":"2026-09-13T20:41:03.475Z","description":"Job queues appear simple but hide significant complexity in their scheduling semantics. Using a real-world example of git repository repacking (with 7-hour...","image":"https://media.daily.dev/image/upload/s--VDukGCjf--/f_auto/v1722860399/public/Placeholder%2002","thumbnailUrl":"https://media.daily.dev/image/upload/s--VDukGCjf--/f_auto/v1722860399/public/Placeholder%2002","isAccessibleForFree":true,"articleSection":"Lobsters","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Lobsters","logo":"https://media.daily.dev/image/upload/s--tl8v_Fku--/f_auto,t_logo/v1698841318/logos/lobste.jpg","url":"https://daily.dev/sources/lobsters"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/job-queues-are-deceptively-tricky-cj0ydgd0g","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"career,distributed-systems","timeRequired":"PT16M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Lobsters","item":"https://daily.dev/sources/lobsters"},{"@type":"ListItem","position":3,"name":"Job queues are deceptively tricky"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/job-queues-are-deceptively-tricky-cj0ydgd0g#faq","mainEntity":[{"@type":"Question","name":"What are the different ways a job queue can handle a new job being scheduled while the previous one is still running?","acceptedAnswer":{"@type":"Answer","text":"There are four possible semantics: Parallel Spawn (run both if concurrency limit allows), Prefer New (cancel the old job and start the new one), Wait (queue the new job until the old one finishes), and Prefer Old (cancel the new job and let the old one continue). Each implies different assumptions about queue bounding, hard limits, and fault models, and the best choice depends on whether stale results are more harmful than dropped work. Weighing these tradeoffs for a scheduling problem is easier with technical deep dives like this surfaced on daily.dev."}},{"@type":"Question","name":"Why might a 'Prefer Old' job scheduling strategy that cancels the newest job be better than 'Prefer New' or 'Wait'?","acceptedAnswer":{"@type":"Answer","text":"For a workload with 7-hour jobs on a 3-hour schedule interval over a 48-hour weekend, Prefer Old finishes nearly all pending work with only about 1 hour of spillover into Monday, canceling 9 of 16 triggered jobs. Prefer New wastes 48 hours of compute across 16 jobs that never complete, and Wait accumulates 64 hours of backlog that can't be drained before the next weekend. Comparing scheduling strategies against real throughput numbers helps when tuning job queues, a case daily.dev keeps close at hand."}},{"@type":"Question","name":"How much smaller is a wholesale git repack compared to an incremental repack?","acceptedAnswer":{"@type":"Answer","text":"A wholesale repacked git repository can be roughly 50-60% smaller than an incrementally packed one, though wholesale repacking recomputes everything from scratch and can take around 7 hours on a large repo, versus about 2 hours for an incremental repack that only reprocesses recent changes. Engineers tuning storage versus freshness tradeoffs in repo tooling can find grounded comparisons like this on daily.dev."}}]} ```