--- title: How LlamaIndex uses Temporal to scale reliable document... description: LlamaIndex describes migrating its LlamaParse document-processing pipeline from RabbitMQ to Temporal, a durable workflow execution engine. The move addressed... canonical: https://daily.dev/posts/how-llamaindex-uses-temporal-to-scale-reliable-document-orchestration-1el2isiyl twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How LlamaIndex uses Temporal to scale reliable document orchestration | daily.dev og:description: LlamaIndex describes migrating its LlamaParse document-processing pipeline from RabbitMQ to Temporal, a durable workflow execution engine. The move addressed... og:url: https://daily.dev/posts/how-llamaindex-uses-temporal-to-scale-reliable-document-orchestration-1el2isiyl og:image: https://api.daily.dev/og/posts/1eL2iSIYL.png og:image:alt: How LlamaIndex uses Temporal to scale reliable document orchestration og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How LlamaIndex uses Temporal to scale reliable document orchestration **[Temporal](https://daily.dev/sources/temporal)** · 8 min read · 0 upvotes · 0 comments ## Summary LlamaIndex describes migrating its LlamaParse document-processing pipeline from RabbitMQ to Temporal, a durable workflow execution engine. The move addressed problems with durability, fairness/concurrency, and resource usage that had accumulated as ad-hoc code on top of RabbitMQ (heartbeats, retry counters, status columns, custom backpressure). Using Temporal's Workflows, Signals, and workflow-ID uniqueness, the team built semaphore- and permit-based concurrency primitives to gate work without external locks or caches, enabling them to consolidate multiple bespoke layers and scale to tens of millions of pages processed per day for their new Batch API. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I implement a distributed semaphore or concurrency limiter using Temporal workflows? Model the semaphore as a long-lived coordinator Workflow keyed by a deterministic ID like semaphore:job_concurrency:. Callers use signal-with-start with WorkflowIDConflictPolicy.USE_EXISTING so Temporal's ID uniqueness collapses concurrent starters onto the same execution; acquiring is a signal out from the caller and a grant signal back, with the caller waiting via wait_condition until granted. _daily.dev surfaces implementation patterns like this for teams designing concurrency limits with Temporal._ ### Why did LlamaIndex move LlamaParse off RabbitMQ to Temporal? RabbitMQ lacked durable state, forcing LlamaIndex to build status tracking, retry counters, and heartbeat mechanisms on top of queue messages to handle silently killed jobs from OOM crashes and to manage fairness and backpressure so no single user could exhaust provider rate limits. Temporal provided durable workflow execution and built-in coordination primitives, letting them process tens of millions of pages daily for their Batch API while removing caching layers and extra microservices. _daily.dev helps engineers weigh migrations like RabbitMQ to Temporal when scaling document pipelines._ ### How do you build a distributed lock in Temporal using workflow signals? Start a Workflow with a caller-chosen ID that acts as the lock name (e.g. permit:gpu-pool:2); the start either succeeds, meaning the caller holds the permit, or fails with WorkflowAlreadyStartedError, meaning someone else holds it. The workflow waits on a signal named release or times out via a lease_seconds timeout, and ParentClosePolicy.TERMINATE cleans up the permit if the parent workflow closes. _daily.dev keeps developers current on Temporal patterns for locking shared resources in distributed systems._ ## Similar posts on daily.dev - [How Rapidflare built a million\+ document ingestion pipeline for Agents on Temporal](https://daily.dev/posts/how-rapidflare-built-a-million-document-ingestion-pipeline-for-agents-on-temporal-jjkofixes) · Temporal · 1 upvotes · 0 comments - [Temporal hits 3,000 paying customers with its crash-proof workflow engine](https://daily.dev/posts/temporal-hits-3-000-paying-customers-with-its-crash-proof-workflow-engine-0pq440ota) · The New Stack · 1 upvotes · 0 comments --- Tags: [#architecture](https://daily.dev/tags/architecture), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#workflow-orchestration](https://daily.dev/tags/workflow-orchestration), [#rabbitmq](https://daily.dev/tags/rabbitmq) [View this post on daily.dev](https://daily.dev/posts/how-llamaindex-uses-temporal-to-scale-reliable-document-orchestration-1el2isiyl) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How LlamaIndex uses Temporal to scale reliable document orchestration","url":"https://daily.dev/posts/how-llamaindex-uses-temporal-to-scale-reliable-document-orchestration-1el2isiyl","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-llamaindex-uses-temporal-to-scale-reliable-document-orchestration-1el2isiyl"},"datePublished":"2026-09-01T13:06:43.283Z","dateModified":"2026-09-13T19:46:55.647Z","description":"LlamaIndex describes migrating its LlamaParse document-processing pipeline from RabbitMQ to Temporal, a durable workflow execution engine. The move addressed...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/18d023910229e550cc7c3f5277634f4e?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/18d023910229e550cc7c3f5277634f4e?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Temporal","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Temporal","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/65290ca7e3ab46a7b1df76f2757800c2","url":"https://daily.dev/sources/temporal"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-llamaindex-uses-temporal-to-scale-reliable-document-orchestration-1el2isiyl","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"architecture,distributed-systems,workflow-orchestration,rabbitmq","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Temporal","item":"https://daily.dev/sources/temporal"},{"@type":"ListItem","position":3,"name":"How LlamaIndex uses Temporal to scale reliable document orchestration"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-llamaindex-uses-temporal-to-scale-reliable-document-orchestration-1el2isiyl#faq","mainEntity":[{"@type":"Question","name":"How can I implement a distributed semaphore or concurrency limiter using Temporal workflows?","acceptedAnswer":{"@type":"Answer","text":"Model the semaphore as a long-lived coordinator Workflow keyed by a deterministic ID like semaphore:job_concurrency:. Callers use signal-with-start with WorkflowIDConflictPolicy.USE_EXISTING so Temporal's ID uniqueness collapses concurrent starters onto the same execution; acquiring is a signal out from the caller and a grant signal back, with the caller waiting via wait_condition until granted. daily.dev surfaces implementation patterns like this for teams designing concurrency limits with Temporal."}},{"@type":"Question","name":"Why did LlamaIndex move LlamaParse off RabbitMQ to Temporal?","acceptedAnswer":{"@type":"Answer","text":"RabbitMQ lacked durable state, forcing LlamaIndex to build status tracking, retry counters, and heartbeat mechanisms on top of queue messages to handle silently killed jobs from OOM crashes and to manage fairness and backpressure so no single user could exhaust provider rate limits. Temporal provided durable workflow execution and built-in coordination primitives, letting them process tens of millions of pages daily for their Batch API while removing caching layers and extra microservices. daily.dev helps engineers weigh migrations like RabbitMQ to Temporal when scaling document pipelines."}},{"@type":"Question","name":"How do you build a distributed lock in Temporal using workflow signals?","acceptedAnswer":{"@type":"Answer","text":"Start a Workflow with a caller-chosen ID that acts as the lock name (e.g. permit:gpu-pool:2); the start either succeeds, meaning the caller holds the permit, or fails with WorkflowAlreadyStartedError, meaning someone else holds it. The workflow waits on a signal named release or times out via a lease_seconds timeout, and ParentClosePolicy.TERMINATE cleans up the permit if the parent workflow closes. daily.dev keeps developers current on Temporal patterns for locking shared resources in distributed systems."}}]} ```