--- title: The Lambda Bug That Silently Ate Our Queue | daily.dev description: A Lambda function was silently dropping queue items due to un-awaited async HTTP calls made via axios. When the handler returned, Lambda froze the execution... canonical: https://daily.dev/posts/the-lambda-bug-that-silently-ate-our-queue-tirhgawjv twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The Lambda Bug That Silently Ate Our Queue | daily.dev og:description: A Lambda function was silently dropping queue items due to un-awaited async HTTP calls made via axios. When the handler returned, Lambda froze the execution... og:url: https://daily.dev/posts/the-lambda-bug-that-silently-ate-our-queue-tirhgawjv og:image: https://api.daily.dev/og/posts/TirHGAWjV.png og:image:alt: The Lambda Bug That Silently Ate Our Queue og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The Lambda Bug That Silently Ate Our Queue **[Satyam Sharma](https://daily.dev/sources/r8lt3tm4yxsbxw0zrahv2)** · [@satyuuum](https://daily.dev/satyuuum) · 5 min read · 1 upvotes · 0 comments ## Summary A Lambda function was silently dropping queue items due to un-awaited async HTTP calls made via axios. When the handler returned, Lambda froze the execution environment mid-request, killing in-flight network calls with no errors surfacing. The key insight: Lambda's runtime only tracks work that's part of the handler's returned promise — fire-and-forget calls are invisible to it. The fix is either to await all async work before returning, or to offload fire-and-forget work to a durable messaging system like SQS or EventBridge that guarantees delivery. ## Content For the better part of a week, items were disappearing from our processing queue. Not failing. Disappearing. No errors, no exceptions, no dead-letter entries — just a slow, random bleed of data that left no trace. This is the story of what was happening, why it was nearly impossible to see, and the one-line mental model that explains all of it. ### The setup We had a Lambda function that, as part of handling each queue item, made an HTTP call via axios to another internal server. The call was fire-and-forget: we kicked it off but didn't `await` it, because we didn't need the response to continue. The handler did its main work, returned, and moved on. Classic "send it and don't block" pattern — the kind that looks perfectly reasonable in a code review and behaves fine in local testing. In production, items kept vanishing. There was no discernible pattern to which ones, no correlation with payload size or time of day, and — most maddeningly — no error anywhere to catch. We instrumented the handler with console logs at every step. The logs showed the axios call being initiated. They just didn't always show it completing, and when it didn't complete, nothing downstream registered a problem. ### Why the logs didn't help CloudWatch is not built for tracing the lifecycle of async work that outlives your handler. Lambda logs are organized around invocations: a request comes in, your handler runs, you get a log stream for that execution. Anything that happens _after_ the handler returns falls into a gap the logging model doesn't really account for. So when our axios request was still in transit at the moment the handler finished, the tail end of that request's life — the part where it either succeeded or failed — happened in a window where nothing was listening and nothing was being recorded. We spent days scattering console logs, convinced we were missing a branch in our own logic. We weren't. The bug wasn't in our code's logic at all. It was in our assumption about _when Lambda decides it's done_. ### What was actually happening Here's the core of it. AWS Lambda runs your code inside an execution environment — a small, reusable sandbox. When your handler returns, Lambda considers the invocation complete and is free to **freeze** that environment, suspending all activity in it, or to **terminate** it entirely. Freezing halts the event loop mid-flight: any pending timers, open sockets, or in-progress network requests are simply paused. The environment might thaw later for a subsequent invocation, or it might get torn down and never resumed. Now apply that to an un-awaited axios call. The moment we returned from the handler, our HTTP request was frequently still open — bytes in transit, waiting on the remote server. Lambda saw the handler return, decided the invocation was over, and froze the environment. The socket died mid-write. No connection-reset error bubbled up to our code, because our code had already returned and there was no longer any awaited promise chain for the failure to propagate into. The request just… stopped existing. Sometimes the request finished a few milliseconds before the freeze and everything worked — which is exactly why the loss was intermittent and impossible to reproduce on demand. That intermittency was the whole reason it took days rather than minutes. ### The mental model that fixes it The one sentence worth internalizing: **in Lambda, if you don't await your async work, the runtime doesn't know it exists, and it won't wait for it.** The runtime's definition of "finished" is tied to what the handler's returned promise resolves. An `async` handler returns the instant its promise settles. A fire-and-forget axios call is deliberately _not_ part of that promise chain — that's what makes it fire-and-forget — so from the runtime's point of view there is nothing left to wait for. It freezes with a clear conscience. A common misconception is that `context.callbackWaitsForEmptyEventLoop` protects you here. It does not. That flag governs the older callback-style handler and controls whether Lambda waits for the event loop to drain before returning via the callback. With an `async` handler, resolution is driven entirely by the returned promise, not by the state of the event loop — so an orphaned request sitting in the loop has no bearing on when the function is considered complete. ### What to do instead If you need the work to finish, `await` it. If you're fanning out several calls, collect them and await the batch together before returning. The cost is that your handler now blocks until the calls complete, but that's the price of certainty — and it's almost always cheaper than losing data. If you genuinely want the work decoupled from the response — you truly don't care to wait — then make it someone else's job to guarantee delivery. Push a message to SQS or emit an event to EventBridge and let a downstream consumer own the actual HTTP call. That way the work lives in a system designed to run it to completion and retry on failure, instead of riding on a soon-to-be-frozen execution environment that has every right to walk away the instant your handler says it's done. The lesson underneath the lesson: fire-and-forget is a promise you're making to yourself that the work will happen. Lambda never signed that contract. #aws #lambda #systemdesign ## Similar posts on daily.dev - [I Added SQS to My Backend :\)](https://daily.dev/posts/i-added-sqs-to-my-backend--bnqclheep) · Medium · 0 upvotes · 0 comments - [AWS Lambda Pushes Serverless Toward Long-Running Workloads](https://daily.dev/posts/aws-lambda-pushes-serverless-toward-long-running-workloads-2zbrcptyu) · InfoQ · 0 upvotes · 0 comments - [How to Lose Data Using AWS Serverless Without Even Knowing It](https://daily.dev/posts/how-to-lose-data-using-aws-serverless-without-even-knowing-it-4ra8u44mm) · Serverless Life · 1 upvotes · 0 comments --- Tags: [#career](https://daily.dev/tags/career), [#nodejs](https://daily.dev/tags/nodejs), [#serverless](https://daily.dev/tags/serverless), [#aws-lambda](https://daily.dev/tags/aws-lambda) [View this post on daily.dev](https://daily.dev/posts/the-lambda-bug-that-silently-ate-our-queue-tirhgawjv) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/the-lambda-bug-that-silently-ate-our-queue-tirhgawjv","headline":"The Lambda Bug That Silently Ate Our Queue","text":"A Lambda function was silently dropping queue items due to un-awaited async HTTP calls made via axios. When the handler returned, Lambda froze the execution environment mid-request, killing in-flight network calls with no errors surfacing. The key insight: Lambda's runtime only tracks work that's part of the handler's returned promise — fire-and-forget calls are invisible to it. The fix is either to await all async work before returning, or to offload fire-and-forget work to a durable messaging system like SQS or EventBridge that guarantees delivery.","url":"https://daily.dev/posts/the-lambda-bug-that-silently-ate-our-queue-tirhgawjv","datePublished":"2026-07-21T17:24:38.486Z","dateModified":"2026-09-13T21:26:38.450Z","author":{"@type":"Person","name":"Satyam Sharma","url":"https://daily.dev/satyuuum","image":"https://media.daily.dev/image/upload/s--ydnW70t---/f_auto/v1784656462/avatars/avatar_R8lt3tM4yXSbxw0zRAhV2?_a=BAMAMicg0","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":210}},"image":"https://media.daily.dev/image/upload/s--tnp1pSjP--/f_auto/v1784654846/posts/TirHGAWjV?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/r8lt3tm4yxsbxw0zrahv2","name":"Satyam Sharma"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Satyam Sharma","item":"https://daily.dev/sources/r8lt3tm4yxsbxw0zrahv2"},{"@type":"ListItem","position":3,"name":"The Lambda Bug That Silently Ate Our Queue"}]} ```