<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/Wwl0wNTrW" -->

---
title: How we built an async-aware Python profiler | daily.dev
description: Datadog engineers describe how they rebuilt their Python profiler to be async-aware, reconstructing parent-child relationships between asyncio tasks into...
canonical: https://daily.dev/posts/how-we-built-an-async-aware-python-profiler-wwl0wntrw
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: How we built an async-aware Python profiler | daily.dev
og:description: Datadog engineers describe how they rebuilt their Python profiler to be async-aware, reconstructing parent-child relationships between asyncio tasks into...
og:url: https://daily.dev/posts/how-we-built-an-async-aware-python-profiler-wwl0wntrw
og:image: https://api.daily.dev/og/posts/Wwl0wNTrW.png
og:image:alt: How we built an async-aware Python profiler
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# How we built an async-aware Python profiler

**[Datadog](https://daily.dev/sources/datadog)** · 19 min read · 14 upvotes · 1 comments

## Summary

Datadog engineers describe how they rebuilt their Python profiler to be async-aware, reconstructing parent-child relationships between asyncio tasks into 'stacked stacks' so flame graphs reflect actual call context instead of flattening everything. They detail switching from process_vm_readv to a signal-handler-protected memcpy to safely read process memory, which cut sampling overhead roughly 50% for synchronous code and over 60% overall in production. Real-world findings included an infinite-loop decompression bug fixed upstream, wasteful log.debug string generation, and NumPy call optimizations discovered via an AI agent using MCP tools, all enabled by the higher-fidelity async profiling now available in the latest ddtrace releases.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://www.datadoghq.com/blog/engineering/async-python-profiler>

## Questions this post answers

### How did Datadog reduce Python profiler overhead for asyncio applications?

They replaced the process_vm_readv system call with a protected memcpy guarded by a signal handler that catches SIGSEGV and SIGBUS faults, avoiding costly kernel transitions on each memory read. Combined with removing C++ exceptions from the sampling path (15% reduction) and interning strings (25% reduction), this cut overhead by about 50% for synchronous code and over 60% overall in production asyncio services.

_Engineers optimizing async Python performance can track profiling techniques like these on daily.dev._

### Why do traditional flame graphs fail to represent asyncio task relationships correctly?

Each asyncio task has its own independent execution stack, so a traditional profiler shows work like a Cassandra query as free-floating rather than attributed to the task or coroutine that depended on it. This flattening loses the caller context, making it impossible to tell which application path triggered particular work, and produces incorrect reported time for callers.

_Developers debugging async Python performance issues follow profiling deep dives like this on daily.dev._

### What causes a Python asyncio task's stack to appear under the wrong parent in a profiler?

A task can outlive the task that created it, since creating a task does not require immediately awaiting it. Datadog's profiler attributes time in the child task to the creating task while the creator is still running, then re-attributes that time to whichever task is currently awaiting it once the creator finishes, rather than fixing attribution to only the original creator.

_Teams debugging asyncio task lifecycles can find profiling deep dives like this through daily.dev._

## Community discussion

Top comments from developers on daily.dev.

**@ristotoldsep** · 0 upvotes

> Async-aware flame graphs? yes please. The flattened ones tell half the story.

## Similar posts on daily.dev

- [\#497: Faster than light profiling](https://daily.dev/posts/497-faster-than-light-profiling-iqkcdikzo) · Planet Python · 0 upvotes · 0 comments
- [What's new in Pyroscope: profile more of your workflows, troubleshoot with AI agents, and more](https://daily.dev/posts/what-s-new-in-pyroscope-profile-more-of-your-workflows-troubleshoot-with-ai-agents-and-more-q3gcvhrna) · Grafana Labs · 0 upvotes · 0 comments
- [profiling.sampling — Statistical profiler ¶](https://daily.dev/posts/profiling-sampling-statistical-profiler--bd9rldcma) · Hacker News · 1 upvotes · 0 comments
- [Python 3.15 Has a Production Profiler. Python 3.15 Is Not Production-Ready.](https://daily.dev/posts/python-3-15-has-a-production-profiler-python-3-15-is-not-production-ready--wgbuvk9t9) · Medium · 1 upvotes · 0 comments
- [Introducing on-demand CPU and memory profiling with flamegraphs for Workers and Durable Objects](https://daily.dev/posts/introducing-on-demand-cpu-and-memory-profiling-with-flamegraphs-for-workers-and-durable-objects-dpdget4vv) · Cloudflare · 2 upvotes · 0 comments

---

Tags: [#python](https://daily.dev/tags/python), [#devops](https://daily.dev/tags/devops), [#performance](https://daily.dev/tags/performance)

[View this post on daily.dev](https://daily.dev/posts/how-we-built-an-async-aware-python-profiler-wwl0wntrw)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"How we built an async-aware Python profiler","url":"https://daily.dev/posts/how-we-built-an-async-aware-python-profiler-wwl0wntrw","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-we-built-an-async-aware-python-profiler-wwl0wntrw"},"datePublished":"2026-09-30T13:29:08.508Z","dateModified":"2026-09-30T13:29:32.875Z","description":"Datadog engineers describe how they rebuilt their Python profiler to be async-aware, reconstructing parent-child relationships between asyncio tasks into...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3e9112cc4dbde3a4b4188df5c377b476?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3e9112cc4dbde3a4b4188df5c377b476?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Datadog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Datadog","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/8f9d7e3e4bd64bd7b9db9b2ea83e0c6a","url":"https://daily.dev/sources/datadog"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/how-we-built-an-async-aware-python-profiler-wwl0wntrw","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":14},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"python,devops,performance","timeRequired":"PT19M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Datadog","item":"https://daily.dev/sources/datadog"},{"@type":"ListItem","position":3,"name":"How we built an async-aware Python profiler"}]}
{"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/how-we-built-an-async-aware-python-profiler-wwl0wntrw","comment":[{"@type":"Comment","text":"Async-aware flame graphs? yes please. The flattened ones tell half the story.","datePublished":"2026-10-02T11:41:39.788Z","url":"https://daily.dev/posts/Wwl0wNTrW#c-cYzvK4u9h","author":{"@type":"Person","name":"Risto Tõldsep","url":"https://daily.dev/ristotoldsep","image":"https://media.daily.dev/image/upload/s--i6RRoxB7--/f_auto,pg_1/v1794383659/avatars/avatar_ryDlbifZ1EuRuh3g51ryC?_a=BAMAMiiw0"}}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-we-built-an-async-aware-python-profiler-wwl0wntrw#faq","mainEntity":[{"@type":"Question","name":"How did Datadog reduce Python profiler overhead for asyncio applications?","acceptedAnswer":{"@type":"Answer","text":"They replaced the process_vm_readv system call with a protected memcpy guarded by a signal handler that catches SIGSEGV and SIGBUS faults, avoiding costly kernel transitions on each memory read. Combined with removing C++ exceptions from the sampling path (15% reduction) and interning strings (25% reduction), this cut overhead by about 50% for synchronous code and over 60% overall in production asyncio services. Engineers optimizing async Python performance can track profiling techniques like these on daily.dev."}},{"@type":"Question","name":"Why do traditional flame graphs fail to represent asyncio task relationships correctly?","acceptedAnswer":{"@type":"Answer","text":"Each asyncio task has its own independent execution stack, so a traditional profiler shows work like a Cassandra query as free-floating rather than attributed to the task or coroutine that depended on it. This flattening loses the caller context, making it impossible to tell which application path triggered particular work, and produces incorrect reported time for callers. Developers debugging async Python performance issues follow profiling deep dives like this on daily.dev."}},{"@type":"Question","name":"What causes a Python asyncio task's stack to appear under the wrong parent in a profiler?","acceptedAnswer":{"@type":"Answer","text":"A task can outlive the task that created it, since creating a task does not require immediately awaiting it. Datadog's profiler attributes time in the child task to the creating task while the creator is still running, then re-attributes that time to whichever task is currently awaiting it once the creator finishes, rather than fixing attribution to only the original creator. Teams debugging asyncio task lifecycles can find profiling deep dives like this through daily.dev."}}]}
```

