<a href="https://github.com/coroot/coroot" target="_blank" rel="noopener nofollow">Coroot</a> is an Apache 2.0 open source platform that simplifies observability with no-code configuration. The <a href="https://github.com/coroot/coroot-node-agent" target="_blank" rel="noopener nofollow">Coroot node-agent</a> already collects CPU profiles for any process on the node <a href="https://coroot.com/blog/troubleshooting-java-applications-with-coroot" target="_blank" rel="noopener nofollow">using eBPF</a>, with zero integration from the application side. For Java, we <a href="https://coroot.com/blog/java-profiling-with-async-profiler" target="_blank" rel="noopener nofollow">dynamically inject</a> async-profiler into the JVM to get memory and lock profiles. But Go processes were still a blind spot for non-CPU profiling unless the app exposed a pprof endpoint and the cluster-agent scraped it.
We wanted the same zero-config experience for Go heap profiles. This post is about how we got there.
<h1>The runtime already profiles</h1>
Go’s runtime has a built-in memory profiler. On every allocation, the runtime samples with probability <code>size / MemProfileRate</code> and records the call stack. The default rate is <code>512 * 1024</code>, or about 1 sample per 512KB allocated. Samples are aggregated into a linked list of “buckets”, where each bucket represents a unique (stack trace, size class) combination and accumulates four counters: total allocations, total frees, bytes allocated, bytes freed.
This is what <code>runtime.MemProfile()</code> returns and what <code>go tool pprof http://.../debug/pprof/heap</code> renders. The overhead is negligible and it’s been production-grade since forever.
There’s one catch. The Go linker has an optimization: if no code in the binary references <code>runtime.MemProfile</code>, it sets an internal <code>disableMemoryProfiling</code> flag, and the runtime sets <code>MemProfileRate = 0</code> on init. No samples, no buckets, nothing to read. A binary that doesn’t import <code>runtime/pprof</code> or <code>net/http/pprof</code> (directly or transitively) has no heap profile available, even though the runtime fully supports it. We’ll come back to this.
This list is what <code>runtime.MemProfile()</code> walks when <code>pprof</code> asks for a heap profile. It’s literally the global variable <code>runtime.mbuckets</code>:
<pre><code>// runtime/mprof.go
var (
 mbuckets atomic.UnsafePointer // *bucket, memory profile buckets
 ...
)
</code></pre>
So the data is already there, being collected continuously, for free. The only question is how to read it from outside the process.
<h1>Reading process memory from outside</h1>
Linux exposes every process’s virtual address space via <code>/proc/&lt;pid&gt;/mem.</code> With the right permissions (our node-agent already has <code>CAPSYSPTRACE</code>), you can <code>pread()</code> arbitrary addresses. It’s read-only, it doesn’t suspend the process, the target doesn’t even know you’re there.
The plan:
<ol>
<li>
Find the virtual address of <code>runtime.mbuckets</code> in the Go binary’s symbol table.
</li>
<li>
Read the pointer value at that address from <code>/proc/&lt;pid&gt;/mem.</code>
</li>
<li>
Walk the linked list, reading each bucket’s header, stack PCs, and memRecord.
</li>
<li>
Convert to pprof format and upload.
</li>
</ol>
<h1>Finding runtime.mbuckets without loading the symbol table</h1>
The first gotcha: Go binaries embed their own symbol table (pclntab) for runtime use, but <code>runtime.mbuckets</code> is not a function. It’s a variable, which lives in the <code>ELF .symtab</code> section. On a stripped binary (<code>go build -ldflags=&quot;-s&quot;</code>), there’s no <code>.symtab</code> and we can’t find the symbol. We skip those.
On an unstripped binary, <code>.symtab</code> can be huge. For <code>k3s</code>, it’s ~11MB. Using <code>debug/elf.File.Symbols()</code> loads all of it into memory at once. For a node-agent that profiles dozens of Go processes, that’s not OK.
So we wrote a streaming scan that reads one <code>Elf64_Sym</code> entry at a time and reads only the bytes we need from the string table:
<pre><code>func findSymbolValue(ef *elf.File, sectionName, symName string) (uint64, error) {
 section := ef.Section(sectionName)
 strtab := ef.Sections[section.Link]

 symReader := section.Open()
 entry := make([]byte, 24) // Elf64_Sym
 target := []byte(symName)
 nameBuf := make([]byte, len(target)+1)

 for {
 if _, err := symReader.Read(entry); err != nil {
 break
 }
 nameIdx := ef.ByteOrder.Uint32(entry[0:4])
 value := ef.ByteOrder.Uint64(entry[8:16])

 n, _ := strtab.ReadAt(nameBuf, int64(nameIdx))
 if n &gt; len(target) &amp;&amp; nameBuf[len(target)] != 0 {
 continue
 }
 if string(nameBuf[:len(target)]) == symName {
 return value, nil
 }
 }
 return 0, fmt.Errorf(&quot;%s not found&quot;, symName)
}
</code></pre>
Peak memory: a 24-byte buffer plus a 17-byte buffer (<code>len(&quot;runtime.mbuckets&quot;)+1</code>), regardless of binary size.
Before doing this expensive scan we also check if the binary is Go at all via the <code>.go.buildinfo</code> section: one section header lookup, zero allocations.
<h1>The bucket layout, and two traps</h1>
The bucket struct itself is just a 48-byte header:
<pre><code>type bucket struct {
 _ sys.NotInHeap
 next *bucket
 allnext *bucket
 typ bucketType
 hash uintptr
 size uintptr
 nstk uintptr
}
</code></pre>
But the runtime allocates extra space after it and stores two more things in the same contiguous region: the stack trace (nstk program counter addresses, 8 bytes each) and a memRecord struct holding the alloc/free counters.
So from our point of view, each bucket is a variable-sized blob: <code>48 bytes header + nstk*8 bytes of PCs + 128</code> bytes of <code>memRecord.</code> We read the header first to get <code>nstk</code>, then the rest.
Two traps we fell into:
Trap 1: the first field, <code>_ sys.NotInHeap</code>, looks like 8 bytes of padding. It’s zero bytes. Sizing the header at 56 bytes gave us nicely parsed garbage: valid-looking pointers that turned out to be hash values, and typ values in the quintillions. Go 1.17 through 1.19 used a <code>//go:notinheap</code> comment directive instead; Go 1.20 switched to the <a href="https://github.com/golang/go/commit/a719a78c1b36141af68d84970695fe95263fb896" target="_blank" rel="noopener nofollow">typed marker</a>, but the binary layout didn’t change. The real header is 48 bytes.
Trap 2: there are two pointer fields, <code>next</code> and <code>allnext.</code> They are not the same list. next is the hash table chain within a size class. allnext is the global list of all memProfile buckets. We want <code>allnext.</code>
<h1>The delta problem</h1>
The counters in <code>memRecord</code> are cumulative: they grow monotonically over the lifetime of the process. If we want an allocation rate, we need to compute the delta between two collection cycles.
We keep a map per PID of <code>bucket address -&gt; previous counters</code> and subtract on each cycle to get the delta. We key by bucket address rather than stack hash: the Go runtime never frees mprof buckets, so the address is a stable unique identifier, and it’s a single <code>uint64</code> instead of a variable-length string, which avoids a huge amount of allocation churn in the hot path.
<h1>Too many syscalls</h1>
Early profiles showed our collector spending 30-40% of its CPU in <code>syscall.Pread</code>. Each bucket needs at least 2 reads: one for the header (to get <code>nstk</code>), then one for the variable-length <code>stk[nstk] | memRecord</code> block. With 1000+ buckets per process and a dozen Go processes on a node, that’s thousands of syscalls per minute.
We tried a read-ahead cache: on a miss, pull 256KB centered around the requested address. The idea was that Go’s <code>persistentalloc</code> places buckets in large arenas, so consecutive buckets in the <code>allnext</code> chain might be physically close.
We instrumented jump distances between consecutive buckets for one process with 1230 buckets. 40% of jumps are &gt;16MB. Buckets are scattered across the entire process address space, not clustered.
A 256KB cache hits ~20% of the time: better than nothing, but the best we could do without multi-MB buffers that cost more than they save.
<h1>The linker-disabled profiling problem</h1>
After deploying, we saw some Go processes return an empty bucket list (<code>runtime.mbuckets pointer was 0x0</code>) even though they were clearly allocating memory (tens of MB RSS, actively running).
Turns out the Go linker has an optimization: if no code in the binary references <code>runtime.MemProfile</code>, it sets a <code>disableMemoryProfiling</code> flag, and the runtime sets <code>MemProfileRate = 0</code> on init. No <code>profilealloc()</code> calls, no buckets ever created.
This hits any Go binary that doesn’t import <code>runtime/pprof</code> or <code>net/http/pprof</code>, directly or transitively. In our case it was a small load generator: no pprof, no HTTP server, no dependencies that would drag pprof in. The profile endpoint the runtime would serve is dead code, so the linker dropped it.
The fix: we can write to <code>/proc/&lt;pid&gt;/mem</code> too. If we detect <code>MemProfileRate == 0</code>, we write 524288 (the default) back to the <code>runtime.MemProfileRate</code> address. The runtime checks this variable on every allocation, so the change takes effect immediately: no restart, no signal, nothing. Just a single atomic 8-byte write to a known address in the data segment.
This is gated behind a <code>--go-heap-profiler=force</code> flag for users who want the “always on” behavior:
<pre><code>--go-heap-profiler=disabled # off
--go-heap-profiler=enabled # default, passive only
--go-heap-profiler=force # write MemProfileRate if zero
</code></pre>
The overhead of re-enabling profiling is whatever the Go default overhead is: ~1 sample per 512KB. For any workload where this matters, you’d want it on anyway.
<h1>Allocation rate metrics</h1>
Since we already compute per-bucket alloc deltas, exposing total allocation rate as Prometheus counters is free:
<pre><code>container_go_alloc_bytes_total # total bytes allocated
container_go_alloc_objects_total # total objects allocated
</code></pre>
Summed across all buckets in the process. Coroot uses them to draw the allocation rate chart alongside the flamegraph.
–That’s all for now on account of character count! You can read the full article and view chaos experiments <a href="https://coroot.com/blog/zero-config-go-heap-profiling/" target="_blank" rel="noopener nofollow">here.</a> To set up zero-config Golang Heap Profiling on your system (completely open source) you can visit our (Github.)[<a href="https://github.com/coroot/coroot" target="_blank" rel="noopener nofollow">https://github.com/coroot/coroot</a>]
<h1></h1>

A supportive squad for Go developers who want to dive deeper into what makes Go special. We focus on the language's clean simplicity, powerful concurrency, and reliable performance.

Go Developers

Coroot's node-agent now supports zero-configuration Go heap profiling by reading Go's built-in memory profiler data directly from process memory via /proc/<pid>/mem, without requiring any changes to the target application. The approach locates the runtime.mbuckets symbol in the ELF symbol table using a streaming scan to avoid loading the full symbol table, then walks the linked list of allocation buckets to reconstruct heap profiles in pprof format. Key engineering challenges covered include: streaming ELF symbol lookup to minimize memory overhead, correctly parsing the variable-sized bucket struct (including a zero-byte NotInHeap field trap), computing allocation deltas between collection cycles, reducing syscall overhead with a read-ahead cache, and handling Go binaries where the linker disabled memory profiling. An optional --go-heap-profiler=force flag can re-enable profiling on such binaries by writing the default MemProfileRate value directly into the process's memory segment. Allocation rate metrics are also exposed as Prometheus counters.

No-Configuration Golang Heap Profiling