--- title: Reading the code is the hard part | daily.dev description: Rewind VM, a deterministic VM, found a real race condition in git notes by replaying thread schedules. Running `git notes add` concurrently on two commits... canonical: https://daily.dev/posts/reading-the-code-is-the-hard-part-j4rk5nqar twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Reading the code is the hard part | daily.dev og:description: Rewind VM, a deterministic VM, found a real race condition in git notes by replaying thread schedules. Running `git notes add` concurrently on two commits... og:url: https://daily.dev/posts/reading-the-code-is-the-hard-part-j4rk5nqar og:image: https://api.daily.dev/og/posts/j4rk5nQAr.png og:image:alt: Reading the code is the hard part og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Reading the code is the hard part **[Farid Zakaria’s Blog](https://daily.dev/sources/fzakaria)** · 7 min read · 0 upvotes · 0 comments ## Summary Rewind VM, a deterministic VM, found a real race condition in git notes by replaying thread schedules. Running `git notes add` concurrently on two commits loses one note in 15 of 32 Rewind schedules, and in 25 of 30 runs on a normal Linux machine. Using `rewind diff`, `rewind blame`, and `rewind where`, the author traced the bug to notes-utils.c:64, which updates a ref without checking it still points where it did when the lock was taken. The author argues deterministic replay pairs well with AI agents, since narrowing a failure to one step and one line keeps an LLM's context small enough to read the code and suggest a fix. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why does running git notes add concurrently on two different commits sometimes lose one of the notes It's a read-modify-write race in git's ref update logic. One git notes add process reads the notes ref before another process has saved its change, then overwrites it with a stale version. The specific flaw is in notes-utils.c, which updates the ref without checking it still points where it did when the lock was taken. Testing showed 15 of 32 runs under a deterministic VM replay lost a note, and 25 of 30 runs on a normal Linux machine showed the same loss. _Developers chasing flaky concurrency bugs can find deeper debugging workflow write-ups through daily.dev._ ### how can I use deterministic replay tools to narrow down a concurrency bug to a single line of code Replay the program under many thread schedules until one fails, then use tooling to compare file outputs between a passing and failing run, identify which process wrote the differing bytes and when, and use debug symbols to show the exact source line and call stack at that step. In one case this process, using commands like rewind check, rewind diff, rewind blame, and rewind where, took three commands to pinpoint a single line in git's notes-utils.c. _daily.dev surfaces write-ups like this for engineers building reproducible debugging workflows._ ## Similar posts on daily.dev - [Rewind VM: a flaky build you only catch once](https://daily.dev/posts/rewind-vm-a-flaky-build-you-only-catch-once-a2u3ugkt4) · Farid Zakaria’s Blog · 0 upvotes · 0 comments - [Nix wrote half of my debugger](https://daily.dev/posts/nix-wrote-half-of-my-debugger-gnc47givc) · Farid Zakaria’s Blog · 0 upvotes · 0 comments - [Who wrote this byte?](https://daily.dev/posts/who-wrote-this-byte--pkswobr3u) · Farid Zakaria’s Blog · 0 upvotes · 0 comments - [Always Be Blaming](https://daily.dev/posts/always-be-blaming-yi37onju1) · matklad · 0 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#git](https://daily.dev/tags/git) [View this post on daily.dev](https://daily.dev/posts/reading-the-code-is-the-hard-part-j4rk5nqar) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Reading the code is the hard part","url":"https://daily.dev/posts/reading-the-code-is-the-hard-part-j4rk5nqar","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/reading-the-code-is-the-hard-part-j4rk5nqar"},"datePublished":"2026-10-29T03:38:50.321Z","dateModified":"2026-10-29T03:39:19.195Z","description":"Rewind VM, a deterministic VM, found a real race condition in git notes by replaying thread schedules. Running `git notes add` concurrently on two commits...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1492ebca85c06e6e13520815dc8bbf78?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1492ebca85c06e6e13520815dc8bbf78?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Farid Zakaria’s Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Farid Zakaria’s Blog","logo":"https://media.daily.dev/image/upload/s--YqqvRt90--/f_auto/v1723789280/logos/fzakaria","url":"https://daily.dev/sources/fzakaria"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/reading-the-code-is-the-hard-part-j4rk5nqar","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,git","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Farid Zakaria’s Blog","item":"https://daily.dev/sources/fzakaria"},{"@type":"ListItem","position":3,"name":"Reading the code is the hard part"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/reading-the-code-is-the-hard-part-j4rk5nqar#faq","mainEntity":[{"@type":"Question","name":"why does running git notes add concurrently on two different commits sometimes lose one of the notes","acceptedAnswer":{"@type":"Answer","text":"It's a read-modify-write race in git's ref update logic. One git notes add process reads the notes ref before another process has saved its change, then overwrites it with a stale version. The specific flaw is in notes-utils.c, which updates the ref without checking it still points where it did when the lock was taken. Testing showed 15 of 32 runs under a deterministic VM replay lost a note, and 25 of 30 runs on a normal Linux machine showed the same loss. Developers chasing flaky concurrency bugs can find deeper debugging workflow write-ups through daily.dev."}},{"@type":"Question","name":"how can I use deterministic replay tools to narrow down a concurrency bug to a single line of code","acceptedAnswer":{"@type":"Answer","text":"Replay the program under many thread schedules until one fails, then use tooling to compare file outputs between a passing and failing run, identify which process wrote the differing bytes and when, and use debug symbols to show the exact source line and call stack at that step. In one case this process, using commands like rewind check, rewind diff, rewind blame, and rewind where, took three commands to pinpoint a single line in git's notes-utils.c. daily.dev surfaces write-ups like this for engineers building reproducible debugging workflows."}}]} ```