--- title: "How Tailscale helped find the SQLite WAL-Reset bug" url: https://daily.dev/posts/how-tailscale-helped-find-the-sqlite-wal-reset-bug-fmdss8i8l source_url: https://tailscale.com/blog/sqlite-wal-reset-bug type: article source: "Tailscale" published: 2026-08-12T14:06:28.669Z updated: 2026-08-24T07:10:31.522Z tags: ["sqlite", "tailscale"] reading_time: 15 upvotes: 2 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How Tailscale helped find the SQLite WAL-Reset bug **[Tailscale](https://daily.dev/sources/tailscale)** · 15 min read · 2 upvotes · 0 comments ## Summary Tailscale traces a series of database corruption incidents affecting its control plane back to a rare data race in SQLite called the WAL-Reset bug, present for at least 16 years. Working with SQLite's core developers over months, they built forensic tooling including a custom virtual filesystem shim to catch the bug in production, leading to a fix in SQLite 3.51.3. The rollout also surfaced a second, unrelated bug involving stale expression indexes and floating-point rounding changes, which caused false corruption alarms and forced SQLite to withdraw version 3.52.0. Tailscale details its aggressive manual checkpointing configuration as the root cause of exposure to the otherwise extremely rare race condition. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the SQLite WAL-Reset bug and which version fixes it? The WAL-Reset bug is a rare data race between a checkpoint operation and a write transaction in SQLite that causes pages to be silently lost, corrupting the database file. It had existed in SQLite for at least 16 years but was rare enough to require deliberately triggering it in test environments. SQLite developers fixed it in version 3.51.3, after an initial fix in 3.52.0 was withdrawn due to an unrelated regression. _Track SQLite release notes on daily.dev before upgrading production databases past a known corruption fix._ ### Why was SQLite 3.52.0 withdrawn shortly after release? SQLite 3.52.0 was withdrawn because, alongside its WAL-Reset bug fix, it changed the rounding behavior of text-to-floating-point conversions, which caused stale expression indexes on computed columns to report false corruption via PRAGMA integrity_check. SQLite developers replaced it with 3.51.3, containing only the WAL-Reset fix, and later added a self-healing index feature in 3.53.0 to prevent the stale index issue. _Developers evaluating risky point releases can follow SQLite version changes and regressions on daily.dev._ ### Why is manually controlling SQLite's checkpoint process risky? Taking manual control of checkpointing and running it aggressively, as Tailscale did to support fast backups, increases exposure to rare internal race conditions like the WAL-Reset bug because it deviates from SQLite's standard, heavily-tested automatic checkpoint scheduling. Most SQLite deployments never encounter this issue since they rely on the default checkpoint behavior rather than a custom, high-frequency manual trigger. _Engineers weighing non-standard database configurations can compare trade-offs and real incidents on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 2 discussions and 82 comments across hackernews, lobsters (as of 2026-08-24). **TL;DR:** The writeup is widely praised as an excellent, transparent postmortem, with particular admiration for Tailscale funding SQLite's core developers to help find and fix the bug; discussion branches into tangents about SSO/identity, single points of failure in distributed systems, and the technical mechanics of the WAL-reset race. **Sentiment:** 65% positive · 25% mixed · 10% skeptical **The case for** - Praised as a rare example of a company paying to fund open-source maintainers and tooling rather than just consuming their work for free. - Seen as a well-written, technically deep and satisfying postmortem. - Admiration for the persistence needed to track down a 16-year-old, nearly unreproducible race condition. **The pushback** - Some found parts of the explanation of page-copying counts internally inconsistent or confusing. - Questioned why an automated way to trigger the bug wasn't built sooner to reduce uncertainty during the fix period (though this was later clarified as already done). - One thread noted the release notes downplayed that this was discovered via 19 real production corruption incidents rather than theoretical/testing discovery, giving a misleading impression of how the bug was found. **By community** - hackernews (positive): Overwhelmingly appreciative of the writeup and Tailscale's decision to fund SQLite's developers, with side discussions on SSO, distributed systems tradeoffs, and the technical details of the bug. - lobsters (mixed): Appreciates the postmortem but flags that the original release notes were misleadingly evasive about how the bug was actually discovered (via real production corruption, not theoretical testing). **Hottest debate:** Whether Tailscale's per-shard control plane design constitutes an acceptable single point of failure or a design flaw, with some arguing the alternative (full Byzantine/consensus resilience) would be worse for this kind of system. **Open questions** - Why checkpoint so aggressively in the first place, and how does that tradeoff compare to something like etcd's snapshot frequency? - Whether this bug or the checkpointing pattern disproportionately affected other tools like Litestream that also intervene in the checkpoint process. - Whether static analysis or a memory-safe language could have caught this class of data race. **Highlights** > That constraint is the part I find most interesting here. The wal-index lives in the -shm file, which SQLite never really uses as a file: clients mmap it and treat it as shared memory, and access to it is coordinated through xShmLock rather than ordinary file locks. The race needs two connections because it needs that shared coordination layer to exist at all. It also hints at why it could hide for sixteen years. Almost everything below the pager can be swapped out through the VFS interface, and there are plenty of unusual VFSes exercising those paths. The shared memory methods are the exception. WAL normally requires xShmMap, xShmLock, xShmBarrier and xShmUnmap, and unix and windows are effectively the only two implementations of them that see real traffic. Everyone else opts out rather than implementing them, because SQLite documents an escape hatch: set locking_mode=exclusive before the first access and the wal-index is kept in heap memory with no shm file at all. That is the road the browser builds take. The WASM build has no shared memory APIs, so WAL on an OPFS database is only possible in exclusive mode, and the docs are blunt that this removes all concurrency in exchange. So the alternative VFS world contributes close to nothing to the coverage of the exact code path this bug lived in. Everyone who might have been a third implementation stepped around it instead, which leaves finding it to someone on unix doing something unusual with checkpoints. > — [maitrungduc on hackernews · 1 comments](https://news.ycombinator.com/item?id=49274604) > This is maybe one of the purest examples of the Bell Curve Meme in software engineering. The things you would do to the system Tailscale operates to eliminate all single points of failure (generally, and in the specific case where, where the "single point of failure" applies only to a small cohort of customers) would make the system less resilient, and increase failures. Generally, you do complex distributed systems without on-paper single-points-of-failure anywhere when you absolutely have to, because those systems don't have transient failures. That's not mesh networks like Tailscale at all. As always: https://how.complexsystems.fail/ > — [tptacek on hackernews](https://news.ycombinator.com/item?id=49276068) > Odd not to highlight the sentence where they answer the obvious question "Why Tailscale in particular?": > They also explained why we were more likely to hit the bug than other SQLite users: we take manual control of the checkpointing process, and we checkpoint very aggressively. Even a bug triggered by a rare condition was bound to hit us eventually. > — [Ariarule on hackernews](https://news.ycombinator.com/item?id=49274270) > When the release notes wrote > On 2026-03-03, one of the SQLite developers (Dan) found and fixed a bug that could, in rare cases, lead to database corruption. and > The developers have never been able to reproduce the bug organically and had to add special testing logic to SQLite that deliberately triggers the circumstances of the the bug in order to verify that the issue has been fixed. I very much had the impression that this was a nearly-theoretical race condition discovered by formal methods or some kind on intensive testing. There was no mention that this was discovered by a user who experienced 19 prod corruption incidents, but I guess I missed the implication of: > the problem has never been observed during development and testing > — [pushcx on lobsters · 1 points, 1 comments](https://lobste.rs/s/e0lkmi/how_tailscale_helped_find_sqlite_wal#c_tcersi) > This was really, really interesting - what a triumphant adventure. A few (very, very, very pedantic) things that stood out: > We wanted a way to restore service that didn’t involve rolling back to the last known-good backup (which would lose a lot of data) or repairing the known-corrupted database (which was potentially risky). (Emphasis mine) - it would be "risky", not "potentially risky" - then the "calculated risk period" starts and it's "potentially problematic". In the SQLite report[0] (11.2) I wish they downplayed this less - a mention of the rarity, then technical details - I'm friendly with a few of the devs/previous-devs, have the utmost respect for their skill and accomplishments (and by extension, faith that the developers I do not personally interact with are also excellent), appreciation and fondness for the huge accomplishment that is SQLite, and on and on... this is world-class work. Maybe section 11.2 wasn't really aimed at me, or I'm too critical. To be fair to all involved, what a minor quibble for such an interesting problem/fix. I hope this comment isn't a fly in the ointment. Last bugfix point[1] - ugh. What a sinking feeling that must've been to deploy a fix then be flooded with not-green - and a lesson[2] against smuggling other changes in a changeset "just because we're already here"? Happy it turned out non-catastrophic, but did result in a rare (not remembering other instances of top of head) recall[3] from SQLite. That it was throwing errors at the same time SQLite and Tailscale were testing the other WAL-issue bug must've upset some stomachs for a moment. [0] https://sqlite.org/wal.html#the_wal_reset_bug [1] https://tailscale.com/blog/sqlite-wal-reset-bug#fixed-with-a... [2] Nobody conceptually learned anything here - we're all just reminded of what we know: that sometimes "perfect storms" do actually occur. [3] https://sqlite.org/releaselog/3_52_0.html > — [bch on hackernews](https://news.ycombinator.com/item?id=49275212) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49272832) · 448 points · 80 comments - [lobsters](https://lobste.rs/s/e0lkmi/how_tailscale_helped_find_sqlite_wal) · 30 points · 2 comments --- Tags: [#sqlite](https://daily.dev/tags/sqlite), [#tailscale](https://daily.dev/tags/tailscale) [View this post on daily.dev](https://daily.dev/posts/how-tailscale-helped-find-the-sqlite-wal-reset-bug-fmdss8i8l)