Tailscale Traces Database Corruption to a 16-Year-Old SQLite Bug

Toward the end of last year, Tailscale's uptime got shaky. The culprit wasn't infrastructure — it was a 16-year-old bug buried inside SQLite, one the team only unearthed after six months of forensics and 19 separate incidents of database corruption.

On August 12, 2026, Tailscale published a detailed postmortem titled "How we tracked down a 16-year-old SQLite bug." This is not a story about a startup misusing SQLite. Tailscale ran SQLite exactly the way its documentation recommends — single-writer, one process per database — and still hit a bug so rare that SQLite's own authors estimate it had been lurking for at least 16 years.

Tailscale's Database Architecture

Tailscale's control plane is split into coordination servers, called "shards." Each tailnet lives on one shard at a time, and each shard owns a single SQLite database accessed exclusively by one Go process. That is precisely how SQLite is meant to be run.

Tailscale adopted SQLite in 2022 because it is "boring technology" in the best sense — stable, ubiquitous, battle-tested. Their backup pipeline snapshots the entire database file every few minutes and uploads it to S3, and that setup ran without incident from early 2023 until August of last year.

In August 2025, a pipeline reading those backups reported an error. PRAGMA integrity_check confirmed the database was corrupted. They repaired it, investigated, found nothing — and then it happened again. In total: 19 corruption incidents over six months.

The corrupted data was only tailnet and device metadata, never private keys or network traffic. But every incident forced them to stop the control plane process on that shard to repair or restore the database, cutting off new device connections and admin-console access. Early recoveries took over an hour.

A Hunt With No Pattern

What made this bug so hard to catch was its lack of a pattern. It wasn't tied to a specific shard, customer, feature, time of day, or load level. Incidents landed hours apart, then weeks apart — including a six-week quiet stretch from October to December 2025.

Unable to reproduce the bug synthetically, Tailscale deployed passive, forensic telemetry into production to catch it in the act. They also bought a professional support contract from the SQLite developers — a decision they later credited as the turning point.

Together they enumerated theories: POSIX locks canceled on close(), mismanaged memory, or using SQLite from multiple threads with thread safety disabled. Each was systematically ruled out as new incidents arrived.

The Clue: Transactions That Didn't Bark

While hunting, Tailscale still had a platform to run. They built a transaction-logging pipeline that streamed every data-modifying SQL statement to a separate log. Because SQLite is a single-writer database with serializable transactions, their history was perfectly linear and could be replayed to restore a database from the last known-good backup.

The pipeline worked — and then it did something better: it exposed a clue. In two incidents, the logs failed to replay cleanly. Data written and committed by one transaction was inexplicably invisible to later ones. A write had vanished without raising an error. That should be impossible.

Suspicion fell on the checkpoint process. With WAL (Write-Ahead Logging) enabled, new pages are written to the WAL file first, then copied back into the main database file during a process called checkpointing. Tailscale controls checkpoints manually to produce fast, consistent backups — a non-standard setup compared to most users.

During corruption incidents, one metric stood out: SQLite reported copying more pages from the WAL than actually existed. If the WAL has 10 pages and 20 get copied, something is clearly wrong.

The WAL-Reset Bug

The SQLite authors wrote a new debugging tool — tmstmpvfs, a shim wrapping the virtual-filesystem layer to emit extra tracing. Tailscale funded it, deployed it to production, and waited. The next incident arrived sooner than expected.

The traces pinpointed the cause: a rare data race between a checkpoint and a write transaction. If a write lands at a specific moment during a checkpoint, the checkpoint process believes some pages have been copied from the WAL into the main database file when they haven't. Those pages are never written, the data is permanently lost, and the database becomes corrupt because other pages — like an index — still reference them.

The SQLite authors named it the "WAL-Reset bug" and estimate it existed for at least 16 years, ever since WAL shipped in 2010. It survived that long because it was so rare the team had to add code to deliberately trigger it in testing. The fix adds a check to the checkpointing function that detects when the WAL has been reset by another thread.

Tailscale hit it harder than most precisely because of how they run SQLite: they control checkpoints manually and checkpoint aggressively, so a rare condition was bound to catch up with them eventually.

A Fix, and a False Alarm

The fix shipped in SQLite 3.52.0. Tailscale rolled it out cautiously — a few canary shards first, then the rest. Their backup monitor immediately turned red, reporting corruption across 13 databases. But it was a false alarm: those databases weren't really corrupted, just caught in a second bug around stale expression indexes.

The trigger: Tailscale stored high-precision timestamps as text, then converted them to floats in a VIRTUAL generated column. The 3.52.0 release that fixed the data race also changed text-to-float rounding behavior, leaving indexes with mismatched values that integrity_check flagged as corruption. Their canary shards happened not to hold timestamps that hit the new rounding, so the bug slipped through the phased rollout.

SQLite withdrew 3.52.0 and published 3.51.3, containing only the WAL-Reset fix. Later, 3.53.0 added a self-healing index feature to close out that class of bug.

Tailscale still wouldn't declare victory — an absence of incidents doesn't prove a fix, as their six-week false calm had shown. They patched their driver to log a warning whenever a checkpoint and a WAL-reset overlapped, then waited. Two months later, the alert finally fired — proof that the exact conditions for the bug do occur in production, and that the fix had saved them from yet another corruption. As of writing, Tailscale has run four more months without a single database incident.

Key Takeaways

• The bug is real, and it may be in your code too. The WAL-Reset bug lived in SQLite for at least 16 years. If you use WAL with manual checkpoints, upgrade to at least 3.51.3.
• "Boring technology" is only safe on the well-trodden path. Tailscale ran SQLite correctly except for one thing: manual checkpoints. That deviation pushed them into territory nobody had stress-tested.
• Replay logs are a debugging tool, not just a recovery tool. Tailscale built transaction logging to restore data, but it was the replay that surfaced the decisive clue.
• Paying experts is an investment. The SQLite support contract, plus funding the tmstmpvfs tool, dramatically shortened an investigation into the heart of the database engine.

---

The bigger lesson here isn't about SQLite; it's about operating systems at scale. Every "stable" technology has a blind spot, and blind spots only surface when someone runs enough load, at enough scale, and digs all the way to the bottom. Tailscale paid with six months of shaky uptime — but the entire SQLite community gained from a bug that had been sleeping for 16 years.

---

Content assisted by AI (Amy 🌸). Reviewed by the author.