Repository navigation
fix(storage): bound the hard WAL drain behind a stuck reader - #4164
Conversation
A hard-pressure checkpoint drain waited for every reader to release its WAL read slot, so one reader that never did kept the drain pending and general writes refused indefinitely (a 229 MB WAL held 6.8 hours on 2026-10-16). The drain now runs the design's bounded fence. Past one read deadline it revokes this pool's older snapshots: the worker interrupts the running statement and rolls the read back. The fence lasts at most 6 s; after that the holders are named (reader id, or another process from /proc), writes resume, and retries back off from 1 s to 60 s. Past twice the hard limit writes wait again, so the WAL stays bounded. Status reports fence_expired and Doctor a WalCheckpointPinned storage finding naming the holders.
|
| /// After the fence expired, the checkpoint after each batch waits for the | ||
| /// backoff instead of retrying on every commit. | ||
| pub(crate) fn defers_scheduled(&self, now: Instant) -> bool { | ||
| matches!(self.drain, Drain::Expired { retry_at, .. } if now < retry_at) | ||
| } |
There was a problem hiding this comment.
🔴 WAL write ceiling stays unenforced during backoff
After a fence expires, defers_scheduled skips checkpoint samples until the retry deadline. blocks_general uses the last sampled WAL size, so writes can exceed twice the hard limit throughout a 60-second backoff.
Learn more
The writer samples WAL size in evaluate_interruptible. The expired-drain policy skips that call until retry_at, while blocks_general reads wal_bytes stored on the last checkpoint attempt. Each committed batch can grow the WAL during this interval. The writer continues processing batches because the stale value remains below twice the hard limit, even when the physical WAL crosses it.
Example: With a 256 MiB hard limit and a pending 260 MiB WAL, an expired retry enters a 60-second backoff. Writes adding another 300 MiB commit before the next retry; the cached 260 MiB still admits them past the intended 512 MiB ceiling.
Recommended fix: Continue sampling WAL size after commits during expired backoff without running a RESTART checkpoint, and update the drain's cached size and published pressure when the ceiling is reached. Keep the expensive restart on its backoff schedule.
Was this helpful? React with 👍 or 👎 to provide feedback.
| /// A store whose WAL checkpoint cannot restart the log because a reader | ||
| /// still holds a read slot past the checkpoint fence: a wedged reader of | ||
| /// this daemon or another process, named in the evidence. Writes run | ||
| /// until the WAL reaches its ceiling, then wait. | ||
| WalCheckpointPinned, |
There was a problem hiding this comment.
🟡 Dashboard rejects pinned WAL findings
When Doctor emits WalCheckpointPinned, the dashboard decoder rejects its unrecognized storage_kind. The generated schema lacks the new variant, so clients cannot display the finding.
Learn more
Doctor serializes its typed storage finding kind through DoctorReportEntryV1 into the dashboard API. The generated DoctorStorageFindingKindV1Schema only accepts the six existing values; its generated TypeScript union likewise lacks wal_checkpoint_pinned. The dashboard status list STORAGE_KINDS also omits it.
Example: A store emits a pinned WAL finding with storage_kind: "wal_checkpoint_pinned"; the current browser decoder rejects that response rather than showing the blocked store.
Recommended fix: Add the variant to STORAGE_KINDS, then regenerate the dashboard contracts through contracts:generate and verify them with contracts:check. Do not edit generated files manually.
Was this helpful? React with 👍 or 👎 to provide feedback.
| let holds = descriptors.flatten().any(|descriptor| { | ||
| std::fs::read_link(descriptor.path()).is_ok_and(|target| targets.contains(&target)) | ||
| }); | ||
| if !holds { | ||
| continue; |
There was a problem hiding this comment.
🟡 Idle processes falsely named as WAL blockers
When foreign_wal_holders scans an expired fence, it names any process with the database open. An idle connection has no WAL read slot, so Doctor can blame the wrong process.
Learn more
SQLite keeps database and sidecar descriptors open across transactions. A file descriptor proves only that a process has opened the file, not that it currently holds the WAL reader mark preventing RESTART. run_scheduled_checkpoint publishes this inventory as holders when the pool reports no snapshots. Doctor then presents it as the pinning process.
Example: An idle sqlite3 process has the database open while a different process owns a long-lived read transaction. Both hold descriptors; the scan can name the idle process as the WAL blocker.
Recommended fix: Distinguish candidate open connections from proven read-slot holders in the telemetry and Doctor wording, or obtain a reliable reader-lock attribution before reporting a process as the blocker.
Was this helpful? React with 👍 or 👎 to provide feedback.
Step 6 of the storage redesign (
docs/plans/storage/README.md), first slice, and the coordinator task "Fix SQLite WAL hard drain that can pend forever" (the #4097 class).A hard-pressure WAL drain used to wait for every reader to let go. A reader that never did kept the drain pending forever: on 2026-10-16 a
project_sessionsWAL sat at 229 MB through 245,264 retry wakes (about 6.8 hours) while every general write was refused. The writer's drain now has the bounded fence the design specifies (section 3), on today's writer:SnapshotExpired. The log restarts and queued writes commit./proc/<pid>/fdlinks to the database or its-wal/-shm). The fence drops, writes resume, and the checkpoint retries with backoff from 1 s to 60 s instead of every 100 ms. Past twice the hard limit (512 MiB at the default) writes wait again, so the WAL stays bounded and nothing spins.CheckpointPressure::FenceExpired; status shows it asfence_expiredwithforeign_connectionholders, and Doctor reports a typedWalCheckpointPinnedstorage finding naming the holders and the WAL size.The policy is a pure state machine (
Drain { Open, Fenced, Expired }incheckpoint/controller.rs) so step 7 can move it onto executor lanes unchanged. README "Implementation reconciliation" anddecisions.tsvrecord the constants and calls.Test plan
repository::attachment::tests::a_hard_drain_converges_behind_a_reader_that_never_releases: a snapshot that is never released, a WAL past the hard limit, a queued write. The write commits and the drain completes (pressureOpen, WAL below soft) while the snapshot is still held, and the snapshot's next read fails. Mutations: removing the revocation call, or the worker's rollback, each fail it ("the queued write is not held until the reader lets go: Timeout").checkpoint::tests::a_hard_drain_fence_expires_backs_off_and_blocks_only_past_the_write_ceiling: every drain transition, the 2/4/8/16/32/60/60/60 s backoff, and the write ceiling. Mutation: an expired fence that keeps blocking writes fails it.checkpoint::foreign::tests::a_process_holding_a_wal_sidecar_open_is_nameddoctor_kernel::tests::an_expired_wal_checkpoint_fence_is_a_finding_naming_its_holder,store_runtime::tests::checkpoint_wire_names_the_process_holding_an_expired_fencerusqlite_suite,contracts_suite//crates/tracedecay:unit_test,:daemon_suite,:mcp_suite,:session_suitebazel build --config=clippyandrustfmt_check_2024on touched crates