Repository navigation
fix(sessions): back off history passes when retained ingest is Blocked instead of re-driving a broken store every wake - #4573
Conversation
A Blocked outcome (for example projection_store_corrupt, retryable=false upstream) warned and continued, so every wake re-ran a full pass that read the same permanently broken store before failing again — 9,241 worker-hog warnings over two days on one machine, on top of a corrupt 1.6 GB sessions.db re-read at full cadence. History passes now back off exponentially, 1 min doubling to a 1 h cap; the first Blocked outcome logs one error-level event with the reason codes and the retry schedule, later ones a bounded warn. Any pass that makes progress resets the backoff. Projections keep their cadence. Scheduler half of ScriptedAlchemy#4569; the relational-store rebuild protocol stays tracked there.
|
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
A pass where one provider advances while another stays blocked reports
Blocked { made_progress: true }; resetting the backoff on progress let
the blocked store be re-read at full wake cadence again. The backoff
now clears only on an outcome with no blocker.
|
Addressed in the follow-up commit: a |
96fe61b
into
ScriptedAlchemy:master
Scheduler half of #4569. Related: #4481 (the
registry.rsworker-hog load), #4564, #4567.Problem
run_session_temporal_refresh_schedulertreated aBlockedhistorical-ingest outcome as warn-and-continue.Blockedis terminal per host (e.g.projection_store_corrupt,retryable=falseupstream), so every wake re-ran a full pass that read the same permanently broken store before failing identically.Measured on one machine over two days, against a corrupt 1.6 GB
sessions.db:retained historical session ingest is blocked reason_code="claude:projection_store_corrupt,cursor:projection_store_corrupt"(plus 68× each across four host catch-up paths)native task held its worker for one poll past 100msfromregistry.rsspawn sites — the re-driven passes are the CPU costChange
Blocked: 1 min doubling, capped at 1 h. Projections keep their cadence; only the history pass is suppressed.Blockedoutcome logs oneerror-level event with the reason codes and the retry schedule; subsequent ones log a bounded warn with the attempt number.The relational-store rebuild protocol (the real fix for a corrupt
sessions.db) stays tracked in #4569 — this PR removes the unbounded re-drive cost, not the corruption.Tests
tracedecay-session-runtimescheduler tests: 49/49 green (--features tracedecay-rusqlite-runtime/test-runtime; without the feature these tests fail on cleanmastertoo — cargo harness lacks the process runtime, CI runs the Bazel targets).cargo checkclean.