Repository navigation
fix(sessions): stop repeating permanent ingest and hook failures - #4577
Conversation
|
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Deterministic per-item failures were re-driven forever: the in-process source ledger forgot every converged source at each daemon restart, the cover-past cursor advance was the only convergence mechanism for refused records (and a non-retryable advance failure re-failed the whole file every pass), and owed hook replay work was redriven every sweep with no backoff or cap. - pi: durable per-path completion and header-quarantine memos keyed by (len, sub-second mtime); an in-process discovery skip set for malformed headers; unchanged converged files never re-enter admission - shared JSONL admission (pi, codex, kimi, vibe, cursor): a durable, generation-scoped refusal high-water memo acts as a never-regressing secondary frontier; a non-retryable cover-past advance failure now advances in-memory with one warning instead of failing the file; repeats below the memo frontier log at debug - hook v2 replay: per-envelope exponential backoff (30 s to 1 h) and an in-process terminal cap after 16 attempts, with one terminal warning naming the envelope; the durable ledger is untouched Fixes the non-converging loop family behind #4481 and #4564. Co-authored-by: spa5k <79936503+spa5k@users.noreply.github.com>
- refusal memo floor only applies with a resume fingerprint, so the scanner validates the skipped prefix; a rewritten file diverges into committed-checkpoint recovery instead of skipping changed bytes - the floor re-baselines the in-memory cursor so later frames advance contiguously; the cover-past fallback warns once per pass - path memos bind the unix inode, closing the equal-length timestamp-preserving replacement case - header quarantine (in-process and durable) records only deterministic InvalidSessionHeader failures; SessionHeaderUnavailable stays retryable Co-authored-by: spa5k <79936503+spa5k@users.noreply.github.com>
The parse-offset writer is monotonic on mtime, and a prefix fingerprint is unordered: a later refusal with a smaller fingerprint value wrote as a no-op and stranded the durable floor behind the in-memory one. The memo now persists via the exact compare-and-swap replace, which orders on nothing but the (generation, offset, fingerprint) pair it carries. The coverage-fallback once-per-pass flag was inverted: mem::take reset it to false on every frame, warning per frame after all. mem::replace sets it, keeping the warn to the first fallback of the pass. Co-authored-by: spa5k <79936503+spa5k@users.noreply.github.com>
A get_parse_offset read failure is not an absent row: fabricating a zero expected value CAS-fails against a real row and buries the read failure inside a replace conflict. The read failure now keeps the in-memory mark, surfaces its outcome, and defers the persist. Co-authored-by: spa5k <79936503+spa5k@users.noreply.github.com>
Co-authored-by: Zack Jackson <25274700+ScriptedAlchemy@users.noreply.github.com>
Use canonical rewrite witnesses and scoped compare-and-swap for Pi completion and invalid-header decisions. Keep quarantined coverage partial, respect scan interruption, and recheck the current project at dequeue. Bind Pi header scope to the admitted JSONL prefix before any mutation. Retain first-frame evidence in the existing prefix digest without changing durable fingerprints, including resumed and zero-frame pages. Keep hook replay deferrals visible and retryable for the consumer lifetime. Let the canonical ledger settle or expire work and prune retired gate state. Add real-path regressions for rewrites, stale CAS, scope races, interruption, and recovery after more than sixteen unavailable producer attempts.
c4ba0e2 to
34f8eb2
Compare
Exercise legacy refusal offset generations and missing fingerprints through the current Codex admission seam. Verify committed coverage advances, remains stable on replay, and changes generation after an in-place rewrite. Replace the stale RefusalMemo-only test whose production type was removed by the canonical cursor cutover.
Clippy's assert_is_empty rejects assert!(…is_empty()); assert_eq shows the unexpected failures when the check trips. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # crates/tracedecay-sessions/src/runtime/hosts/pi.rs # crates/tracedecay/src/daemon/hook_v2_replay_consumer.rs
Production registers a project's hook orchestrator once a complete code-index generation exists, so admitted work owed to an unmounted orchestrator is waiting, not failing. Keep it owed with the same backoff, but publish it as a new hook_replay `waiting` status; genuine admission failures still publish `failed`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Devin Review found 3 new potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| /// The spools drained, but producer work they admitted waits for the | ||
| /// project's hook orchestrator to mount. The work stays owed and runs | ||
| /// once it does; nothing has failed. | ||
| Waiting { | ||
| waiting: Vec<StatusHookReplayFailureV1>, | ||
| }, |
There was a problem hiding this comment.
🔴 Waiting status breaks SDK status reads
When replay reports waiting, StatusHookReplayV1 has no matching variant in the generated SDK schema. The SDK decoder rejects the entire project status response.
Learn more
The TypeScript SDK decodes status responses against its generated StatusResultV1 schema. That schema references a StatusHookReplayV1 definition with four variants, and decodeCanonicalSchema rejects any other status. A project with owed orchestration work now sends a fifth variant, so an otherwise healthy status read fails decoding. The repository requires regenerating wire contracts after changing a Rust type that crosses the public API boundary.
Example: An admitted edit remains pending before the orchestrator mounts. The daemon sends hook_replay: {"status":"waiting","waiting":[...]}; an SDK status request rejects this response instead of returning the project's status.
Recommended fix: Regenerate and commit the canonical TypeScript SDK status schema and types using the dashboard contract generation workflow, then verify contracts:check and exercise a waiting status through the SDK decoder.
Was this helpful? React with 👍 or 👎 to provide feedback.
| Some(Some(sweep)) if !sweep.failures.is_empty() => StatusHookReplayV1::Failed { | ||
| failures: sweep.failures.clone(), | ||
| }, | ||
| Some(Some(sweep)) if !sweep.waiting.is_empty() => StatusHookReplayV1::Waiting { | ||
| waiting: sweep.waiting.clone(), |
There was a problem hiding this comment.
🟡 Failed sweeps conceal waiting producer work
If one spool fails while another host awaits orchestration, hook_replay_status returns only sweep.failures. The owed work disappears from project status until every unrelated spool failure clears.
Learn more
drain_all_hosts visits every host and can add failures for one spool while redrive_pending_work adds waits for another. A published sweep retains both lists, but the Failed branch returns only failures. Clients cannot observe the otherwise valid pending obligations while any unrelated failure remains.
Example: Claude's pending edit awaits the orchestrator and Kimi's receipt spool cannot be read. The sweep records one waiting entry and one failure; status reports Kimi's failure and says nothing about Claude's owed work.
Recommended fix: Extend the failed wire variant to include its waiting entries (or use a shared aggregate status shape), regenerate dependent contracts, and test mixed failure/waiting sweeps.
Was this helpful? React with 👍 or 👎 to provide feedback.
| HookV2AdmissionOutcomeV1::Admitted { | ||
| orchestration: HookOrchestrationAdmissionV1::Unavailable, | ||
| .. | ||
| } => Some(AWAITING_ORCHESTRATOR), | ||
| HookV2AdmissionOutcomeV1::Unavailable => Some("was unavailable"), |
There was a problem hiding this comment.
🟡 Broken orchestrators appear to be waiting
When a mounted orchestrator stops accepting work, unresolved_pending_work classifies its Unavailable result as awaiting mount. The runtime also returns Unavailable after task failure, so project status hides the failure.
Learn more
HookOrchestrationAdmissionV1::Unavailable does not identify why the orchestrator refused admission. admit_registered_hook_orchestration returns it when no runtime is registered, while admit_with_cancellation also returns it for a registered runtime whose task owner failed or stopped accepting work. The classifier maps both to a healthy wait, and the Waiting wire comment promises that work will run after mounting, even though a mounted failed runtime does not remount in that path.
Example: A project's orchestrator task panics and the next hook is admitted to the ledger. Redrive receives nested Unavailable; project status says waiting rather than identifying the failed orchestrator.
Recommended fix: Preserve a distinct no-registration outcome from the orchestration authority and mark only that state as awaiting mount; report registered-runtime failures as failures while leaving their obligations retryable.
Was this helpful? React with 👍 or 👎 to provide feedback.
Why
#4567 was merged independently while its writable repair branch was being verified. Its Pi completion shortcut can miss preserved-mtime rewrites, invalid headers can bypass durable reconciliation, and a process-lifetime hook retry cap can suppress recovery from transient prerequisites. This PR follows that merge with the reviewed canonical source-evidence and replay corrections.
Changes
Validation