UP-03: Restart and recover safely #109

Activity

1. andersalm

   added this to the 0.7.1 milestone on Sep 29, 2026
2. andersalm

   self-assigned this

   on Sep 29, 2026
3. andersalm

   added a parent issue on Sep 29, 2026

   • G1 Install and update: Signed update #89
4. andersalm

   marked this as blocked on Sep 29, 2026

   • UP-02: Stage one complete release #108
5. andersalm

   marked this as blocking 2 issues on Sep 29, 2026

   • UP-04: Add the System update flow #110
   • UP-07: Accept updates on Mac, Linux and Jetson #113
6. andersalm commented on Sep 30, 2026

   

   andersalm

   on Sep 30, 2026

   ContributorAuthor

   More actions

   UP-03 owns automatic restart after the exit-75 self-replacement observed in UP-01 proof, candidate 47658723 / PR #132.
   The accepted recovery used an explicit CLI retry and explicit startup. Automatic restart, child cleanup, bounded readiness and rollback still require this issue's installed journey proof after #108 supplies complete-set staging.
   Reuse the UP-01 recovery receipt as input; it proves data preservation and explicit recovery only.
7. andersalm commented on Oct 3, 2026

   

   andersalm

   on Oct 3, 2026

   Last edited by andersalm

   ContributorAuthor

   More actions

   • UP-03: Restart and recover safely #109 PR227 stays draft for the approved PR230 union. Published 37cbb74d includes the Mac-only cache correction and gateway fixture repair, normally integrates current develop, and matches API, fetched pull/head, remote and local HEAD; origin 0/0. Uncommitted updater-owned bootstrap and Mac hop harness drafts now use the owned worktree; current published proof remains bound to the unchanged commit.
   • Separate append-only commits b111a7f1 and 90d44c2a address the Release review findings on failed-start child ownership and alternate-binary recovery. The same independent high rounds and f324 local Opus PASS the repaired source; f3244aa4 supplies the private backup parent needed by the positive migration fixture. Every refusal case remains.
   • Exact f324 local format, fresh workspace all-target check (39.324s), common/server all-target Clippy (49.350s), 514 unique changed-module Rust cases and 170 JS/Python cases PASS. The extra exact migration rerun also PASS. All 40 gate log hashes match and 41 recorded PID/groups are absent. Earlier fixture and source failures retain their exact cause receipts.
   • Fresh exact f324 native N/N+1 builds and the isolated signed install / authenticated initial Home / plain Carrier check, apply, repeat / all 12 refusals PASS. N SHA-256 6e694ab12940d8ef4b7cb9a81ccf9aedbede0d1f8088cae55f9c3631c0a9f372; N+1 990fb8334406677eefd7ed2cbd7703dfa998fe9fcd681cd2087a4641538c82b9; installed result 1d8d161b82098723655e6ec3432cf21a538f89c3afe941536b7f7b0ff166236b. Install/check/apply/repeat took 3.305/2.235/6.665/0.044s, with four empty successful stderr streams, 17 clean holders, zero M2 HTTP fallback and preserved config/data/frozen support. Independent receipt audit PASS; 57 observer and 3 build processes ended, PID/group census is empty and the lease is free.
   • Anders limits fresh CI builds to Mac N/N+1 receipt and Release holder package jobs; all other job caches match develop. The failed bff pre-push gate proved a partial readiness-JSON race. Append-only 37cbb74d atomically publishes that marker and stops the owned child before propagating test assertions. Controlled visible-marker failure / delayed pending-marker success, six gateway module passes / one ignored, format, server all-target Clippy, independent high source review and local Opus PASS; all 41 proof PIDs / 12 groups ended. The repaired actual-ref full gate and normal push PASS749.697s: nine workspace all-target checks, common/runtime/server all-target Clippy, 2,983 parent units / 26 ignored / zero failures, 92 product-data cases and three fresh provider builds. All 200 observed PIDs / 48 groups ended and the stable lease is free. The unchanged 59 gate fixtures retain historical source-bound proof. Current CI is terminal GREEN on exact37 with10 successful jobs and dispatch-only MacCLI/publication skipped; current API head matches37. Root and the independent Sol high audit now PASS the compact current Mac artifact: exact tested merge parents/tree, fresh same-source N/N+1 intermediate domain, signed install, authenticated initial Home, Carrier check/apply/repeat, all12 preserved-state refusals,17 clean holders and57 observer/3 builder processes with empty cleanup. Its receipt binds the binary and manifest hashes; raw binaries stay outside the compact archive. Desktop opening is suppressed, so operator UX and the exact union installer-rerun journey keep their acceptance gates. The superseded automatic holder is terminal canceled; Release PR232 owns its replacement. Current exact-head posted Opus FAIL repeats the installer rerun after private-pair migration dependency. Release owns complete writer-lock/pending-fence/preflight adoption and exact migrated-N to N+1 rerun/interruption proof in PR230; Update retains consumed-pair authority and refusals. All nine observed review PIDs / one group and the exact temporary checkout ended. Earlier f324 native PASS covers initial install/update. Operator owns #229 from14:20UTC; Update preserves its source/history and finishes227. Current union CI, development/Opus and operator Home/account/model/update acceptance remain. Irzhy follows up after task merge. The current union installer blocker confirms the same UP-02: Stage one complete release #108/UP-03: Restart and recover safely #109 writer-adoption gap. Update prepares the shared four-input bootstrap adapter draft; Release wires the shell and owns installed proof. The draft keeps existing five-entry contracts and all authority/refusal checks; local tests, owner coordination and publication gates follow source review. The bootstrap draft is now frozen with seven Update-owned Rust files and 17 added unit cases. Root and the independent high reviewer matched every file hash; root diff check PASS and direct installed rustfmt completed as a source edit. The same development round is interim FAIL on one remaining P1: atomic host-lock file admission. Stopped-controller authority, fresh source-writer serialization, equal SemVer precedence and final admission with owned scratch are repaired in source. Runtime’s checked-file constructor remains coordinated in the existing #217 acknowledgement. Both source agents are terminal; compilation, unit execution, local Opus and installed proof remain ahead of publication. The Mac metadata/default setup extension is now frozen in the same two harness files. Root matched the exact source/diff and all seven unchanged Rust hashes; both source agents are terminal and the same independent high round PASS. Current light proof has111 observer passes,92 product-data passes, data/diff gates PASS and exact original archive admission for all five payload files. The first light wrapper retained111PASS then failed because root used a Home layout for the qualified support data root; corrected remaining checks PASS with source unchanged and the completed observer suite was retained without replay. Both immutable receipts remain. All40 recorded light PIDs/six groups are absent. Current bounded local Opus PASS85.640s on the metadata/setup boundary; both client PIDs/one group ended, exact source/output/diff match and the prior33.052s four-artifact PASS plus180.667s timeout keep their earlier scopes. Release still owns the original archive provenance. Exact native setup proof needs Release/Operator’s clean same-source union candidate, an agreed Release-priority lease and15% reserve including planned growth. The seven Rust bootstrap files retain Runtime’s atomic checked host-lock P1 and17 unrun unit cases. Published37 and its admitted proof stay unchanged; the leaf installer-adoption FAIL keeps the exact union acceptance gate. Anders now assigns the urgent union hop repair directly to Update. The isolated hop-only branch starts at the approved union, has a clean local commit and retains the same source hashes,111 observer/92 product-data passes and same-round high PASS. Update records local build custody on PR230. Exact clean union hop 5c0e4fba / tree 585bbd91 now PASS the local Mac rehearsal: signed M1 install/default setup, authenticated initial Home with opener suppressed, plain M2 check/apply/repeat and all12 preserved-state refusals. N/N+1 bind this source; result SHA-256 d4903db4c1d4f850b9b59abe499f95a5535c0d5af62ab0ee750a36c9252a3313,17 clean holders, zero HTTP fallback/bootstrap errors and156 observed PIDs/55 groups ended. Source-only high, local Opus and independent native receipt admission PASS. The exact-ref pre-push gate FAIL43.808s on unchanged base ipfs-provider unit fixture drop(reader) / clippy::drop_non_drop at2326;64 gate PIDs/34 groups ended, stable lease released and remote d703 stayed unchanged. Update posted the stopped publication and owner default under Anders’s product-code boundary. Operator has since published the verified hop with one unit-test lint correction. Root and the same independent high round admit the exact published hop bytes and sole test-only delta; the updated proof records matching API/fetched head/origin. Update’s clean native source remained preserved one commit behind that published union; Operator owns final union CI/current Opus/merge and accepted handoff. The failed root gate receipt remains immutable. The operator decision retains installer-rerun recovery as a UP-02: Stage one complete release #108 gate before M3 and preserves d703 installed Home acceptance for this fixture-only delta. Product source remains under its owners. At22:15UTC, API and fetched PR230/union refs match cb299ad3 / tree 174349f0; both hop files retain the reviewed bytes. Operator’s added engine setup change has product scope. The compiled Mac receipts continue to prove exact 5c0e4fba; the e2d0b8b publication admission keeps its test-only scope. Release owns renewed installed Home proof and Operator records the review corrections. The current posted Opus review PASSes its stated security scope. The current union CI is terminal GREEN on the current union, and the renewed installed Home workflow is terminal GREEN. Update read run metadata only. Release owns receipt and installed-journey admission; Operator owns final acceptance, merge and handoff. The review leaves installed journeys and detailed transaction/child/publication checks to their existing gates. Update preserves its clean native source two commits behind the union, remote/local divergence2/0, with all owned processes ended. Operator has now merged the union normally into develop and dispatched the key-free package build. Update verified fetched merge parents and exact tree identity, with both the published leaf and tested hop contained. Release admits the renewed installed Home proof on the current union; this remains Release-owned admission. Release’s current package report records reviewed workflow correction 924ec461 and fresh package run 37248458078 against the same admitted product merge. Hosted capacity and package proof are pending. The repair retains model preparation, 70 GiB growth and the 15% reserve; the earlier capacity and retry-admission failures retain their receipts, and Linux success keeps its scoped review. Whole PR232 develop merge remains REFUSE. The operator catalogue correction assigns V3 catalogue signing to the maintainer key, with finalization pinning that signer and head CID. Update read owner reports only. Release reviews packages; Operator owns finalization, signing, installed verification and Anders’s approvals. G1 Install and update: Signed update #89 remains open; UP-02: Stage one complete release #108 installer recovery and the operator update journeys retain their gates.
8. andersalm commented on Oct 3, 2026

   

   andersalm

   on Oct 3, 2026

   ContributorAuthor

   More actions

   PR227’s review leaves this nonblocking follow-up.

   • The retained launch receipt keeps the owner’s full environment in a private 0600 file. Recovery documentation explains that custody; a later allow-list should preserve only required launch inputs.
9. andersalm commented on Oct 4, 2026

   

   andersalm

   on Oct 4, 2026

   Last edited by andersalm

   ContributorAuthor

   More actions

   • Signal authority retains the record if the child exits between settlement observation and a forced signal. Historical 385dcd7 cleanup passed 25 repeats; keep this race in installed shutdown coverage. Recovery diagnostics can retain ESRCH context when an AwaitingExit deadline reports remaining children.
   • Current local and posted Opus PASS for f3244aa retain the startup-cost note: ordinary Serve/Gateway admission repeats controller hashing, and normal installed Home performs three full support admissions. A regular installed external model is ordinarily read twice per admission; overlap and launch paths can change that count. Missing optional files remain admitted; present corrupt checksummed models refuse admission. Measure exact union startup before changing authority checks; the review's size/duration example is unmeasured.
   • A retained receipt with a missing installed binary currently produces the filesystem error. Startup should give the operator one clear recovery step while retaining the same refusal.
   • The early-start helper test uses the existing 200 ms drain deadline with a reaper thread. All nine current gateway-child tests pass; keep exact owned cleanup evidence if a loaded runner exposes that timing limit.
   • A subordinate managed Serve can refuse while the installation restart journal remains pending. Its isolated child Home usually hits the journal Home mismatch before the PID/generation check; the journal retires after readiness or stays retained for recovery. Exact PR230 installed acceptance must cover the first app request and retry through this window. Initial Home health/cleanup proof establishes a narrower scope.
   • ec1 posted review notes LaunchPlan filters USER/USERNAME and TERM. Explicit, stored and attached identity take precedence; host fallback identity and guest terminal configuration lose those values. Any targeted allowlist correction retains secret exclusion. This is source evidence; installed identity/terminal acceptance stays with the exact union.
   • Complete signed-support admission owns ordinary matched installed Home startup as well as update: drifted components, providers or checksummed models produce a repair refusal. The current37 review repeats the legacy first-hop acceptance gap. Existing legacy empty-pin and changed/missing support tests prove source behavior; intact-support Home, clear repair and the installed first hop remain exact union acceptance.
   • Installer cleanup selects the installed gateway binary; the controller uses update-controller/runtime and later reports Home stopped. Retain this shutdown/recovery UX note. Source inspection establishes the path, while exact installed cleanup proves termination timing.- The same independent high source round classified the other current37 review notes: the actual receipt roundtrip reads through the exact0600 reader; creation/replacement and unsafe0640/0660/04600 refusal tests also pass in the current gate. The Mac workflow restores rust-cache before checking both path and symlink absence, creates the private700 directory, then exports its build directory; N/N+1 and the receipt use that domain. These existing checks cover the two notes. All other Cargo caches remain. This is source evidence; hosted Mac execution and exact union installed acceptance remain separate.
   • Current Mac metadata local Opus source PASS records two fixture notes once: a corrupt deflate stream can leave the archive reader as zlib.error after checksum admission, and the output census can omit an extra symlinked directory because it filters directories. The qualified original archive has a complete safe entry inventory and retained checksum; expected output paths and cache files are checked. Keep error presentation and the extra-directory case in the same exact-native setup acceptance, with signed authority intact.
10. andersalm commented on Oct 4, 2026

    

    andersalm

    on Oct 4, 2026

    ContributorAuthor

    More actions

    Release gateway review of local 385dcd71 / tree 5ab40b72 finds two repairs before publication; PR227's API, named branch and pull ref still agree on eb05b39c.

    • P2: runtime_control.rs:1422 reaps an early-exited leader with try_wait; runtime_control/gateway_children.rs:173 then gets ECHILD and leaves its same-group helper running. Retain WNOWAIT ownership through failed-start cleanup and test a helper that outlives its leader.
    • P1: install_transaction.rs:1635 finds recovery only beside current_exe (host_lock.rs:99); an alternate binary can start or run ordinary offline migration against the same data root while its installed binary has a pending journal. Resolve retained/trusted installation ownership first and test separate binary parents with unchanged migration targets.
      I read the actual local result a5c14ff6…, its build receipt, both exact native binary hashes and the install/Home/Carrier/repeat/refusal/cleanup evidence; that passing disposable journey does not cover these two sequences.
      Update owns these repairs and new refused cases. Recheck the corrected exact local source before publication, then obtain current posted review/CI and finished owner approval for the PR230 union.
11. andersalm commented on Oct 4, 2026

    

    andersalm

    on Oct 4, 2026

    Last edited by andersalm

    ContributorAuthor

    More actions

    Release scoped source review PASS for local f3244aa4f38ce963dc83d566c01f79fa6dd92261, tree 05f9df0ad3eb2bbf517b36a803bf793701e1fad4, under PR227. The checkout is clean, 0 behind / 42 ahead; current develop e8d9564f is integrated. Public named/API/pull refs agree at eb05b39c.
    The prior failed-start finding is resolved: readiness observes the exited leader with WNOWAIT; cleanup drains the owned group before reaping. The new test covers a surviving helper, lost-anchor refusal, retained record and preserved unrelated group.
    The prior same-Home recovery finding is resolved: ordinary startup and both offline migration paths resolve retained/trusted installed ownership before the caller's journal. Refused cases preserve target state for alternate parents, redirected sources and unsafe/foreign authority; signed controller repair and transaction-owned SupportActivation migration remain admitted.
    Two independent scoped Sol reviews and Release's source trace PASS. Release verified current gate 9e93bff2…, 514 Rust / 170 script passes and cleanup; authenticated local Opus 8bbd77c2… PASS with all six source blobs and review-output hash matched.
    Fresh exact-source disposable native install, authenticated initial Home, plain Carrier check/apply/repeat and all 12 refusals now PASS: result 1d8d161b…, build 5e9a82dc…, matching binaries 6e694ab1… / 990fb833…, zero M2 HTTP and empty owned cleanup. The first missing-runner-pin refusal is retained; current publication/posted review/CI and finished owner approval still precede the PR230 union and its full operator journey.
12. andersalm commented on Oct 6, 2026

    

    andersalm

    on Oct 6, 2026

    ContributorAuthor

    More actions

    Status (6 Oct): ticked 1 box from PR #227.
    Open: one-return proof for loader failure/crash/hung start; Undo data and legacy first hop proof.
13. andersalm commented on Oct 7, 2026

    

    andersalm

    on Oct 7, 2026

    ContributorAuthor

    More actions

    Real Undo alpha.8 → alpha.7 (Anders, Mac, 7 Oct): refused cleanly and restored the alpha.8 files — the trusted source served alpha.8's media-provider for alpha.7's CID. Cause and fix: #287. Undo acceptance waits for a release pair that both contain the #287 fix.
14. andersalm commented on Oct 7, 2026

    

    andersalm

    on Oct 7, 2026

    ContributorAuthor

    More actions

    After the refused Undo, Home opens normally and System → About shows 0.8.0-alpha.8 (Anders, 7 Oct): a failed Undo restores the current release.
15. andersalm commented on Oct 8, 2026

    

    andersalm

    on Oct 8, 2026

    ContributorAuthor

    More actions

    Decision (7 Oct): load failure, crash before main and a hung start are proven by the real-controller tests in #269 (exactly one return to the previous release); they are not induced on a real install. Real runs on alpha.9 cover restart, interruption and Undo.
16. andersalm commented on Oct 8, 2026

    

    andersalm

    on Oct 8, 2026

    ContributorAuthor

    More actions

    Done: every box ticked with evidence (8 Oct, alpha.9/alpha.10 real runs and merged PRs).
17. andersalm

    closed this as completedon Oct 8, 2026