Skip to content

ci: installed journeys name the observer failure, and the SmolLM2 proof uses a vendored license - #413

Draft
SashaMIT wants to merge 11 commits into
developfrom
fix/ci-journey-failure-causes
Draft

SashaMIT wants to merge 11 commits into
developfrom
fix/ci-journey-failure-causes

Conversation

@SashaMIT

@SashaMIT SashaMIT commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Head: 9849af79. Three CI changes, plus a merge of the zip 9 branch so CI can run and two merges of develop. Each one makes installed-journey failures easier to diagnose, keeps local data out of uploaded receipts, or removes a failure source outside the repository. There is one commit for each concern.

What this branch carries:

  • 2e70c052: the receipt names the observer failure (1 below).
  • 1ba344a6: the SmolLM2 proof uses a vendored license (2 below).
  • 493f40ed: the observer failure text is redacted before it is saved (3 below).
  • 269bbe19: the redaction also covers URL user info, Windows network and drive paths, and paths with spaces (3 below).
  • d241d0f6: the redaction also covers secret values under quoted and JSON-escaped keys, such as {"token":"x"} (3 below).
  • 11b579ee: the finite observer's message and phase fields from develop also go through redact() (see "Merge with develop").
  • 04b621d0: port (cherry-pick -x, author SashaMIT) of 619c0961 from Reclaim completed Browser bridge workers and preserve finite shutdown #411, "Use a platform-neutral app in Home link regression". See "Ported test fix" below.
  • 69e2bae3: merge of fix/zip-9-rustsec (756dad1c, zip 9.x for RUSTSEC-2026-0367/0368/0369, PR deps: move zip to 9.x for RUSTSEC-2026-0367/0368/0369 #417). It must land first. Remove it from this diff when reviewing.
  • e1de6e52: merge of origin/develop at 8ea4aac9. It conflicted with develop's finite observer work (cd620ab5, Installed journeys in source-home-linux (ubuntu-24.04) fail intermittently, on develop too #433). See "Merge with develop" below.
  • 9849af79: merge of origin/develop at bfefe92f. develop fixed the Chat and app web links require Home confirmation of the exact destination (0.8.0 F1 gate) #430 Home link test itself (it uses Library); that conflicted with the port in 04b621d0 (System), which was this branch's only change to gateway_tests/home_system.rs, so the file is taken from develop and the port is superseded.

Ported test fix (04b621d0)

After the develop merge, test-elastos failed on 11b579ee (run 39084029014, job 117548156624) in develop's own test home_system::home_external_link_names_the_launched_app_and_the_exact_address (#430): assertion left == right failed: home launch token is not authorized for this provider, left: 403, right: 200 at home_system.rs:11964. The test launches Browser, and Browser is not a launchable provider on a host without it (the CI runner, and this Linux machine). develop's own test-elastos never ran at 8ea4aac9 (its run was cancelled). #411's 619c0961 already fixes the test by using System, which every host can launch; assertions and production checks are unchanged. Reproduced locally before the port (same 403 vs 200 at :11964) and passing after it (2 passed; 0 failed). It no-ops once develop carries the same fix.

Superseded by 9849af79: develop's own fix (bfefe92f, Library) is now in the branch. At 9849af79: home_external_link 2 passed, ci-release-policy-test.py 119 OK, git diff --check clean.

Merge with develop (e1de6e52, 11b579ee)

develop's cd620ab5 (#433) also records a finite observer failure, with a stricter rule. The merge keeps both sides:

  • From develop: FINITE_OBSERVER_MESSAGES and finite_observer_failure(). The finite observer saves only a fixed harness message; any other text becomes observer detail redacted, and its str() is never called. The phase names only a validated stage action (stage_engine_running, ...), else stage_request. The new group-exit waits in action() are unchanged.
  • From this branch: runtime_exited in the finite failure, runtime_exited_before_cleanup in finish(), the repair control_failure (RepairRuntime.observe()), and redact().
  • Redaction covers develop's fields too (11b579ee): finite_observer_failure() passes its message and phase through the same redact() with the journey Home and data folder, so a message or phase added to develop's list later cannot carry a path or a secret value out. Every current fixed message is saved unchanged (tested).
  • Shared: both observers take the error type name from one list, PUBLIC_ERROR_TYPES (develop's list, moved to module level). A type not on the list is saved as redacted.
  • One test changed in the merge: test_observer_failure_redacts_home_paths_and_tokens_before_the_record_is_saved now expects observer detail redacted for the finite receipt and CI log line (develop's rule), and still checks the redacted forms in the repair record.

1. The receipt names the observer failure (2e70c052)

FiniteRuntime.observe() and RepairRuntime.observe() in scripts/ci-installed-journeys.py caught every exception and kept only a fixed code (finite_resource_observation_failed / installed_model_repair_control_failed). So the job log ended with RuntimeError: finite installed UI journey failed (run 38899842659, job 116995610376) or installed Runtime exited before health, and the cause was gone.

Now:

  • The finite observer records phase (census, stage_request, stage_<action>, deadline), the exception type, a fixed message (develop's list), and runtime_exited, read from the Runtime pidfd. A census failure that says "Runtime left its unit" with runtime_exited: true means Runtime ended. Any other census message names the check that failed while Runtime was still alive.
  • FiniteRuntime.finish() writes this as finite_memory.observer_failure (and runtime_exited_before_cleanup) into installed-journeys.json. CI already uploads that file. The repair thread writes model_repair.control_failure from observer_failure(): phase, type and a bounded, redacted one-line message.
  • The same text is added to the two CI log lines: wait_healthy and the finite UI journey RuntimeError.
  • The Runtime log stays private (journey-runtime.private.log, unchanged). Only messages the harness writes itself go in, and no observer reads that log. A test confirms that a line from the private log does not appear in the receipt.

2. The license comes from the repository (1ba344a6)

scripts/pinned-model-consumer-proof.sh --fetch downloaded https://www.apache.org/licenses/LICENSE-2.0.txt on every run. In ARM run 38899867022 (job 116983440588), apache.org returned 503 four times and the job failed:

[pinned-model-proof] downloading LICENSE
curl: (22) The requested URL returned error: 503   (x4)
##[error]Process completed with exit code 22.

Now:

  • scripts/fixtures/licenses/Apache-2.0.txt contains the pinned text: 11358 bytes, sha256 cfc7749b…3d30. It is byte-identical to the apache.org download and to Debian's /usr/share/common-licenses/Apache-2.0.
  • In scripts/pinned-smollm2-fixture.json, the license has path in place of url.
  • The proof script (copy_vendored) and ci-model-package.mjs produce (copyVendored) check the vendored bytes against the pin, copy them, and check the copy again. The license check runs before the model download. There is no network fallback.
  • loadFixture accepts exactly one license source: the pinned Hugging Face revision LICENSE (Qwen, unchanged) or the one vendored Apache path.

3. The failure text is redacted before it is saved (493f40ed, 269bbe19, d241d0f6)

An OSError names the file it failed on, so str(error) could put the runner home, the journey Home or its data folder into the uploaded receipt and the CI log line. One redact() (scripts/ci-installed-journeys.py, above observer_failure) runs on the repair control message (observer_failure()) and on the finite observer's message and phase (finite_observer_failure()), each with its own Home and data folder.

  • URL user info: the user and password in any scheme://user:pass@host (or scheme://user@host) become <redacted>; the scheme and host stay (http://<redacted>@127.0.0.1:8090/). This runs first (URL_USERINFO).
  • Paths: the journey Home, its data folder (on macOS it holds a space: Library/Application Support) and $HOME are replaced first, longest first. Then file: URLs, ~/ paths, Windows network and device paths (\\server\share, //server/share, \\?\C:\, \\.\pipe\), drive paths with either slash (C:\Program Files, C:/Users) and any other absolute path (PATH_START). Each becomes <path>. Only a last part that looks like a harness file stays (<path>/request.json, <path>/memory.events), from a fixed list of suffixes. Any other last part can be a user, device or key name, so it is dropped. http://127.0.0.1:8090/... URLs stay.
  • Where a path ends (PATH_STOP, PATH_TAIL): at a mark that ends a path in an error message, which is a quote, a backtick, a bracket, ,, ;, |, a line end, or a colon followed by a space or the end (open /a b/c: denied). Spaces inside stay in the path, so /Volumes/Backup Disk/ci-person (exit 2) becomes <path> (exit 2). A path with no such mark after it takes the rest of the text with it: losing words is better than leaking a folder name.
  • Secret-named keys (SECRET_VALUE, new in d241d0f6; it replaces the bare-key rule that was in REDACTIONS): one rule for token, password, api_key, access_token, client_secret, session, ... whether the key is bare, single- or double-quoted, or JSON-escaped (\"token\":\"x\"), with = or : and any spaces. A quoted value is redacted up to its own closing quote with no backslash before it, so an escaped quote inside it ("a\"b c") does not end it; with no closing quote, the rest of the line goes. A bare value ends at a space, a quote, ,, ;, } or ]. The key and the keys after it stay readable: {"access_token":"<redacted>","phase":"stage_engine_running"}.
  • Auth schemes: the value after Bearer / Basic / Token becomes <redacted>. Since d241d0f6 this rule does not take = or : as the value, which left x9 in token = x9.
  • Runs of 20 or more hex/base64/base58 characters that hold a letter and a digit become <redacted>. That includes digests and unit ids; a failure message does not need them.
  • DIDs: did:key:<redacted>. A DID names a key, so it can name a person or a device.

Tests

  • scripts/ci-release-policy-test.py:
    • FiniteInstalledJourneyTests.test_finite_observer_failure_names_its_phase_type_message_and_runtime_exit: three cases (Runtime left its unit, census check failed while alive, stage deadline).
    • test_finite_observer_failure_reaches_the_uploaded_installed_journey_receipt: real run(), wait_healthy, poll() and finish(), with the message checked in installed-journeys.json.
    • From develop (cd620ab5), kept: test_finite_observer_saves_phase_type_and_only_public_messages, test_finite_observer_stage_phase_uses_only_validated_action_text and the engine group-exit tests.
    • test_finite_observer_fixed_messages_and_phase_still_pass_redaction (negative, new in 11b579ee): every message in FINITE_OBSERVER_MESSAGES is saved unchanged; a message and phase added to the list with {"token":"..."} and a $HOME path come out as {"token":"<redacted>"} and <path>. With the two redact(...) calls in the finite_observer_failure() return reverted to plain phase / message[:480], it fails: AssertionError: 'finite journey control said {"token":"tq-41"} at /home/ci-person/x' != 'finite journey control said {"token":"<redacted>"} at <path>'.
    • test_repair_control_failure_reaches_its_record and test_observer_failure_text_is_bounded_and_single_line.
    • test_observer_failure_redacts_home_paths_and_tokens_before_the_record_is_saved (negative): a failure message with $HOME, a data path with a space, other absolute paths, a file: URL, a Bearer token, token=/password: values, a sha256, a did:key, a base64 run and (since d241d0f6) {"token": "q-secret-9"} goes through the real finite observer, run() and finish(), and through the repair control thread. The saved installed-journeys.json, the repair record and the CI log line hold none of them.
    • test_observer_failure_redacts_url_user_info_windows_network_paths_and_paths_with_spaces (negative, 269bbe19): 13 causes through observer_failure(), none under a known root. With scripts/ci-installed-journeys.py put back to 493f40ed, all 13 fail.
    • test_observer_failure_redacts_secret_values_under_quoted_and_escaped_keys (negative, new in d241d0f6): 12 causes through observer_failure(), each checked against its exact saved text: {"token":"x"}, {"api_key": "abc9"}, {'password': 'p4ss'}, "secret" : "s-9 9", \"token\":\"xq7\", {"access_token":"at1","phase":"stage_engine_running"}, an escaped quote inside a value, a doubly escaped value, "client_secret" = 'c c', a value with no closing quote, token = x9 and {token:k5}. With the two REDACTIONS lines in scripts/ci-installed-journeys.py reverted (the SECRET_VALUE entry put back to the old bare-key rule re.compile(rf"\b({SECRET_NAME})(\s*[=:]\s*)(?!(?:Bearer|Basic|Token)\s)(['\"]?)[^\s'\",;]+\3", re.I), and (?[=:]) taken out of the auth-scheme rule), all 12 cases fail, for example AssertionError: '{"token":"x"}' != '{"token":"<redacted>"}' and AssertionError: 'token <redacted> x9 then' != 'token = <redacted> then', and the home-paths test fails with 'q-secret-9' unexpectedly found in ....
    • PinnedModelProofLicenseTests: runs the real proof script with a curl that records any call. The good copy is verified and copied with no curl call. A tampered copy (same size, appended, empty) is refused before copy or download.
  • scripts/ci-model-package.test.mjs: copyVendored refuses a tampered vendored file and writes nothing. loadFixture refuses an apache.org URL, an escaping path, or both sources at once.
  • Full runs at 11b579ee, locally on Linux: ci-release-policy-test.py 119 OK. git diff --check, check-product-data (+ its 99 tests) and public-copy-entropy-check are clean. At 04b621d0: cargo test -p elastos-server --lib -- home_external_link 2 passed (it failed with 403 vs 200 before the port), cargo fmt --check clean. No Rust is changed by this PR's own commits other than the ported test. CI on 9849af79 is pending.

Gaps


Generated by Claude Code

cargo-audit fails on every PR: zip 2.4.2 in elastos/Cargo.lock and in
capsules/object-provider/Cargo.lock (through its elastos-server path
dependency) has three advisories against ZipArchive::extract and the
stream extract API. The runtime calls neither; it reads each entry
itself. zip 9.0.4 is the patched release.

- elastos-server uses zip 9.0.4, still without default features. 9.x has
  no separate flate2 feature; deflate-flate2 now enables the dependency.
- Both lockfiles change only for zip: nix 0.31.3 and typed-path come in,
  derive_arbitrary goes. `cargo update -p zip` also moved unrelated range
  dependencies (windows-sys, getrandom); those lines stay as they were,
  and `cargo metadata --locked` accepts both files.
- ZipFile::name now returns a Result. The ZIP readers report an error as
  "invalid ZIP archive entry".
- Whole-archive extract keeps its rule for entry names. zip 9
  enclosed_name turns "/a.txt" into "a.txt" and folds "x/../a.txt" into
  "a.txt", so an entry that extract refuses today would be written. The
  runtime now checks the name with the rule zip 2.x applied: no NUL byte,
  no root or drive prefix, no climb above the archive root.

test_library_provider_extract_zip_refuses_absolute_dotdot_and_symlink_entries
covers absolute, "..", inner "..", backslash and symlink entries. It
passes on zip 2.4.2 and with this change, and fails on the Absolute.zip
case when extract uses zip 9 enclosed_name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6Qn9RX7LDqrwiAfNC2F2K
… the run

The finite-memory observer and the model-repair control thread caught every
exception and kept only a fixed code, so a failed job said "installed Runtime
exited before health" or "finite installed UI journey failed" and nothing
else. The uploaded installed-journeys.json now carries the phase (census,
stage request, engine stage, deadline), the exception type, a bounded
one-line message, and whether the Runtime process had already exited. The
same text follows the CI log line. Only harness-written messages go in; the
Runtime log stays private.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6Qn9RX7LDqrwiAfNC2F2K
…downloading it

Every installed-journey run downloaded LICENSE-2.0.txt from apache.org, and a
503 there failed ARM CI (run 38899867022). The pinned text now lives at
scripts/fixtures/licenses/Apache-2.0.txt with the same size and SHA-256 as the
fixture pin. pinned-model-consumer-proof.sh and ci-model-package.mjs produce
check the vendored bytes against the pin, copy them, and check the copy. A
changed vendored file stops the run before the model download, and there is
no network fallback. The Qwen fixture keeps its pinned Hugging Face revision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6Qn9RX7LDqrwiAfNC2F2K
zip 9 for RUSTSEC-2026-0367/0368/0369 so cargo-audit passes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6Qn9RX7LDqrwiAfNC2F2K
…fore the receipt is saved

observer_failure() copied str(error) into installed-journeys.json and the
CI log line. An OSError names the file it failed on, so the runner home,
the journey Home and its data folder could reach an uploaded artifact.

One redact() now runs on the message before it is bounded. It replaces the
journey Home, its data folder (which can hold a space on macOS) and $HOME
first, then file: URLs, ~ paths, drive paths and any other absolute path,
with <path>. Only a last part that looks like a harness file
(request.json, memory.events) stays. It also replaces the value after
Bearer/Basic/Token and after secret-named keys (token=, password:), the
key part of a DID, and long hex/base64/base58 runs. Phase, type and
runtime_exited are unchanged. A new policy test puts a home path, a data
path with a space, other absolute paths and tokens in both the finite
observer and the repair control failure, and checks the saved receipt,
the repair record and the CI log line hold none of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6Qn9RX7LDqrwiAfNC2F2K

Copy link
Copy Markdown
Contributor Author

493f40ed: observer_failure() now runs one redact() before the message is saved. It removes the Home, the data folder, $HOME, other absolute paths, Bearer/token=/password: values, DID keys and long hex/base64 runs. phase/type/runtime_exited are unchanged.
Proof: test_observer_failure_redacts_home_paths_and_tokens_before_the_record_is_saved checks the saved receipt, the repair record and the CI log line. It fails when the redact() line is reverted ('/home/ci-person' unexpectedly found). ci-release-policy-test.py: 109 OK.


Generated by Claude Code

@SashaMIT

Copy link
Copy Markdown
Contributor Author

Round 3 review: fixes needed (C+). Policy tests pass.
redact() lets through URL user info, Windows network paths and paths with spaces.

…paths and paths with spaces

redact() kept the user and password in scheme://user:pass@host, did not
know \\server\share, //server/share, \\?\C:\ or \\.\pipe\ paths, and cut
an unknown path at its first space, so "/Volumes/Backup Disk/ci-person"
left "Disk/ci-person" in the receipt.

The user part of any URL is now replaced (http://<redacted>@127.0.0.1:8090/).
Network, device and drive paths (either slash) start a path. A path now
runs until a mark that ends a path in an error message: a quote, a
bracket, a comma, a semicolon, a pipe, a line end, or a colon followed by
a space. Spaces inside stay in the path. A path with no such mark after it
takes the rest of the text: losing words is better than leaking a folder
name. A new policy test checks 13 such causes (all 13 fail on the old
rules) and that a plain URL and a cause with no path are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6Qn9RX7LDqrwiAfNC2F2K

Copy link
Copy Markdown
Contributor Author

269bbe19: redact() now replaces URL user info (http://<redacted>@host), Windows network/device/drive paths (\\server\share, //server/share, \\?\, \\.\pipe\, C:\Program Files, C:/...), and keeps spaces inside a path until a quote, bracket, ,;| or : .
Proof: test_observer_failure_redacts_url_user_info_windows_network_paths_and_paths_with_spaces, 13 cases, all 13 fail with the script at 493f40ed; full ci-release-policy-test.py 110 OK.


Generated by Claude Code

@andersalm andersalm mentioned this pull request Oct 17, 2026
9 tasks
@SashaMIT

Copy link
Copy Markdown
Contributor Author

Round 4 at 269bbe1: fixes needed (C+). URL user info, UNC and space paths now redacted.
Still leaks secrets under quoted JSON keys, e.g. {"token":"x"}. Conflicts with develop's observer work: merge develop.

develop's finite observer (#433) keeps its fixed-message list, its
validated stage names and its public error type names. This branch keeps
runtime_exited, runtime_exited_before_cleanup, the repair control failure
and redact() for that repair text. Both observers now name the error type
from one shared list (PUBLIC_ERROR_TYPES).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6Qn9RX7LDqrwiAfNC2F2K
…escaped keys

redact() kept the value in {"token":"x"}, {'password': 'p'} and
\"token\":\"x\", because the secret-key rule only matched a bare key.
One rule (SECRET_VALUE) now matches a bare, quoted or JSON-escaped
secret-named key before = or :, and redacts a quoted value up to its
own closing quote (or the line end when there is none). The auth-scheme
rule no longer takes "=" as the value in "token = x", which left x.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6Qn9RX7LDqrwiAfNC2F2K

SashaMIT commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor Author

11b579ee: merged develop (e1de6e52): develop's fixed-message finite observer is kept, this PR's runtime_exited and repair cause are kept, and develop's message/phase fields now also pass redact(). One SECRET_VALUE rule redacts values under bare, quoted and JSON-escaped secret keys ({"token":"x"}, \"token\":\"x\", 'password': 'p').
Proof: test_observer_failure_redacts_secret_values_under_quoted_and_escaped_keys (all 12 cases fail with the two REDACTIONS lines reverted) and test_finite_observer_fixed_messages_and_phase_still_pass_redaction (fails with the redact() calls removed); ci-release-policy-test.py 119 OK.


Generated by Claude Code

claude and others added 2 commits October 18, 2026 00:46
develop's finite observer saves only fixed messages, but its message and
phase did not pass redact(). A message or phase added to that list later
could carry a path or a secret value into the receipt. Both now pass
redact() with the journey Home and data folder; every current fixed
message is saved unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6Qn9RX7LDqrwiAfNC2F2K
Use the installed System fixture for valid and invalid Home-link requests so the test exercises link authority on every host. Keep all assertions and production launch checks unchanged.

Verified the exact Home-link test passes, reverting only the System token to Browser fails the intended 403-versus-200 assertion, and restoring the fixture passes again.

Co-authored-by: Codex
(cherry picked from commit 619c096)

Copy link
Copy Markdown
Contributor Author

test-elastos at 11b579e failed in develop's own Home link test, home_external_link_names_the_launched_app_and_the_exact_address: 403 vs 200, because Browser cannot be launched on this host. It is not caused by this PR.
Ported #411's fix 619c0961 as 04b621d0 (System instead of Browser, author kept). The test fails locally before the port and passes after it (2 passed).


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

source-home-linux (ubuntu-24.04) at 04b621d (job 117562372696) failed model-repair restarted_home with a timeout. This is the #433 failure, not this PR's change. The runtime restarted cleanly (2 runtime runs, cleanup passed) and no observer error was recorded. home-failure.png shows Home ready with Marketplace running in the dock but no Marketplace window. The 4 GiB finite journey passed on this run. No fix exists yet; I can't re-run (403).


Generated by Claude Code

develop now carries its own fix for the #430 Home link test (bfefe92:
the test uses Library, which every platform launches). This branch had
ported #411's 619c096 for the same test (System instead of Browser),
and the two edits conflicted in gateway_tests/home_system.rs. That port
was this branch's only change to the file, so the file is taken from
develop; 04b621d's change is superseded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6Qn9RX7LDqrwiAfNC2F2K

Copy link
Copy Markdown
Contributor Author

Merged develop (9849af79). develop fixed the Home link test itself in bfefe92f (Library), and that conflicted with the port in 04b621d0 (System). The port was this branch's only change to that file, so the file now comes from develop.
home_external_link 2 passed and ci-release-policy-test.py 119 OK at 9849af79.


Generated by Claude Code

@SashaMIT

Copy link
Copy Markdown
Contributor Author

Re-review at 9849af7: PASS (A). Develop merge replaces the 619c096 port; redaction tamper fails all 12 inputs; all CI jobs green, including both Linux source-home jobs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants