Repository navigation
alpha.13 train: Qwen2.5 1.5B, download resume and retry, memory admission and idle release (#84, #98) - #322
Merged
Merged
Conversation
alpha.12 on a real Jetson stopped once when the only holder (the seed) missed the 5 s answer deadline on 8 queued part reads together. A part read that fails with a provider error (holder deadline, failed Carrier invocation, lost connection) is now repeated up to 3 times after 0.5 s, 1 s and 2 s, keeping its in-flight slot; each repeat rechecks cancellation, expiry and authority first. Integrity, authority and capacity failures are never repeated. Cause of the burst: the client sends parallel streams on one Carrier connection with no lock of its own (server spawns a task per stream), but the seed's ipfs-provider bridge holds one I/O lock per request until the answer (bridge.rs send_json_line), so one slow read there or one stalled connection runs every queued read's budget out together. Tests: two parts stalling once past the deadline admit identical bytes (each read twice); a part that always fails stops after 4 reads with WeightsRead; cancel during backoff ends Cancelled with no further read.
…e-updated models produce takes a pinned fixture and writes OUT/entry.json; PROVENANCE.md takes license, quantization and repositories from the fixture; the new catalog step writes one unsigned 1-8 entry payload for release-signer.py --model-catalog. SmolLM2 still rebuilds to bafybeiew3vuq32fvuz2kmps7lmgl4rxpvwsvklxykx4covogbrkhgy5qky. The model projection reports dispatch_unavailable_reason catalog_updated for an admission bound to an earlier catalogue head; Marketplace and System say "Model list updated. Press Use to keep using this model." with a Use button. Marketplace shows a signed model's own description.
Review of 3bd50d2: every ProviderError::Provider was repeated, which includes holder refusals and malformed answers, and a repeat skipped the capacity check and other parts' refusals. Carrier now marks a bounded read whose every endpoint failed in transit (answer deadline, connect failure, a stream that could not be opened, written or read: CarrierTransportFailure) as a typed I/O error; the availability provider reports code carrier_fetch_in_transit only when every holder failed that way, and Content maps that code to ProviderError::Io. Preparation repeats only Io. Each repeat runs the same per-part checks as a first read (authority, expiry, capacity, and the shared refusal, which now keeps its failure phase). Tests: a refused or oversized answer is read once and stops (WeightsRead); a repeat is not sent after another part's capacity check was refused (Capacity); the stall tests now use real 5 s deadlines.
A model download that stopped on a read failure or a Home restart kept nothing, so the next Use fetched the whole package again. settle_failure now keeps the stage (inventory kept_stage, owned by the Failed/Expired record) after a read failure, a stop (cancel aside: deadline, Home stopping) or restart reconciliation, and removes it on cancel and on any integrity, header, authority, policy or storage failure. The next Use takes the stage over only when its _elastos_object.json equals the signed object_manifest: complete files are re-hashed, a partial file is cut to a 64 KiB boundary and its prefix hashed into the running digest, and only the remaining parts are read. completed_bytes comes from disk, since the record is bumped before fsync. Per-file SHA-256 and the package CID check are unchanged; any mismatch removes the stage. Reopened files pass the same owner-only, single-link, no-symlink checks. Free-space checks for the next preparation count the kept bytes on disk. Tests: a read failure at part 20 resumes reading only the remaining parts with identical admitted bytes; an aborted worker plus reconcile resumes at the last whole part on disk (record ahead of disk, torn tail); a corrupted kept byte fails WeightsIntegrity, removes the stage, and the next Use reads everything; a stage holding another package is not adopted.
…ow catalog_updated Fixture loading refuses non-plain weight names, non owner/name repositories, short revisions, other weight or license URLs and multi-line descriptions before any download; produce asserts the weights path stays in OUT/inputs. catalog now takes the publisher DID and writes payload.json only after release-signer.py's own check_model_catalog accepts it. current_entry checks grant, trusted catalogue and current offer before the head, so only a still-offered, still-granted package reports catalog_updated.
…gine, ARM64 setup skip
Keep rule now matches the in-transit typing: the stage stays after a read that failed in transit (ProviderError::Io), a stop (deadline, Home stopping) or restart reconciliation, and is removed on cancel, holder refusals, malformed answers, integrity, authority, policy and storage failures. A part refused after another part's failed check stays recognisable as stopped when the preparation was stopped. Resume tests now stop the first Use with holder stalls past the 5 s answer budget (fixture stall_from); the refusal test also checks the stage is removed.
Re-review of 8e0c13b: a refreshed ticket's timeout replaced an earlier refusal for the same holder (last wins), and an answer that was not valid UTF-8 counted as a transport failure. A holder's in-transit flag is now ANDed over its announced and refreshed attempts, and a failed answer read is in transit only for reset, aborted, not connected, broken pipe, unexpected EOF or timed out; InvalidData and other read errors stay permanent. Tests: refusal then refreshed-ticket timeout reports carrier_fetch_failed (both timeouts report carrier_fetch_in_transit); an invalid UTF-8 answer returns a provider error, not I/O.
…nux honours cgroup v2 limits; host-independent fixtures
…M covers the need
Review of c798c7c: a kept stage released its cache charge but kept its bytes with no way to clean it up. It now stays charged against the cache budget at its size on disk (kept_stage.bytes) until the next non-shared preparation takes it over at start (its own reservation replaces the stage's charge, so nothing is counted twice). Cancel on the failed or expired attempt that owns it removes the stage and the charge. Model-provider startup, under the worker lock, removes a kept stage older than 7 days or whose model the current trusted catalogue no longer offers. Tests: the kept bytes count against the budget and a new Use's charge covers them; Cancel on a failed attempt removes the stage and frees the budget; startup keeps a recent stage and removes a 7-day-old one and one whose model is no longer offered.
Review of 223be87: after capacity retirement, prepare_capacity moved the pending Use to Preparing with its full reservation but left kept_stage set, so the stage was charged twice and Cancel on the old failed owner could delete the running preparation's stage. It now clears kept_stage in that ownership transfer, as the normal start path does. Test: a capacity-pending Use of the same model started by prepare_capacity owns the stage with only its own charge, and Cancel on the old failed attempt leaves the stage in place.
Admission asks for ~2.9 GiB on Linux/Jetson (weights, KV cache, scratch and 1 GiB headroom). Only capsule.json and _elastos_object.json change; package CID bafybeievyi6xp5b62qkuh65ywkd72medhx6uxrmik2fhnr3aameeqnxvwe.
Every Use or Retry gets a new request id, and Failed, Expired and
Cancelled records were never pruned, so after 64 attempts reserve_at
refused every new Use ("preparation inventory is full") and the Home
could never get a model again. Before refusing on the cap, reserve_at
now prunes the oldest Failed, Expired or Cancelled records that own
nothing: no kept stage, no retirement, no other record's admission and
no reclaimed history. Admitted, Reclaimed and live records are never
pruned, so the cap still bounds live work.
Test (replaces the cap-refusal case): after 100 cancel/fail cycles a Use
still reserves, and the record owning admitted bytes or a kept stage
survives with its bytes.
# Conflicts: # elastos/CHANGELOG.md # scripts/model-preparation-failure-smoke.mjs
Review of aca1312: pruning on the record cap could delete a request's replay history or an attempt that had just failed while the UI still polled its status. A record now carries settled_at (set when it becomes Failed, Expired or Cancelled; older records count from their deadline), and only terminal records that settled at least an hour ago and own nothing are pruned, longest-settled first. If none is old enough the cap refuses as before. A pruned request id that is replayed starts a new reservation. Tests: with the cap full of attempts from the last hour a Use is refused and the just-failed attempt still answers status; an hour later the oldest is pruned, the Use reserves, and replaying the pruned request id starts a new reservation; 100 retries over 2.5 h (fake clock) still allow a Use while the admitted-bytes and kept-stage owners stay.
Review of a03728a: reserve_at expired an overdue reservation (setting settled_at) but returned the record-cap refusal before saving, so every later Use expired it again with a fresh settled_at and the cap never cleared. reserve_at now saves expiry before any refusal. Test: 63 never-pruned admitted records and one overdue reservation; a Use is refused but the reservation is saved as Expired at that time, and a Use an hour later (fake clock) reserves.
Review of 244ad68: a replay refused for a changed request identity still returned after expire() and before any save, discarding the expiry and its settled_at. reserve_at now saves expiry right after it runs, before replay validation, so no refusal path skips it. Test: replaying an overdue reservation's request id with a changed interface is refused, and the reservation is saved as Expired at that time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
alpha.13 train: a second local model and robust model downloads (#84, #97, #98).
Cross-family reviews: piece-retry FAIL x2 -> PASS (ef5c7db); resume FAIL x5 -> PASS (3f965b9); resource-bounds FAIL x2 -> PASS (dd684cd); qwen-catalog FAIL -> PASS (b923e2a). Full remote check PASS at 2f192e9 (gate 25 min, test-elastos, test-capsules). Supersedes the resource parts of draft #136 (lease, cgroup CI services dropped).
Real-install runs follow on the published alpha.13 (Linux, Jetson, then Mac).