Repository navigation
perf(code-graph): reread only re-sealed segments when proving a seal - #4523
Merged
Merged
Conversation
Every sealed graph publication proves its generation's durable source before building rows, and that proof read and hashed every file segment, evidence segment, resolution index page, and graph page of the generation. A one-file edit re-seals a handful of segments, so each incremental refresh reread the whole generation: about 18k files and 770 MiB on this repository, 300 s on a machine whose endpoint scanner inspects each first open. The project shard now records, for the generation it last proved, each segment's digest with the file identity (device, inode, size, mtime, ctime) its bytes were hashed at. The next proof skips a segment whose path still names that identity and hashes everything else. A rewrite, truncation, or replacement moves the identity, so the segment is read and refused if corrupt. The record holds one generation per segments root and is replaced only by a proof that succeeds. Refs #4507 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A one-file edit runs an incremental pass that rereads the whole sealed generation (#4507). The release-build profile of a clone of this repo (6.6k files) shows that each graph publication verifies its durable source before it builds rows (
daemon.session_registry.publish_snapshot.verify_source). That step opens, reads, and SHA-256 hashes every file segment, evidence segment, resolution index page, and graph page. On this repo that is about 17.9k files and 772 MiB for each edit. The layered graph build that follows takes 2 s, because it reads only the pages that changed.On this machine the endpoint scanner inspects each file on its first open. That makes the proof cost between 3 s (files already scanned) and 301 s (files not yet scanned) for each edit. I measured the 301 s case in the first edit after a cold build, out of a 444 s pass.
Root cause
verify_partitioned_sealedhas no record of earlier proofs. Segments are content-addressed and written once, so a successor generation shares all but a few segments with its parent. Even so, every publication re-proves all of them from disk.Fix
verify_partitioned_sealedtakes analready_verified(digest, size)predicate. It skips the read and hash of a whole segment when the predicate vouches for it.VerifiedSealedSegmentsV1lives in the session registry, one per project publication shard. The publication locks are minted per runtime, so they cannot hold it. For each segment the last successful proof hashed, it records the digest and the file identity the bytes were read at: device, inode, size, mtime, and ctime, taken from the opened handle and checked against the path. A later proof skips a segment only while its path still has that identity. A rewrite, truncation, or replacement changes the identity, so the segment is read and hashed again and refused if corrupt. The record holds one generation for each segments root. A proof takes the record out when it starts and puts it back only if it succeeds. On non-unix platforms nothing is recorded, so every segment is read as before.The first publication after a daemon restart still proves the whole generation once.
Verification
Before and after, same daemon, isolated profile, retained store, one appended comment line per edit. Base and fix runs alternated, 3 edits per daemon. The time is the
verify_sourcespan busy time (tracedecay_store_runtime=trace). Edit 1 follows a restart, so its record is empty in both builds.Same-daemon edits: the proof takes 3.0 to 301 s on base and 0.10 to 0.12 s with the fix. Every edit published
reextracted_files=1 changed_chunks=2.What this does not change: in warm state the whole incremental pass stays at about 26 s (base 17 to 29 s). Peak RSS stays at about 6.7 GB. Both are now bound by the sparse successor, which decodes about 7.3k parent file segments and re-seals about 3.1k graph pages for a comment-only edit. That is tracked in #4520.
Tests:
//crates/tracedecay-store-runtime:unit_test: the newa_repeated_source_proof_reads_only_segments_whose_file_changedfails with the record disabled ("the second proof read 15 times after the first read 15") and passes with it. The test also checks that a segment rewritten in place is still refused as corrupt. The rest of the suite passes exceptreplay_and_fence_read_awaited_on_a_native_worker_never_park_it, which this PR does not touch and which reads/proc/self/task, so it fails on macOS.//crates/tracedecay-code-index:code_index_suite(production_orchestration, 49 passed).bazel build --config=clippyfor both crates, andrustfmt_check_2024for both crates.Refs #4507
🤖 Generated with Claude Code