Skip to content

feat(cache): restore symlinks a task creates - #918

Merged
wan9chi merged 2 commits into
feat/cache-empty-foldersfrom
feat/cache-created-symlinks
Nov 5, 2026
Merged

wan9chi merged 2 commits into
feat/cache-empty-foldersfrom
feat/cache-created-symlinks

Conversation

@wan9chi

@wan9chi wan9chi commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Motivation

A task that creates a symbolic link doesn't have it among its outputs unless an output glob matches it. Nitro's dependency tracer, for example, links .output/server/node_modules/tslib into .nf3. fspy records nothing for symlink, so a hit leaves the link out. When the task checks the path first, as nf3 does, the link is a plain input and a clean checkout misses.

Changes

fspy records the path of a new link as a write, before the call:

  • Unix preload: symlink and symlinkat.
  • Linux seccomp supervisor: the same calls for static binaries.
  • Windows: NtFsControlFile with FSCTL_SET_REPARSE_POINT(_EX), which covers CreateSymbolicLinkW and junctions.

vt needs no change: a written symlink counts as a file, and the archive stores links as links (#885).

E2E cases: symlink_in_output (synthetic, vtt symlink) and nitro_auto_outputs_restore_links (the nitro_build_cache Nitro build with automatic tracking only). Both run only on Unix in the first commit, because Windows already behaves differently there, and on every platform after.

Part of a stack that lets automatic tracking cache tasks which read their own outputs and list the folders holding them, so they also hit on a clean checkout.

🤖 Generated with Claude Code


Note

Medium Risk
Changes core file-access tracing on all platforms (new syscalls/NT detour); mistakes could mis-classify I/O or miss links, affecting cache correctness for builds that rely on symlinks.

Overview
Automatically tracked outputs now include symlinks a task creates, so cache hits restore them (including Nitro-style dependency links under .output) without matching output globs, and a clean checkout can still hit when the task stat’d the link path first.

fspy records the new link path as a write before the syscall returns: Unix symlink / symlinkat (preload + seccomp), and on Windows NtFsControlFile when FSCTL_SET_REPARSE_POINT(_EX) runs (symbolic links and junctions). Link targets are not tracked. rebuildable documents that Windows junctions count as written symlinks, not empty mkdir folders.

Adds unit/integration tests and e2e fixtures (symlink_in_output, Nitro build-auto). vtt symlink works on Windows for tests; changelog entry for #918.

Reviewed by Cursor Bugbot for commit 93cb131. Bugbot is set up for automated code reviews on this repo. Configure here.

@wan9chi
wan9chi added this pull request to stack #920 October 30, 2026 03:17
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from ce2ff95 to 6f295b5 Compare October 30, 2026 03:19
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

fspy benchmark

linux

dynamic/launch             change  +0.58%  [ -3.65% ..  +6.69%]  overhead   +60.37%
dynamic/access             change  +0.80%  [ -6.08% ..  +6.04%]  overhead   +19.83%
dynamic/access-relative    change  +1.17%  [ -3.90% ..  +6.27%]  overhead   +73.46%
dynamic/access-contended   change  +0.23%  [ -9.09% ..  +7.13%]  overhead   +51.32%
static/launch              change  +1.47%  [ -2.41% ..  +6.27%]  overhead  +200.38%
static/access              change  +0.26%  [ -1.67% ..  +1.98%]  overhead +1455.99%
static/access-relative     change  +0.75%  [ -0.15% ..  +2.64%]  overhead +1940.17%
static/access-contended    change  +0.45%  [ -1.84% ..  +1.55%]  overhead +2121.24%

macos

dynamic/launch             change  +1.10%  [ -3.16% ..  +4.91%]  overhead  +235.23%
dynamic/access             change  +0.71%  [ -4.18% ..  +4.01%]  overhead    +3.28%
dynamic/access-relative    change  -3.31%  [ -5.14% ..  -1.32%]  overhead  +246.94%
dynamic/access-contended   change  -0.18%  [ -5.49% ..  +3.70%]  overhead    +5.02%

windows

dynamic/launch             change  -0.06%  [ -5.14% ..  +5.17%]  overhead   +27.51%
dynamic/access             change  +0.56%  [ -0.56% ..  +1.50%]  overhead    +1.50%
dynamic/access-relative    change  +0.00%  [ -1.09% ..  +1.09%]  overhead    +0.91%
dynamic/access-contended   change  +0.18%  [ -0.53% ..  +1.07%]  overhead    +1.09%

@wan9chi wan9chi added the ci: full Run the slow CI jobs (macOS x64, musl) on this pull request label Oct 30, 2026
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from 6f295b5 to 50eb6eb Compare October 30, 2026 04:07
@wan9chi wan9chi added ci: full Run the slow CI jobs (macOS x64, musl) on this pull request and removed ci: full Run the slow CI jobs (macOS x64, musl) on this pull request labels Oct 30, 2026
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from 50eb6eb to 3265412 Compare October 30, 2026 06:31
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from 3265412 to 25a585d Compare October 30, 2026 07:08
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from 25a585d to f7f77b5 Compare November 1, 2026 11:55
@wan9chi wan9chi added ci: full Run the slow CI jobs (macOS x64, musl) on this pull request and removed ci: full Run the slow CI jobs (macOS x64, musl) on this pull request labels Nov 1, 2026
@wan9chi wan9chi added the read-write overlap Caching tasks that read and write the same paths label Nov 4, 2026
@wan9chi
wan9chi removed this pull request from stack #920 November 5, 2026 08:52
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from f7f77b5 to f7dc7d7 Compare November 5, 2026 08:54
@wan9chi
wan9chi added this pull request to stack #942 November 5, 2026 08:55
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from f7dc7d7 to 087c868 Compare November 5, 2026 09:02
Two cases for a task that creates symbolic links. In `symlink_in_output`,
the task links `out/link` to `out/target.txt` with `vtt symlink`, which now
works on Windows too, making a directory link when the target is a
directory. In `nitro_auto_outputs_restore_links`, the Nitro build of
`nitro_build_cache` runs with automatic tracking only, without `input` or
`output` globs; nf3 links `tslib` and each package's `tslib` into `.nf3`.

fspy records nothing for `symlink` on Unix, so a link a task creates isn't
an output. Removing `out/link` is a cache hit that leaves it missing. nf3
checks each path before linking it, so Nitro's links are plain inputs,
and removing `.output`, as on a clean checkout, is a cache miss. Several
inputs are then missing, and vt names whichever its parallel validation
finds first, so the step redacts its miss reason.

Both cases are Unix only for now. On Windows, `CreateSymbolicLinkW` opens
the new link without write access, so fspy records it as a read, and
removing `out/link` is a cache miss. nf3 makes junctions there, which
libuv opens with write access before setting the reparse point, so
they're recorded as written already.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A task that creates a symbolic link, like Nitro's dependency tracer
linking `.output/server/node_modules/tslib` into `.nf3`, didn't have the
link among its outputs unless an `output` glob matched it: fspy recorded
nothing for `symlink`, so a hit left the link out, and when the task
checked the path first, as nf3 does, the link was a plain input, so
removing it, as on a clean checkout, was a cache miss.

fspy now records the path of a new link as a write, before the call, so a
failed call is recorded too. The target is stored as text, not accessed,
and needs no record.

- Unix preload: `symlink` and `symlinkat`.
- Linux seccomp supervisor, for static binaries: `symlinkat`, plus
  `symlink` on x86_64.
- Windows: `NtFsControlFile` with `FSCTL_SET_REPARSE_POINT(_EX)`, for the
  file the reparse point is set on. `CreateSymbolicLinkW` creates the link
  with an open that lacks write access, recorded as a read, and then sets
  its reparse point. A junction, which libuv makes for Node's `fs.symlink`
  with `'junction'` and nf3 uses on Windows, is a folder with a mount
  point set on it; libuv opens the folder with write access first, so
  those junctions were recorded as written already.

vt needs no change: `Rebuildable` treats a written symlink as a file, so
the next run may find it missing, and the archive stores links as links
(#885), so with automatic output tracking a hit restores them unless
output exclusions or `ignoreOutput` leave them out. std reports a
junction as a symlink, not a folder, so the mkdir recorded for it doesn't
make it an empty folder, and the write makes it rebuildable. Both e2e
cases now run on Windows too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from 087c868 to 93cb131 Compare November 5, 2026 09:17
@wan9chi
wan9chi merged commit 4b914e2 into main Nov 5, 2026
23 checks passed
@wan9chi
wan9chi deleted the feat/cache-created-symlinks branch November 5, 2026 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci: full Run the slow CI jobs (macOS x64, musl) on this pull request read-write overlap Caching tasks that read and write the same paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant