Skip to content

feat(cache): let input negatives keep entries out of directory listings - #941

Merged
wan9chi merged 2 commits into
feat/cache-created-symlinksfrom
feat/cache-input-negatives-in-listings
Nov 5, 2026
Merged

wan9chi merged 2 commits into
feat/cache-created-symlinksfrom
feat/cache-input-negatives-in-listings

Conversation

@wan9chi

@wan9chi wan9chi commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Motivation

An exclusion in a task's input, like "!build/**", leaves out only the paths the task reads. When the task lists a folder, every entry still counts, so an excluded folder appearing there is a cache miss: a lint task listing the package root misses once another task's build output appears in it. That is the case the hard-coded dist skip was added for (#181). #919 removes the skip, so users need a setting to get it back, and today "!dist/**", which docs/inputs.md recommends, doesn't help.

Changes

Listings leave out what the input config excludes, after the entries the task can rebuild (#915):

  • An entry whose path the input negative globs match. A folder entry is matched by its own path; wax's ** matches zero or more components, so build/** leaves out build, while build/* keeps it. The globs are part of the cache key, so validation applies the same ones to the current listing, and an entry that appears after the run is left out too.
  • An entry that a tool-reported ignoreInput path names. Those paths are known only after the run, so the names are recorded in the listing's ignored_entries, including entries that aren't there yet.

The cache schema version is bumped. docs/inputs.md now says negatives also keep entries out of listings.

E2E cases: negative_glob_hides_listing_entry (synthetic, vtt list-dir with "!out/**"; an unrelated new file still misses) and oxlint_negative_build (oxlint -D no-debugger with a gitignored build/ and "!build/**"). Both miss today and hit after. A unit test covers ignoreInput.

Part of a stack that lets automatic tracking cache tasks which read their own outputs and list the folders holding them, so they also hit on a clean checkout.

🤖 Generated with Claude Code


Note

Medium Risk
Changes post-run fingerprint semantics and cache validation for any task that lists directories with input negatives or ignoreInput; incorrect glob matching could cause stale cache hits, mitigated by schema bump and targeted tests.

Overview
Input exclusions now apply to directory listings, not only to paths the task reads. Tasks that list a folder (e.g. a linter scanning the package root) no longer miss the cache when an excluded entry appears—such as another task’s build/ folder after "!build/**"—as long as nothing else in the listing changed.

Post-run fingerprints omit listing entries matched by the task’s input negative globs (same rules as reads: build/** drops the build folder entry; build/* does not) and by tool ignoreInput paths (names recorded in ignored_entries for validation). Cache hits recompile those globs from the entry key and apply the same filters when comparing listings.

Cache schema version is bumped to v22 (fresh cache). Docs and changelog describe the behavior; e2e covers !out/** with list-dir and an oxlint + !build/** scenario, plus a unit test for globs and ignoreInput.

Reviewed by Cursor Bugbot for commit 0655edb. Bugbot is set up for automated code reviews on this repo. Configure here.

@wan9chi wan9chi added the read-write overlap Caching tasks that read and write the same paths label Nov 5, 2026
@wan9chi
wan9chi force-pushed the feat/cache-input-negatives-in-listings branch from c84f0b3 to 157bf82 Compare November 5, 2026 08:54
@wan9chi
wan9chi added this pull request to stack #942 November 5, 2026 08:55
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

fspy benchmark

linux

dynamic/launch             change  +0.41%  [ -2.58% ..  +4.68%]  overhead   +58.06%
dynamic/access             change  +0.43%  [ -4.43% ..  +5.45%]  overhead   +20.68%
dynamic/access-relative    change  -0.31%  [ -3.56% ..  +2.17%]  overhead   +70.01%
dynamic/access-contended   change  +0.53%  [-11.21% ..  +7.48%]  overhead   +34.68%
static/launch              change  -0.05%  [ -5.31% ..  +6.59%]  overhead  +199.04%
static/access              change  -0.55%  [ -2.12% ..  +2.37%]  overhead +1460.03%
static/access-relative     change  -0.58%  [ -2.63% ..  +1.27%]  overhead +1972.82%
static/access-contended    change  -0.33%  [ -2.26% ..  +0.73%]  overhead +1771.40%

macos

dynamic/launch             change  +0.26%  [ -5.11% ..  +5.70%]  overhead  +253.12%
dynamic/access             change  -2.08%  [-60.11% .. +41.18%]  overhead    +1.97%
dynamic/access-relative    change  +1.77%  [ -8.14% .. +75.98%]  overhead  +243.83%
dynamic/access-contended   change  -0.58%  [ -6.93% ..  +9.95%]  overhead    +3.85%

windows

dynamic/launch             change  -0.18%  [ -1.82% ..  +1.45%]  overhead   +26.17%
dynamic/access             change  +0.19%  [ -0.74% ..  +0.74%]  overhead    +0.94%
dynamic/access-relative    change  +0.00%  [ -0.72% ..  +0.55%]  overhead    +1.11%
dynamic/access-contended   change  +0.00%  [ -0.52% ..  +0.70%]  overhead    +1.25%

@wan9chi
wan9chi force-pushed the feat/cache-input-negatives-in-listings branch from 157bf82 to 5bcf4de Compare November 5, 2026 09:02
…ory listings

Two cases for a task that lists a folder in which an entry its input
negatives exclude appears later. In `negative_glob_hides_listing_entry`,
the task lists the package root with `vtt list-dir .` and has `"!out/**"`
in its `input`; an `out` folder appears, then a `notes.txt` file. In
`oxlint_negative_build`, `oxlint -D no-debugger` lints a package whose
`.gitignore` lists `build/`, another task's output, so oxlint skips it; it
lists the package root, and the lint task has `"!build/**"` in its
`input`. Another task's `build` folder appears, holding a file oxlint would
reject if it didn't skip `build/`.

Today an input negative only drops the paths the task read, not entries
inside a folder it listed, so the new folder is a cache miss in both
cases, although `!out/**` and `!build/**` exclude it and everything in it.
The new `notes.txt` is a miss too, as it should be.

The oxlint steps redact oxlint's rule count, which changes between oxlint
releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An exclusion in a task's `input`, like `"!build/**"`, and a path a tool
reports with `ignoreInput` left out only the paths the task read. A
folder the task listed still recorded every entry, so an excluded folder
appearing in it was a cache miss. That is what happens to a lint task
listing the package root when another task's build output appears there,
the case the hard-coded `dist` skip in every listing was added for (#181),
and `"!dist/**"`, which `docs/inputs.md` recommends, didn't help.

Listings now leave out what the input config excludes, after the entries
the task can rebuild are left out:

- An entry whose path the input negative globs match. A folder entry is
  matched by its own path, and wax's `**` matches zero or more
  components, so `build/**` leaves out a `build` folder, while `build/*`
  or `build/**/*.js` exclude only what is in it and keep the entry. The
  globs are part of the cache key, so validation compiles the same ones
  from the entry's key and applies them to the current listing; the names
  aren't recorded, and an entry that appears after the run is left out
  too.
- An entry an `ignoreInput` path names. Those paths are known only after
  a run, so the names they give a listed folder's entries, whether or not
  the entries are there, are recorded in its `ignored_entries`, which
  validation already leaves out of the current listing. A path covers
  itself and everything under it, but a listed folder is never under one,
  since reads under it are left out, so only a path naming an entry of the
  folder matters.

Rebuildable entries and `.DS_Store` work as before. The serialized format
doesn't change, but validation does, so the cache schema version is
bumped. `PathGlobSet` gets `Default`, an empty set for tasks without
fspy, and `is_empty`, so listings skip glob matching when there are no
negatives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi
wan9chi force-pushed the feat/cache-input-negatives-in-listings branch from 5bcf4de to 0655edb Compare November 5, 2026 09:17
@wan9chi
wan9chi merged commit d6754e8 into main Nov 5, 2026
22 checks passed
@wan9chi
wan9chi deleted the feat/cache-input-negatives-in-listings branch November 5, 2026 10:44
wan9chi added a commit that referenced this pull request Nov 5, 2026
## Motivation

Every directory listing skips an entry named `dist`. #181 added the skip
so that a lint task listing the package root stays a cache hit when
another task's build creates `dist` there. It helps no other folder name
and hides a `dist` that does matter. A `dist` folder appearing in a
folder the task listed goes unseen, and the next run replays stale
output: `oxlint`, for example, passes from the cache after a `dist`
folder with lint errors appears.

## Changes

Listings skip only `.`, `..` and `.DS_Store`. A task that should ignore
another task's `dist` gets that back with `"!dist/**"` in its `input`,
which keeps the entry out of its listings since #941. A task's own
`dist` doesn't need the skip, since #915 makes a rebuildable folder an
ignored entry. The cache schema version is bumped, so entries recorded
with the skip aren't reused. No other snapshot changes.

E2E cases: `dist_appears_in_listed_folder` (synthetic, `vtt list-dir`)
and `oxlint_dist` (`oxlint -D no-debugger`, with oxlint's rule count
redacted). Both are false hits today and correct misses after.

Part of a stack that lets automatic tracking cache tasks which read
their own outputs and list the folders holding them, so they also hit on
a clean checkout.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes cache fingerprint semantics for any task that lists
directories, which can increase cache misses but fixes incorrect hits;
schema bump invalidates existing local cache entries for the new version
directory.
> 
> **Overview**
> Directory listings for cache fingerprints **no longer always omit a
`dist` entry**. Only `.`, `..`, and `.DS_Store` are skipped when
fingerprinting folder contents, so a **`dist` folder appearing or
disappearing** in a listed directory now triggers a **cache miss**
instead of replaying stale results (e.g. **oxlint** passing after bad
code lands in a committed `dist/`).
> 
> To keep ignoring another task’s **`dist`**, tasks can use
**`"!dist/**"`** in **`input`** (listing exclusions from #941). A task’s
**own** output **`dist`** still doesn’t affect listings when it only
holds rebuildable outputs (#915). **`CACHE_SCHEMA_VERSION`** is bumped
**22 → 23** so old fingerprints aren’t reused.
> 
> E2E coverage adds **`dist_appears_in_listed_folder`** and
**`oxlint_dist`** snapshots for the new miss behavior.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
625af30. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
wan9chi added a commit that referenced this pull request Nov 5, 2026
The entries of #913-#919 and #941 describe one feature, rebuildable
inputs for tasks whose reads overlap their writes, so they are one
entry now, with every PR linked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wan9chi added a commit that referenced this pull request Nov 5, 2026
The `dist` listing change (#919) and input exclusions applying to
listings (#941) are their own items; the rebuildable inputs item covers
#913-#918 without mentioning listings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wan9chi added a commit that referenced this pull request Nov 5, 2026
## Motivation

Follow-ups to the rebuildable-inputs stack (#913–#919, #940, #941), now
merged:

- The real-world case for files renamed into place (#916) used Vitest's
`fsModuleCache`, which renames cache entries rather than outputs.
- #919's e2e cases test a listing entry like any other, so they add no
coverage of their own.
- The stack added eight changelog entries, most of them for what users
see as one feature.
- Socket flagged Astro 6.4.8, which #917 added for its real-world case:
critical advisory GHSA-26w7-cxv4-gfx2 (remote code execution through
AVIF image optimization), fixed in Astro 7.2.8.

## Changes

- New case `next_build_moved_pages`: Next.js 15's `next build` renders
each static page (`index.html`, `404.html`, `500.html`) into
`.next/export` and renames it into `.next/server/pages`. After `.next`
is removed, the next run is a cache hit that restores the pages. The
build log is redacted, since its progress lines, sizes and chunk hashes
vary. It replaces `vitest_fs_module_cache`, along with the SHA-1
redaction only that case used. `next`, `react` and `react-dom` join
`packages/tools`.
- `dist_appears_in_listed_folder` and `oxlint_dist` are removed.
- The changelog entries of #913–#918 are combined into one **Added**
item for rebuildable inputs; #919 (`dist` in listings) and #941 (input
exclusions apply to listings) stay their own short items, and so does
#940's fix.
- On Windows CI, the e2e harness stages fixtures in `RUNNER_TEMP`, on
the checkout's drive: the staged workspaces resolve `node_modules` into
the checkout, and Next.js builds its client entry from a path relative
to the project, which can't cross drives.
- Astro 6.4.8 → 7.3.8, the newest release that still supports the repo's
Node 22.19. Astro 7 needs upstream Vite 8, so its scoped override now
points at Vite 8.3.2. `astro_empty_collections` passes unchanged; its
comment now says Astro 7 stats the project root rather than listing it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Test harness, changelog, and dependency catalog changes only; no
changes to task caching or file-tracking behavior in this diff.
> 
> **Overview**
> Follow-up after the rebuildable-inputs stack lands: **no runtime cache
logic changes**—mostly release notes, snapshot tests, and toolchain
bumps.
> 
> The **changelog** folds the former #913–#918 narrative into one
**Added** “rebuildable inputs” entry and shortens the #919 / #941
**Changed** bullets; #940 stays separate.
> 
> **E2E snapshots** swap coverage: **`next_build_moved_pages`** (Next.js
15 renaming `.next/export` → `.next/server/pages`, cache hit after
deleting `.next`) replaces **`vitest_fs_module_cache`**;
**`dist_appears_in_listed_folder`** and **`oxlint_dist`** are dropped as
redundant with listing behavior elsewhere. Snapshot **redaction** drops
SHA-1 normalization (Vitest-only). On **Windows CI**, fixtures stage
under **`RUNNER_TEMP`** so Next’s client entry paths stay on the
checkout drive.
> 
> **Deps**: catalog adds **Next 15**, **React 19**, bumps **Astro 6 →
7.3.8** (security advisory), and scopes **`astro>vite`** to upstream
**Vite 8.3.2**. The Astro empty-collections fixture comment reflects
Astro 7 **stat**-ing the project root instead of listing it.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
4c8f40c. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

read-write overlap Caching tasks that read and write the same paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant