Skip to content

feat(cache): let a listed folder lack entries a hit can rebuild - #915

Merged
wan9chi merged 2 commits into
feat/cache-rebuildable-foldersfrom
feat/cache-listing-ignored-entries
Nov 5, 2026
Merged

wan9chi merged 2 commits into
feat/cache-rebuildable-foldersfrom
feat/cache-listing-ignored-entries

Conversation

@wan9chi

@wan9chi wan9chi commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Motivation

A task that lists a folder holding its own outputs pins those outputs as entries of the listing. TypeScript 7's tsc with no include, for example, lists the package root that holds its lib output folder. A clean checkout lacks them, so it misses. Today only an entry named dist is spared, by a hard-coded skip.

Changes

When a listing is fingerprinted, the entries the task can rebuild are left out and their names recorded with the listing: a file or symlink it wrote, or a rebuildable folder. The listing becomes PathFingerprint::Folder(Option<DirListing>), with DirListing { entries, ignored_entries }, so only a listing can carry ignored names. Validation drops the same names from the current listing before comparing, so they may be missing or present. Any other entry that differs still misses and is named. On file systems that don't report entry types, an entry's kind is looked up with fstatat, so listings and the folder climb agree. The cache schema version is bumped.

E2E cases: list_folder_holding_output (synthetic; an unrelated new file still misses) and typescript_lib_out_dir (TypeScript 7, outDir: "lib").

Part of a stack that lets automatic tracking cache tasks which read their own outputs and list the folders holding them, so they also hit on a clean checkout.

🤖 Generated with Claude Code


Note

Medium Risk
Changes post-run fingerprint semantics and invalidates existing cache entries (schema v21), which affects cache hit/miss correctness for any task that lists directories containing its outputs.

Overview
When a cached task lists a parent folder that also contains its outputs, those rebuildable files and output-only subfolders no longer appear in the stored directory listing. Their names are kept in ignored_entries, and cache validation strips the same names from the live listing before comparing, so a clean checkout without lib, gen/, etc. can still cache hit and replay restores outputs.

PathFingerprint::Folder now wraps DirListing { entries, ignored_entries } instead of a plain entry map; CACHE_SCHEMA_VERSION is bumped to 21. Rebuildable gains is_rebuildable_entry (and related helpers) so listing fingerprinting matches the existing “output-only folder” logic. On Unix, directory entries without a reported type are classified via fstatat, aligning listings with rebuildable-folder detection.

New e2e fixtures cover synthetic “list package root after writing gen/” and tsc with outDir: lib; typescript_incremental is no longer Unix-only now that listing behavior is generalized.

Reviewed by Cursor Bugbot for commit c7a32cb. Bugbot is set up for automated code reviews on this repo. Configure here.

@wan9chi
wan9chi added this pull request to stack #920 October 30, 2026 03:17
@wan9chi
wan9chi force-pushed the feat/cache-listing-ignored-entries branch from 7f01a6e to 531c594 Compare October 30, 2026 03:19
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

fspy benchmark

linux

dynamic/launch             change  +0.76%  [ -3.85% ..  +7.08%]  overhead   +60.11%
dynamic/access             change  +0.00%  [ -8.27% ..  +7.74%]  overhead   +25.65%
dynamic/access-relative    change  +0.54%  [ -8.83% ..  +7.07%]  overhead   +75.63%
dynamic/access-contended   change  -0.15%  [-10.34% .. +10.77%]  overhead   +13.64%
static/launch              change  +0.60%  [ -5.48% ..  +6.93%]  overhead  +184.25%
static/access              change  +0.31%  [ -5.70% ..  +8.67%]  overhead +1394.89%
static/access-relative     change  -1.36%  [-14.55% ..  +2.57%]  overhead +1840.74%
static/access-contended    change  -0.20%  [ -4.31% ..  +4.45%]  overhead +1343.91%

macos

dynamic/launch             change  -1.39%  [ -6.28% ..  +3.03%]  overhead  +228.14%
dynamic/access             change  -1.54%  [-55.33% .. +86.18%]  overhead    +0.37%
dynamic/access-relative    change  +0.95%  [ -5.39% .. +30.70%]  overhead  +251.15%
dynamic/access-contended   change  -1.33%  [ -9.25% ..  +9.87%]  overhead    +0.51%

windows

dynamic/launch             change  -0.09%  [ -2.21% ..  +2.55%]  overhead   +26.31%
dynamic/access             change  +0.00%  [ -0.64% ..  +0.85%]  overhead    +1.72%
dynamic/access-relative    change  +0.00%  [ -1.89% ..  +2.09%]  overhead    +2.06%
dynamic/access-contended   change  -0.40%  [ -6.17% ..  +1.20%]  overhead    +2.24%

@wan9chi
wan9chi force-pushed the feat/cache-listing-ignored-entries branch from 531c594 to 524114b Compare October 30, 2026 04:07
@wan9chi wan9chi added the ci: full Run the slow CI jobs (macOS x64, musl) on this pull request label Oct 30, 2026
@wan9chi
wan9chi force-pushed the feat/cache-listing-ignored-entries branch from 524114b to f92e4fd Compare October 30, 2026 06:31
@wan9chi
wan9chi force-pushed the feat/cache-listing-ignored-entries branch from f92e4fd to 912534a Compare October 30, 2026 07:08
@wan9chi wan9chi added the read-write overlap Caching tasks that read and write the same paths label Nov 4, 2026
@wan9chi
wan9chi removed this pull request from stack #920 November 5, 2026 08:52
@wan9chi
wan9chi force-pushed the feat/cache-listing-ignored-entries branch from 912534a to 2181f79 Compare November 5, 2026 08:54
@wan9chi
wan9chi added this pull request to stack #942 November 5, 2026 08:55
…akes it miss

Two cases for a task that lists a folder holding its own output folder. In
`list_folder_holding_output`, the task makes `gen`, writes `gen/a.txt` and
lists the package root, through `vtt fs`. In `typescript_lib_out_dir`,
TypeScript 7's `tsc` with no `include` compiles `src` into `lib`: it lists
the package root and stats `lib`.

Today removing the output folder, as on a clean checkout, is a cache miss
naming the folder as removed from the listing, although the cached outputs
hold everything in it. A new entry in the listing is a miss too, and must
stay one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A task that lists a folder holding its own outputs, like TypeScript 7's
`tsc` with no `include` listing the package root that holds its `lib`
output folder, pinned those outputs as entries of the listing. Removing
them, as on a clean checkout, was a cache miss although the task writes
them again.

When a listing is fingerprinted, the entries the task can rebuild, a file
or symlink it wrote or a folder holding only what it can rebuild, are now
left out, and their names are recorded as the listing's
`ignored_entries`. A listed folder's fingerprint is now a `DirListing`
holding its entries and those names, so only a listing can carry them.
Validation removes the same names from the current listing before
comparing, so they may be missing or present, while any other entry that
differs still misses and is named in the message. A hit restores them if
they're among the task's outputs. `Rebuildable` answers whether an entry
is rebuildable from the written paths and the folders its walk already
found, so no folder is listed again, and a task that wrote nothing skips
the check. `.DS_Store` and `dist` stay skipped in every listing and are
never recorded.

On a file system that doesn't report entry types, a Unix listing now
stats each entry without following symlinks, as std does, instead of
taking every entry for a file, so it tells folders apart the same way
as `Rebuildable`'s walk.

The cache schema version is bumped for the new listing format.

`typescript_incremental` now runs on Windows too, where `tsc` also lists
the package root, which holds the `tsconfig.tsbuildinfo` it wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi
wan9chi force-pushed the feat/cache-listing-ignored-entries branch from 2181f79 to c7a32cb Compare November 5, 2026 09:02

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c7a32cb. Configure here.

@wan9chi
wan9chi merged commit 3f4430c into main Nov 5, 2026
22 checks passed
@wan9chi
wan9chi deleted the feat/cache-listing-ignored-entries branch November 5, 2026 10:44
wan9chi added a commit that referenced this pull request Nov 5, 2026
…gs (#941)

## Motivation

An exclusion in a task's `input`, like `"!build/**"`, leaves out only
the paths the task reads. When the task lists a folder, every entry
still counts, so an excluded folder appearing there is a cache miss: a
lint task listing the package root misses once another task's build
output appears in it. That is the case the hard-coded `dist` skip was
added for (#181). #919 removes the skip, so users need a setting to get
it back, and today `"!dist/**"`, which `docs/inputs.md` recommends,
doesn't help.

## Changes

Listings leave out what the input config excludes, after the entries the
task can rebuild (#915):
- An entry whose path the input negative globs match. A folder entry is
matched by its own path; wax's `**` matches zero or more components, so
`build/**` leaves out `build`, while `build/*` keeps it. The globs are
part of the cache key, so validation applies the same ones to the
current listing, and an entry that appears after the run is left out
too.
- An entry that a tool-reported `ignoreInput` path names. Those paths
are known only after the run, so the names are recorded in the listing's
`ignored_entries`, including entries that aren't there yet.

The cache schema version is bumped. `docs/inputs.md` now says negatives
also keep entries out of listings.

E2E cases: `negative_glob_hides_listing_entry` (synthetic, `vtt
list-dir` with `"!out/**"`; an unrelated new file still misses) and
`oxlint_negative_build` (`oxlint -D no-debugger` with a gitignored
`build/` and `"!build/**"`). Both miss today and hit after. A unit test
covers `ignoreInput`.

Part of a stack that lets automatic tracking cache tasks which read
their own outputs and list the folders holding them, so they also hit on
a clean checkout.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes post-run fingerprint semantics and cache validation for any
task that lists directories with input negatives or ignoreInput;
incorrect glob matching could cause stale cache hits, mitigated by
schema bump and targeted tests.
> 
> **Overview**
> **Input exclusions now apply to directory listings**, not only to
paths the task reads. Tasks that list a folder (e.g. a linter scanning
the package root) no longer miss the cache when an excluded entry
appears—such as another task’s `build/` folder after `"!build/**"`—as
long as nothing else in the listing changed.
> 
> Post-run fingerprints **omit listing entries** matched by the task’s
input negative globs (same rules as reads: `build/**` drops the `build`
folder entry; `build/*` does not) and by tool **`ignoreInput`** paths
(names recorded in `ignored_entries` for validation). Cache hits
recompile those globs from the entry key and apply the same filters when
comparing listings.
> 
> **Cache schema version is bumped to v22** (fresh cache). Docs and
changelog describe the behavior; e2e covers `!out/**` with `list-dir`
and an oxlint + `!build/**` scenario, plus a unit test for globs and
`ignoreInput`.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
0655edb. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
wan9chi added a commit that referenced this pull request Nov 5, 2026
## Motivation

Every directory listing skips an entry named `dist`. #181 added the skip
so that a lint task listing the package root stays a cache hit when
another task's build creates `dist` there. It helps no other folder name
and hides a `dist` that does matter. A `dist` folder appearing in a
folder the task listed goes unseen, and the next run replays stale
output: `oxlint`, for example, passes from the cache after a `dist`
folder with lint errors appears.

## Changes

Listings skip only `.`, `..` and `.DS_Store`. A task that should ignore
another task's `dist` gets that back with `"!dist/**"` in its `input`,
which keeps the entry out of its listings since #941. A task's own
`dist` doesn't need the skip, since #915 makes a rebuildable folder an
ignored entry. The cache schema version is bumped, so entries recorded
with the skip aren't reused. No other snapshot changes.

E2E cases: `dist_appears_in_listed_folder` (synthetic, `vtt list-dir`)
and `oxlint_dist` (`oxlint -D no-debugger`, with oxlint's rule count
redacted). Both are false hits today and correct misses after.

Part of a stack that lets automatic tracking cache tasks which read
their own outputs and list the folders holding them, so they also hit on
a clean checkout.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes cache fingerprint semantics for any task that lists
directories, which can increase cache misses but fixes incorrect hits;
schema bump invalidates existing local cache entries for the new version
directory.
> 
> **Overview**
> Directory listings for cache fingerprints **no longer always omit a
`dist` entry**. Only `.`, `..`, and `.DS_Store` are skipped when
fingerprinting folder contents, so a **`dist` folder appearing or
disappearing** in a listed directory now triggers a **cache miss**
instead of replaying stale results (e.g. **oxlint** passing after bad
code lands in a committed `dist/`).
> 
> To keep ignoring another task’s **`dist`**, tasks can use
**`"!dist/**"`** in **`input`** (listing exclusions from #941). A task’s
**own** output **`dist`** still doesn’t affect listings when it only
holds rebuildable outputs (#915). **`CACHE_SCHEMA_VERSION`** is bumped
**22 → 23** so old fingerprints aren’t reused.
> 
> E2E coverage adds **`dist_appears_in_listed_folder`** and
**`oxlint_dist`** snapshots for the new miss behavior.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
625af30. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci: full Run the slow CI jobs (macOS x64, musl) on this pull request read-write overlap Caching tasks that read and write the same paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant