feat(cache): let a hit rebuild folders that hold only the task's outputs

CHANGELOG.md

@@ -1,5 +1,6 @@
 # Changelog
 
+- **Changed** A task that checks or lists a folder holding only files it wrote, like `tsc` checking its `dist` output folder, is now a cache hit when the folder is missing, as on a fresh clone, and the hit restores the folder if its files are among the task's outputs. Previously the missing folder was a cache miss ([#914](https://github.com/voidzero-dev/vite-task/pull/914)).
 - **Fixed** On Linux and macOS, a task that writes outside the workspace through a relative path with `..`, like `cp -r dist ../site`, no longer fails with `Cache update failed: paths in archives must not have ..`, and is now cached. Like any other path outside the workspace, what the task writes there isn't among its outputs, and what it reads there isn't among its inputs ([#940](https://github.com/voidzero-dev/vite-task/pull/940)).
 - **Fixed** Tasks that read the task cache folder, `node_modules/.vite/task-cache`, now hit the cache. `vp run` changes that folder after every run, and tools such as Parcel and Gatsby read it once they rebuild with their own cache in place, so from then on those tasks missed every time with a reason like `'node_modules/.vite/task-cache/v20/last-summary.json' modified` ([#937](https://github.com/voidzero-dev/vite-task/pull/937)).
 - **Changed** Tasks that write a file they also read are now cached, instead of being reported as `not cached because it modified its inputs`. For example, `tsc --incremental` reads and rewrites `tsconfig.tsbuildinfo`: the next run is a cache hit if the file is as `tsc` left it, or if it is missing, as on a fresh clone, in which case the hit restores it ([#913](https://github.com/voidzero-dev/vite-task/pull/913)).

crates/vt/docs/task-cache.md

@@ -197,6 +197,7 @@ Vite Task uses `fspy` to monitor file system access during task execution:
 │    • Check if file exists                                    │
 │    • If file: Hash contents with xxHash3                     │
 │    • If directory: Record structure                          │
+│      (skipped if it holds only files the task wrote)         │
 │    • If missing: Mark as NotFound                            │
 │         │                                                    │
 │         ▼                                                    │

crates/vt/src/session/execute/cache_update.rs

@@ -33,8 +33,8 @@ struct TrackingOutcome {
     /// Auto-output writes after output exclusions are applied. Empty when
     /// `output_config.includes_auto` is false.
     path_writes: FxHashSet<RelativePathBuf>,
-    /// Every write fspy recorded, before output exclusions and `ignoreOutput`
-    /// paths are applied.
+    /// What the task can rebuild, from every write fspy recorded before output
+    /// exclusions and `ignoreOutput` paths are applied.
     rebuildable: Rebuildable,
 }
 
@@ -256,10 +256,15 @@ fn observe_fspy(
                 } else {
                     FxHashSet::default()
                 };
+            let mut rebuildable = Rebuildable::new(tracked.path_writes);
+            // Only fingerprints of the paths the task read ask about folders.
+            if !filtered_path_reads.is_empty() {
+                rebuildable = rebuildable.with_folders(workspace_root);
+            }
             TrackingOutcome {
                 path_reads: filtered_path_reads,
                 path_writes: filtered_path_writes,
-                rebuildable: Rebuildable::new(tracked.path_writes),
+                rebuildable,
             }
         })
     }

crates/vt/src/session/execute/fingerprint.rs

@@ -117,7 +117,9 @@ impl PostRunFingerprint {
     /// Negative glob filtering is done upstream (see
     /// [`super::tracked_accesses::TrackedPathAccesses::from_raw`]).
     /// Paths already present in `globbed_inputs` are skipped — they are
-    /// already tracked by the prerun glob fingerprint. Every other path is
+    /// already tracked by the prerun glob fingerprint. Folders holding only
+    /// what the task can rebuild are skipped too: a hit restores the files in
+    /// them if they're among the task's outputs. Every other path is
     /// fingerprinted as the task left it, including a file the task rewrote.
     ///
     /// # Arguments
@@ -138,7 +140,9 @@ impl PostRunFingerprint {
     ) -> anyhow::Result<Self> {
         let inferred_inputs = inferred_path_reads
             .par_iter()
-            .filter(|(path, _)| !globbed_inputs.contains_key(*path))
+            .filter(|(path, _)| {
+                !globbed_inputs.contains_key(*path) && !rebuildable.is_rebuildable_folder(path)
+            })
             .map(|(relative_path, path_read)| {
                 let full_path = Arc::<AbsolutePath>::from(base_dir.join(relative_path));
                 let fingerprint = fingerprint_path(&full_path, *path_read)?;
@@ -708,4 +712,54 @@ mod tests {
             Some(PostRunMismatch::Input { kind: InputChangeKind::ContentModified, .. })
         ));
     }
+
+    #[test]
+    fn create_skips_folders_holding_only_written_files() {
+        let tmp = tempfile::TempDir::new().unwrap();
+        let workspace_root = AbsolutePathBuf::new(tmp.path().to_path_buf()).unwrap();
+        let rel = |path: &str| RelativePathBuf::new(path).unwrap();
+        for dir in ["out/sub", "mixed", "holds_empty/empty"] {
+            std::fs::create_dir_all(workspace_root.join(dir).as_path()).unwrap();
+        }
+        let written = ["out/a.txt", "out/sub/b.txt", "mixed/a.txt", "holds_empty/a.txt"];
+        for file in written.iter().copied().chain(["out/.DS_Store", "mixed/notes.txt"]) {
+            std::fs::write(workspace_root.join(file).as_path(), "x").unwrap();
+        }
+
+        let path_reads = ["out", "out/sub", "mixed", "holds_empty"]
+            .into_iter()
+            .map(|dir| (rel(dir), PathRead { read_dir_entries: true }))
+            .collect();
+        let fingerprint = PostRunFingerprint::create(
+            &path_reads,
+            &Rebuildable::new(written.into_iter().map(rel).collect()).with_folders(&workspace_root),
+            &workspace_root,
+            &BTreeMap::new(),
+            BTreeMap::new(),
+            BTreeMap::new(),
+        )
+        .unwrap();
+
+        let mut fingerprinted: Vec<&str> =
+            fingerprint.inferred_inputs.keys().map(|path| path.as_str()).collect();
+        fingerprinted.sort_unstable();
+        assert_eq!(fingerprinted, ["holds_empty", "mixed"]);
+    }
+
+    #[test]
+    fn folders_gone_after_the_run_play_no_part() {
+        let tmp = tempfile::TempDir::new().unwrap();
+        let workspace_root = AbsolutePathBuf::new(tmp.path().to_path_buf()).unwrap();
+        let rel = |path: &str| RelativePathBuf::new(path).unwrap();
+        std::fs::create_dir(workspace_root.join("out").as_path()).unwrap();
+        std::fs::write(workspace_root.join("out/a.txt").as_path(), "x").unwrap();
+
+        // `out/tmp` and the file the task wrote in it are gone after the run,
+        // like Astro's `dist/.prerender`.
+        let rebuildable =
+            Rebuildable::new(["out/a.txt", "out/tmp/b.txt"].into_iter().map(rel).collect())
+                .with_folders(&workspace_root);
+        assert!(rebuildable.is_rebuildable_folder(&rel("out")));
+        assert!(!rebuildable.is_rebuildable_folder(&rel("out/tmp")));
+    }
 }

crates/vt/src/session/execute/rebuildable.rs

@@ -6,26 +6,150 @@
 //! would write them again, so a later run may find them missing. A file that
 //! is there must still be as the task left it.
 //!
+//! The same goes for a folder that holds only what the task can rebuild:
+//! every file (or symlink) in it is one the task wrote, and every folder in
+//! it holds only what the task can rebuild too. It isn't fingerprinted at
+//! all. A hit restores its files if they're among the task's outputs, which
+//! recreates it; explicit output config can leave them out. Such folders are
+//! found after the run by climbing from the written paths through their
+//! parent folders inside the workspace, listing each one in the real file
+//! system. A written path or folder that is gone by then plays no part.
+//!
 //! The written paths are fspy's records before output exclusions and
 //! tool-reported `ignoreOutput` paths are applied: those decide what is
 //! archived, not what the task can rebuild.
 
+use std::{cmp::Reverse, fs, io};
+
 use rustc_hash::FxHashSet;
-use vt_path::{RelativePath, RelativePathBuf};
+use vt_path::{AbsolutePath, RelativePath, RelativePathBuf};
 
-/// The paths a task wrote, relative to the workspace root.
+/// The paths a task wrote and the folders holding only what it can rebuild,
+/// relative to the workspace root.
 #[derive(Debug, Default)]
 pub struct Rebuildable {
     written: FxHashSet<RelativePathBuf>,
+    folders: FxHashSet<RelativePathBuf>,
 }
 
 impl Rebuildable {
-    pub const fn new(written: FxHashSet<RelativePathBuf>) -> Self {
-        Self { written }
+    /// Takes the `written` paths. No folder is rebuildable until
+    /// [`Self::with_folders`] finds them.
+    pub fn new(written: FxHashSet<RelativePathBuf>) -> Self {
+        Self { written, folders: FxHashSet::default() }
+    }
+
+    /// Finds the rebuildable folders above the written paths by listing them
+    /// under `workspace_root`, which must be done after the run. Only
+    /// fingerprints of the paths the task read ask about folders, so a run
+    /// without such reads skips this.
+    #[must_use]
+    pub fn with_folders(mut self, workspace_root: &AbsolutePath) -> Self {
+        self.folders = find_folders(&self.written, workspace_root);
+        self
     }
 
     /// Whether a later run may find `path` missing: the task wrote it.
     pub fn may_be_missing(&self, path: &RelativePath) -> bool {
         self.written.contains(path)
     }
+
+    /// Whether `path` is a folder holding only what the task can rebuild.
+    pub fn is_rebuildable_folder(&self, path: &RelativePath) -> bool {
+        self.folders.contains(path)
+    }
+}
+
+/// Finds the folders holding only what the task can rebuild. Each folder above
+/// a written path is listed once, deepest first, so a folder's subfolders are
+/// decided before it. Climbing stops at the first folder that isn't
+/// rebuildable, since no folder above it can be, and at the workspace root. A
+/// folder that is gone plays no part: its parent is decided by the entries it
+/// holds.
+fn find_folders(
+    written: &FxHashSet<RelativePathBuf>,
+    workspace_root: &AbsolutePath,
+) -> FxHashSet<RelativePathBuf> {
+    let mut candidates = FxHashSet::default();
+    for path in written {
+        let mut dir = path.parent();
+        // A folder already reached from another path has its parents too.
+        while let Some(parent) = dir
+            && !parent.as_str().is_empty()
+            && candidates.insert(parent)
+        {
+            dir = parent.parent();
+        }
+    }
+    let mut candidates: Vec<&RelativePath> = candidates.into_iter().collect();
+    candidates.sort_unstable_by_key(|dir| Reverse(dir.as_str().matches('/').count()));
+
+    let mut folders = FxHashSet::default();
+    let mut above_unrebuildable = FxHashSet::default();
+    for dir in candidates {
+        let rebuildable = if above_unrebuildable.contains(dir) {
+            Some(false)
+        } else {
+            holds_only_rebuildable(dir, workspace_root, written, &folders)
+        };
+        match rebuildable {
+            Some(true) => {
+                folders.insert(dir.to_relative_path_buf());
+            }
+            Some(false) => {
+                if let Some(parent) = dir.parent() {
+                    above_unrebuildable.insert(parent);
+                }
+            }
+            None => {}
+        }
+    }
+    folders
+}
+
+/// Whether `dir` holds at least one entry, and every entry is a file (or
+/// symlink) in `written` or a folder in `folders`, or `None` when `dir` isn't
+/// a folder anymore. `.DS_Store` doesn't count, as in directory listing
+/// fingerprints. An empty folder isn't rebuildable: there's no written file to
+/// recreate it.
+fn holds_only_rebuildable(
+    dir: &RelativePath,
+    workspace_root: &AbsolutePath,
+    written: &FxHashSet<RelativePathBuf>,
+    folders: &FxHashSet<RelativePathBuf>,
+) -> Option<bool> {
+    let entries = match fs::read_dir(workspace_root.join(dir)) {
+        Ok(entries) => entries,
+        // Gone, or not a folder anymore: it plays no part.
+        Err(err)
+            if matches!(err.kind(), io::ErrorKind::NotFound | io::ErrorKind::NotADirectory) =>
+        {
+            return None;
+        }
+        Err(_) => return Some(false),
+    };
+    let mut is_empty = true;
+    for entry in entries {
+        let Ok(entry) = entry else {
+            return Some(false);
+        };
+        let name = entry.file_name();
+        if name == ".DS_Store" {
+            continue;
+        }
+        let (Some(name), Ok(file_type)) = (name.to_str(), entry.file_type()) else {
+            return Some(false);
+        };
+        let Ok(name) = RelativePathBuf::new(name) else {
+            return Some(false);
+        };
+        let path = dir.join(name);
+        let rebuildable =
+            if file_type.is_dir() { folders.contains(&path) } else { written.contains(&path) };
+        if !rebuildable {
+            return Some(false);
+        }
+        is_empty = false;
+    }
+    Some(!is_empty)
 }

crates/vt_bin/src/vtt/fs.rs

@@ -21,6 +21,9 @@ const USAGE: &str = "Usage: vtt fs <op> <arg>... [<op> <arg>...]...";
 /// - `walk <dir>`: read every entry of the folder and of every folder under
 ///   it, without following symlinks, like a tool crawling a project. A
 ///   missing folder isn't an error.
+/// - `rm <path>`: remove the file, or the folder and everything in it, like
+///   `rm -r`. A missing path isn't an error, like a tool cleaning its output
+///   folder.
 /// - `mkdir <dir>`: create the folder and any missing parents, like
 ///   `mkdir -p`.
 ///
@@ -41,6 +44,7 @@ pub fn run(args: &[String]) -> Result<(), Box<dyn std::error::Error>> {
             ["stat", path, rest @ ..] => (allow_missing(fs::metadata(path)), rest),
             ["list", dir, rest @ ..] => (allow_missing(list(dir)), rest),
             ["walk", dir, rest @ ..] => (allow_missing(walk(Path::new(dir))), rest),
+            ["rm", path, rest @ ..] => (allow_missing(remove(path)), rest),
             ["mkdir", dir, rest @ ..] => (fs::create_dir_all(dir), rest),
             _ => return Err(format!("{USAGE}\nunexpected arguments: {}", ops.join(" ")).into()),
         };
@@ -65,6 +69,14 @@ fn walk(dir: &Path) -> io::Result<()> {
     Ok(())
 }
 
+fn remove(path: &str) -> io::Result<()> {
+    if fs::symlink_metadata(path)?.is_dir() {
+        fs::remove_dir_all(path)
+    } else {
+        fs::remove_file(path)
+    }
+}
+
 fn allow_missing<T>(result: io::Result<T>) -> io::Result<()> {
     match result {
         Ok(_) => Ok(()),

crates/vt_bin/tests/e2e_snapshots/fixtures/clean_and_rebuild_output_folder/package.json

@@ -0,0 +1,4 @@
+{
+  "name": "clean-and-rebuild-output-folder",
+  "private": true
+}

crates/vt_bin/tests/e2e_snapshots/fixtures/clean_and_rebuild_output_folder/snapshots.toml

@@ -0,0 +1,40 @@
+[[e2e]]
+name = "clean_and_rebuild_output_folder"
+comment = """
+The task reads `src/a.txt`, lists `out`, removes it, makes it again, writes `out/a.txt` and stats `out`, like a build tool cleaning its output folder before writing a new one.
+"""
+steps = [
+  [
+    "vt",
+    "run",
+    "build",
+  ],
+  { argv = [
+    "vtt",
+    "replace-file-content",
+    "src/a.txt",
+    "v1",
+    "v2",
+  ], comment = "change an input, so the next run cleans and rebuilds the existing `out`" },
+  [
+    "vt",
+    "run",
+    "build",
+  ],
+  { argv = [
+    "vtt",
+    "rm",
+    "-r",
+    "out",
+  ], comment = "remove the output folder, as on a clean checkout" },
+  [
+    "vt",
+    "run",
+    "build",
+  ],
+  [
+    "vtt",
+    "stat-file",
+    "out/a.txt",
+  ],
+]

crates/vt_bin/tests/e2e_snapshots/fixtures/clean_and_rebuild_output_folder/snapshots/clean_and_rebuild_output_folder.md

@@ -0,0 +1,44 @@
+# clean_and_rebuild_output_folder
+
+The task reads `src/a.txt`, lists `out`, removes it, makes it again, writes `out/a.txt` and stats `out`, like a build tool cleaning its output folder before writing a new one.
+
+## `vt run build`
+
+```
+$ vtt fs read src/a.txt list out rm out mkdir out write out/a.txt hi stat out
+```
+
+## `vtt replace-file-content src/a.txt v1 v2`
+
+change an input, so the next run cleans and rebuilds the existing `out`
+
+```
+```
+
+## `vt run build`
+
+```
+$ vtt fs read src/a.txt list out rm out mkdir out write out/a.txt hi stat out ○ cache miss: 'src/a.txt' modified, executing
+```
+
+## `vtt rm -r out`
+
+remove the output folder, as on a clean checkout
+
+```
+```
+
+## `vt run build`
+
+```
+$ vtt fs read src/a.txt list out rm out mkdir out write out/a.txt hi stat out ◉ cache hit, replaying
+
+---
+vt run: cache hit.
+```
+
+## `vtt stat-file out/a.txt`
+
+```
+out/a.txt: exists
+```