Repository navigation
Conversation
59a90f5 to
416901a
Compare
ea59e3f to
c46237d
Compare
A previous implementation kept a `visited` set in the Node crawler. Instead, we can also deduplicate and remove overlapping roots (directories within other roots) in the `FileMap` preemptively for all crawlers to prevent duplicate crawls.
c46237d to
c1cea5c
Compare
…ceNodeFilesystemAPI` flag) (#1693) Summary: This is a proposal to drop the native find binary crawler code path entirely. We've recently dropped the native find binary crawler in Expo, which was accidentally activated in some cases. We discovered (which is particularly obvious on large projects/repos and `expo/expo` itself) that it's consistently much slower than the Node FS crawler. To get into the "correct" preference and use the Node crawler, we're currently forcing `useWatchman: null`, which then defaults to use watchman, if it's installed, but falls back to `forceNodeFilesystemAPI: true`. This is outside of what the types specify as valid, but is the cleanest way to disable the native find binary crawling. This crawler basically just adds overhead over the Node.js implementation and is consistently slower, and less efficient, while the Node crawler is pretty optimised already (and has very low overhead once #1677 is merged) The timings of the native find binary crawler in `expo/expo` was also observed to scale with the size of this repo, compared to a minimal reproduction, so the overhead isn't just limited to the child process call and result parsing. Changelog: [Internal] Drop native find binary crawler and remove `forceNodeFilesystemAPI` flag Pull Request resolved: #1693 Test Plan: - Existing tests should pass - Quick generated script to play around with and assess the crawlers' comparative performance: kitten@74dffd4 Reviewed By: huntie Differential Revision: D101985455 Pulled By: robhogan fbshipit-source-id: efe2ab02fc5c0edac4d68f037db57c635fef8e50
Resolves the conflict with react#1693, which removed the native `find` crawler: `findNative` is dropped, and `nodeCrawl` keeps this PR's flattened promise, calling `find` directly.
A crawl root that is the direct parent of `rootDir` normalizes to `..`, which the `'..' + path.sep` check treated as inside `rootDir`, so files under `rootDir` were recorded as `../<rootDir name>/…` instead of normal paths. This is the monorepo case of a `watchFolders` entry one level above the project. Treat `..` as outside, like any other up-fragment, so paths are derived with `absoluteToNormal` until the crawl re-enters `rootDir`. A filesystem root (`/`, `C:\`) already ends with a separator, so appending another produced `//name` for `lstat` and `ignore`, and `removeOverlappingRoots` compared nested roots against `//` and kept them. Only append a separator when there isn't one. Also imports `node:path` in `removeOverlappingRoots`, per lint.
Keep `main`'s `Array.from(new Set(options.roots))` for now, leaving this PR to the crawler hot path. Deduplicating nested roots can follow separately. Also removes the committed `types/` declaration, which `main` no longer uses.
Only a root can already end with a separator (a filesystem root), so check that once per root and pass each directory's prefix down to its children, instead of checking every directory as we recurse.
`exts` was a plain object, so a lookup also found properties inherited from `Object.prototype`: a file such as `foo.constructor` or `foo.toString` was treated as having a listed extension, `lstat`ed and added to the file map. A `Set` has no inherited keys, and is as fast here - within about 1ns a lookup of the object, and much faster than `extensions.includes`, measured over the extension of every entry in a 53k-file crawl with 35 to 100 extensions.
Measured over the Dirent arrays of a 61k-entry crawl, `for...of` is within ~0.25ns an entry of the indexed loop once optimised, and ~0.2ms a crawl slower on a single cold pass, which is within run-to-run noise and well below what's measurable end to end.
With a single crawler implementation, `find` no longer needs a callback and positional arguments shared with an alternative: it takes the crawler options and returns a promise of the file data, which `nodeCrawl` awaits. Completion is checked in one place, which also covers an empty `roots` without a special case. Also rewrites the comment on deriving normal paths to cover both the within-rootDir and outside-rootDir cases, and takes the extension from the entry name.
|
I've made a few superficial changes here and split out the root-deduplication, since that's independent of the other wins and I'd like to make sure the "watchman root within a non-watchman root" is handled to avoid any possibility of regression. But otherwise, this looks good - the headline is ~33% faster crawl, which merits a Changes I've made:
Claude-driven benchmarksBenchmarked the current head (3c9e509) against
Bold where the CI excludes zero. |
Resolves conflicts with react#1677, which rewrote the Node crawler: its crawler already checks the extension before `lstat` and builds normal paths per directory, so this PR's crawler change reduces to skipping the `lstat` for a file with no previous mtime, reading `previousState` from the crawler options. The crawler tests follow: new files are now recorded without an `lstat`, the filesystem root test checks the path passed to `ignore()` rather than `lstat`, and `forceNodeFilesystemAPI` (removed by react#1693) is dropped from this PR's new tests. Also updates the API snapshot for `FileSystem.getMtimeByNormalPath`.
Summary: Publish Metro 0.87.2 from `main`. Changelog: [Internal] Test plan: ``` node scripts/updateVersion.js 0.87.2 sl diff --stat ``` Only the 17 `packages/*/package.json` manifests change, and every `0.87.1` reference is replaced. Draft release notes (from the 95 commits in v0.87.1...main - internal-only changes omitted): ``` - **[Feature]**: Add a built-in `metro:runtime/*` specifier, resolvable from any importer, eg for `transformer.asyncRequireModulePath` (#2083 by @robhogan) - **[Feature]**: When `transformer.enableBabelRuntime` is `true`, pass Metro's own `@babel/runtime` (`metro:babel-runtime`) and its installed version to Babel presets via caller data (#1957 by @robhogan) - **[Fix]**: Resolver: Fix stale resolutions after adding a nearer `package.json` or retargeting a symlinked directory, which previously needed a restart (#1948 by @robhogan) - **[Fix]**: Resolver: Packages and directories named with an asset extension (eg `lodash.zip`) resolve as modules rather than failing as assets (#2055 by @robhogan) - **[Fix]**: Resolver: Fall through to parent conditions when a nested `"exports"` or `"imports"` condition has no matching target (#2074 by @maximilliangrand) - **[Fix]**: Resolver: An exact `"exports"` or `"imports"` subpath with a `null` or unmatched target no longer resolves through a less specific subpath pattern (#1987 by @kwy404) - **[Fix]**: Use indexed watch folder paths (`[metro-watchFolders]/N/`) for assets outside `projectRoot` (#1892 by @robhogan) - **[Fix]**: Fix `/[metro-project]/` and `/[metro-watchFolders]/` bundle serving on Windows (#2015 by @motiz88) - **[Fix]**: Prevent HMR crash on `[metro-watchFolders]` or `[metro-project]` URLs (#2017 by @motiz88) - **[Fix]**: Remove erroneous `[metro-project]` and `[metro-watchFolders]` resolution of entry points from the API and CLI (#2030 by @motiz88) - **[Fix]**: Fix backslashes in async bundle paths on Windows (`includeAsyncPaths`) (#1938 by @robhogan) - **[Fix]**: Fix stale modules on the client when a file is deleted and recreated during a build, or changes during a build that fails (#2007 by @YevheniiKotyrlo) - **[Fix]**: Fix files created or modified while Metro starts up being missed until restart (#2054 by @cpruijsen) - **[Fix]**: Fix watch mode losing files when a directory replaces a file or symlink, eg when unlinking a package (#2009 by @robhogan) - **[Fix]**: Fix the fallback (Linux/Windows) watcher hanging on stop, and missing changes in a replaced directory, after a watch error (#1967 by @robhogan) - **[Fix]**: Fix the fallback watcher missing changes when a directory changes faster than Windows can report (#1967 by @robhogan) - **[Fix]**: Close the watch handles of a deleted directory and every directory under it in `FallbackWatcher`, which kept the process busy on Windows (#2004 by @YevheniiKotyrlo) - **[Fix]**: `metro-file-map`: Fix removing a non-active duplicate mock changing which mock is active (#1756 by @durvesh1992) - **[Fix]**: Propagate transformer initialisation errors from `Bundler` (#1854 by @Boulea7) - **[Fix]**: Release transform workers and file watchers when Metro fails to initialise (#1963 by @robhogan) - **[Fix]**: Ignore hyphenated and namespaced SVG attributes (eg `stroke-width`) when reading SVG asset dimensions (#1972 by @GabrielDumbrava) - **[Fix]**: Fix generated positions when iterating the mappings of indexed source maps, eg in `composeSourceMaps` (#1976 by @kwy404) - **[Fix]**: `composeSourceMaps` keeps every mapping at a generated column that the last map maps more than once (#1981 by @robhogan) - **[Fix]**: Source maps no longer lose lines after a gap of more than about two million unmapped generated lines (#1983 by @robhogan) - **[Fix]**: `metro-symbolicate` no longer fails on stack frames with no source position, such as `[native code]:0:0` in crash reporter output (#1891 by @robhogan) - **[Fix]**: Fix `metro-symbolicate` parsing of stack frames with Windows absolute paths (#1939 by @robhogan) - **[Performance]**: `metro-source-map`: Faster `Consumer` decoding and position lookups, and `composeSourceMaps` encodes its output directly - 3.2x faster composition of a bundle map with its Hermes map, using 15% less memory (#1978, #1979, #1980, #1983 by @robhogan) - **[Performance]**: Resolver: Ask for the package of a directory once for all extension candidates, and check `hierarchicalLookup` candidates from their own file map nodes. First-pass resolution is 35% faster (172ms -> 113ms on a mid-size RN app), net of a ~3% first-build cost from dropping `PackageCache`'s closest-package memo, which also cuts the resolver's retained heap by 37% (#1945, #1946, #1948 by @robhogan) - **[Performance]**: 33% faster non-Watchman file system crawl on startup (#1677 by @kitten) - **[Deprecated]**: `server.enhanceMiddleware` logs a deprecation warning when set to anything other than its default. Use `server.unstable_middleware` or `server.unstable_priorityMiddleware` (#2031 by @vzaidman) - **[Deprecated]**: `runServer`'s `unstable_extraMiddleware` option. Use `unstable_priorityMiddleware`, which runs the same middleware in the same place (#2031 by @vzaidman) - **[Deprecated]**: String values of `transformer.enableBabelRuntime` (use `true`) (#1957 by @robhogan) - **[Types]**: Add TypeScript definitions for `metro-runtime` modules, and generate (rather than hand-maintain) types for `metro-file-map` workers (#1996 by @robhogan) > NOTE: Experimental features are not covered by semver and can change at any time. - **[Experimental]**: Add `server.unstable_middleware` and `server.unstable_priorityMiddleware` config options, and `runServer` options of the same names, to add middleware after and before Metro's own. `mergeConfig` combines them from both configs: a later config's `unstable_middleware` runs last and its `unstable_priorityMiddleware` runs first (#2031 by @vzaidman) - **[Experimental]**: Add `server.unstable_onServerCreated` config option and `runServer` option of the same name, called with Metro's server instance before the HTTP server listens (#2031 by @vzaidman) - **[Experimental]**: Fix mismatched asset paths in (release) builds under `unstable_serverRoot`, make them always relative to `projectRoot` (#1917 by @robhogan) **Full Changelog**: v0.87.1...v0.87.2 ```
Summary: Publish Metro 0.87.2 from `main`. Changelog: [Internal] Test plan: ``` node scripts/updateVersion.js 0.87.2 sl diff --stat ``` Only the 17 `packages/*/package.json` manifests change, and every `0.87.1` reference is replaced. Draft release notes (from the 95 commits in v0.87.1...main - internal-only changes omitted): ``` - **[Feature]**: Add a built-in `metro:runtime/*` specifier, resolvable from any importer, eg for `transformer.asyncRequireModulePath` (#2083 by @robhogan) - **[Feature]**: When `transformer.enableBabelRuntime` is `true`, pass Metro's own `@babel/runtime` (`metro:babel-runtime`) and its installed version to Babel presets via caller data (#1957 by @robhogan) - **[Fix]**: Resolver: Fix stale resolutions after adding a nearer `package.json` or retargeting a symlinked directory, which previously needed a restart (#1948 by @robhogan) - **[Fix]**: Resolver: Packages and directories named with an asset extension (eg `lodash.zip`) resolve as modules rather than failing as assets (#2055 by @robhogan) - **[Fix]**: Resolver: Fall through to parent conditions when a nested `"exports"` or `"imports"` condition has no matching target (#2074 by @maximilliangrand) - **[Fix]**: Resolver: An exact `"exports"` or `"imports"` subpath with a `null` or unmatched target no longer resolves through a less specific subpath pattern (#1987 by @kwy404) - **[Fix]**: Use indexed watch folder paths (`[metro-watchFolders]/N/`) for assets outside `projectRoot` (#1892 by @robhogan) - **[Fix]**: Fix `/[metro-project]/` and `/[metro-watchFolders]/` bundle serving on Windows (#2015 by @motiz88) - **[Fix]**: Prevent HMR crash on `[metro-watchFolders]` or `[metro-project]` URLs (#2017 by @motiz88) - **[Fix]**: Remove erroneous `[metro-project]` and `[metro-watchFolders]` resolution of entry points from the API and CLI (#2030 by @motiz88) - **[Fix]**: Fix backslashes in async bundle paths on Windows (`includeAsyncPaths`) (#1938 by @robhogan) - **[Fix]**: Fix stale modules on the client when a file is deleted and recreated during a build, or changes during a build that fails (#2007 by @YevheniiKotyrlo) - **[Fix]**: Fix files created or modified while Metro starts up being missed until restart (#2054 by @cpruijsen) - **[Fix]**: Fix watch mode losing files when a directory replaces a file or symlink, eg when unlinking a package (#2009 by @robhogan) - **[Fix]**: Fix the fallback (Linux/Windows) watcher hanging on stop, and missing changes in a replaced directory, after a watch error (#1967 by @robhogan) - **[Fix]**: Fix the fallback watcher missing changes when a directory changes faster than Windows can report (#1967 by @robhogan) - **[Fix]**: Close the watch handles of a deleted directory and every directory under it in `FallbackWatcher`, which kept the process busy on Windows (#2004 by @YevheniiKotyrlo) - **[Fix]**: `metro-file-map`: Fix removing a non-active duplicate mock changing which mock is active (#1756 by @durvesh1992) - **[Fix]**: Propagate transformer initialisation errors from `Bundler` (#1854 by @Boulea7) - **[Fix]**: Release transform workers and file watchers when Metro fails to initialise (#1963 by @robhogan) - **[Fix]**: Ignore hyphenated and namespaced SVG attributes (eg `stroke-width`) when reading SVG asset dimensions (#1972 by @GabrielDumbrava) - **[Fix]**: Fix generated positions when iterating the mappings of indexed source maps, eg in `composeSourceMaps` (#1976 by @kwy404) - **[Fix]**: `composeSourceMaps` keeps every mapping at a generated column that the last map maps more than once (#1981 by @robhogan) - **[Fix]**: Source maps no longer lose lines after a gap of more than about two million unmapped generated lines (#1983 by @robhogan) - **[Fix]**: `metro-symbolicate` no longer fails on stack frames with no source position, such as `[native code]:0:0` in crash reporter output (#1891 by @robhogan) - **[Fix]**: Fix `metro-symbolicate` parsing of stack frames with Windows absolute paths (#1939 by @robhogan) - **[Performance]**: `metro-source-map`: Faster `Consumer` decoding and position lookups, and `composeSourceMaps` encodes its output directly - ~3.2x faster composition of a bundle map with its Hermes map, using ~15% less memory (#1978, #1979, #1980, #1983 by @robhogan) - **[Performance]**: Resolver: Ask for the package of a directory once for all extension candidates, and check `hierarchicalLookup` candidates from their own file map nodes. First-pass resolution is ~35% faster, net of a ~3% first-build cost from dropping `PackageCache`'s closest-package memo, which also cuts the resolver's retained heap by ~37% (#1945, #1946, #1948 by @robhogan) - **[Performance]**: ~33% faster non-Watchman file system crawl on startup (#1677 by @kitten) - **[Deprecated]**: `server.enhanceMiddleware` logs a deprecation warning when set to anything other than its default. Use `server.unstable_middleware` or `server.unstable_priorityMiddleware` (#2031 by @vzaidman) - **[Deprecated]**: `runServer`'s `unstable_extraMiddleware` option. Use `unstable_priorityMiddleware`, which runs the same middleware in the same place (#2031 by @vzaidman) - **[Deprecated]**: String values of `transformer.enableBabelRuntime` (use `true`) (#1957 by @robhogan) - **[Types]**: Add TypeScript definitions for `metro-runtime` modules, and generate (rather than hand-maintain) types for `metro-file-map` workers (#1996 by @robhogan) > NOTE: Experimental features are not covered by semver and can change at any time. - **[Experimental]**: Add `server.unstable_middleware` and `server.unstable_priorityMiddleware` config options, and `runServer` options of the same names, to add middleware after and before Metro's own. `mergeConfig` combines them from both configs: a later config's `unstable_middleware` runs last and its `unstable_priorityMiddleware` runs first (#2031 by @vzaidman) - **[Experimental]**: Add `server.unstable_onServerCreated` config option and `runServer` option of the same name, called with Metro's server instance before the HTTP server listens (#2031 by @vzaidman) - **[Experimental]**: Fix mismatched asset paths in (release) builds under `unstable_serverRoot`, make them always relative to `projectRoot` (#1917 by @robhogan) **Full Changelog**: v0.87.1...v0.87.2 ```
This fixes three regressions since 0.87.1, all in paths that reach a file or directory other than the way the file map stores it. ## Symlinked roots on a warm start Since #2064, `build()` drops cached files outside the configured roots on a cache hit, so that files from an earlier instance's `addRoot` don't survive. Watchman lists a root's files under its real path, though, so every file of a root given through a symlink (eg a watch folder like `node_modules/my-linked-lib`) is outside it. They're dropped on every warm start and not re-crawled, so imports into that package fail until `--reset-cache`. This also treats each root's real path as a root, at the cost of one `realpath` per root on a cache hit. A custom crawler's file system need not be the disk's, so its roots are taken as given. ## `hierarchicalLookup` through a symlink to an ancestor of the root Once the walk reaches an ancestor of the root, it took that ancestor's path by slicing `..` segments off the start path, which is wrong when the start path reaches it through a symlink instead (eg `app/monorepo -> ..`). That was a harmless miss until #1946 began resuming from the ancestor's node, since when the closest `package.json` of `app/monorepo/shared` is reported at `app/mo/package.json` and resolution fails with `ENOENT`. This uses the ancestor's canonical path, and the start path's real path relative to it. ## Unnormalised roots in the Node crawler Since #1677 the Node crawler builds paths by appending names to the root as given, so a root with a trailing separator (`path.join(__dirname, 'packages/')`) produced keys like `packages//a.js`, and one with `..` segments reached `ignore` unnormalised, defeating `blockList` patterns built from resolved paths. Roots are now resolved first. Changelog: Internal ## Test plan New unit tests for each, which fail without this change. With real Watchman on macOS, a watch folder linked from `node_modules` keeps all 3 of its files on a warm start, against 1 on `main` and 3 on 0.87.1. A differential fuzz of `hierarchicalLookup` (50k lookups in random trees with symlinks) against the code before #1946 and a simple real-path model: wherever this differs from the code before #1946, it matches the model. Those cases are the regression and lookups that previously threw `Failed to collapse`. The Node crawler on a real tree gives `sub/a.js` instead of `sub//a.js`, and now excludes a blocked `../shared/dist`.
Fixes three regressions since 0.87.1: - #2064 drops cached files outside the roots on a warm start. Watchman lists a symlinked root's files under its real path, so they're all dropped, and imports into it fail until `--reset-cache`: ```js watchFolders: [path.resolve(__dirname, 'node_modules/my-linked-lib')] ``` Real paths of roots now count too (built-in crawlers only). - #1946 broke `hierarchicalLookup` through a symlink in a root to its parent: ``` app/monorepo -> .. (watchFolders: ['/repo']) import './monorepo/shared' -> ENOENT: /repo/app/mo/package.json ``` - #1677 stopped the Node crawler normalising roots: ``` roots: ['/proj/sub/'] -> 'sub//a.js' roots: ['/proj/other/../shared'] -> blockList for '/shared/dist' ignored ``` Changelog: Internal Test plan: - New unit tests for each, failing without this change. - Real Watchman, watch folder linked from `node_modules`: 3/3 files on a warm start (`main`: 1/3). - `hierarchicalLookup` fuzz, 50k lookups in random symlinked trees: every difference from before #1946 matches a real-path model.
…2093) Fixes three regressions since 0.87.1: - #2064 drops cached files outside the roots on a warm start. Watchman lists a symlinked root's files under its real path, so they're all dropped, and imports into it fail until `--reset-cache`: ```js watchFolders: [path.resolve(__dirname, 'node_modules/my-linked-lib')] ``` Real paths of roots now count too (built-in crawlers only). - #1946 broke `hierarchicalLookup` through a symlink in a root to its parent: ``` app/monorepo -> .. (watchFolders: ['/repo']) import './monorepo/shared' -> ENOENT: /repo/app/mo/package.json ``` - #1677 stopped the Node crawler normalising roots: ``` roots: ['/proj/sub/'] -> 'sub//a.js' roots: ['/proj/other/../shared'] -> blockList for '/shared/dist' ignored ``` Changelog: Internal Test plan: - New unit tests for each, failing without this change. - Real Watchman, watch folder linked from `node_modules`: 3/3 files on a warm start (`main`: 1/3). - `hierarchicalLookup` fuzz, 50k lookups in random symlinked trees: every difference from before #1946 matches a real-path model.
Summary
When a larger amount of paths are being crawled, the Node crawler becomes a hot-path and minor tweaks can have a larger impact on startup time, than previously anticipated (at least for Expo projects)
absoluteToNormalcalls (medium impact)forEachcalls for for-of (formatting only; virtually no impact)Changelog:
Test plan