Rstest audit: pools, adapter, browser mode, flake risk, CI shape #576

Companion to #566 (toolchain survey) and #572 (Rsbuild audit). #566 §3 and its "Already compliant" list are not restated; PR #570 (test/rstest-hygiene: pool-path guard + orphan-test guard, publish plugins out of pools, restoreMocks/clearMocks/unstubEnvs/unstubGlobals, deletion of the runtime-playground configs and runtime-playground.browser.test.tsx, github-actions reporter confirmed default) and #567 (sqlite WAL retry, mcp-session-service sleeps) are out of scope; the Workbench browser-fixture forks are #572's P2 and are only cross-referenced here.

Preamble

• Commit audited: d88cc10be (origin/main, 2026-09-05) in a detached worktree; no tracked file was edited, nothing committed.
• Installed: @rstest/core, @rstest/browser, @rstest/playwright, @rstest/adapter-rslib 0.11.10; latest on npm 0.11.12 (same for all four). playwright 1.62.1; Node engine floor 22.19.0. (Version table already in Rsbuild audit: MCP Apps compiler path, Workbench, docsite builder #572 — not repeated as a finding.)
• Skills applied: rstest-best-practices, rstest-debugging (+ its references/*), migrate-to-rstest, rslib-best-practices; option semantics verified against rstest.rs/config/test/*.md, guide/basic/cli.md, guide/advanced/playwright.md; adapter behaviour read from @rstest/adapter-rslib@0.11.10/dist/index.js.
• What was run (worktree, one pool at a time, --reporter=verbose, --trace on the first run of each pool): unit ×3 (+ one run each at --pool.maxWorkers=16 and =32), route-unit ×3, projection ×3, integration ×3, evidence ×1, mcp-conformance ×1 (AGENT_BUNDLE_MCP_CONFORMANCE=1), packed ×1 (scripts/run-packed-tests.mjs). CI reference: main run 33932257270 (green, all three Node legs), plus the last 40 ci.yml runs and a 200-run window for flake history.
• Load caveat: the host (96 vCPU) carried 5–6 other agents' builds; 1-min load at pool start was 40–125. Local wall times are medians of 3 and comparable only to each other; the CI column is the authoritative absolute. Per-file numbers come from the Perfetto trace's worker-side load+setupFiles+collect+tests spans, not from host spans (which include queue wait).

Findings

1. Config surface

P1 · rstest.worker-isolation.ts:68 (rstestWorkerRoot → /tmp/ab-rstest-<hash>), scripts/rstest-worker-roots.mjs:41 (removeOwnedRstestWorkerRoots; only caller scripts/local-ci.mjs:54), rstest.integration.setup.ts (no teardown export) · docs config/test/global-setup.md: globalSetup supports teardown, "cleaning up resources after test runs" · Every worker of every pool creates its own /tmp/ab-rstest-* root and nothing on the pnpm test* path removes it; only scripts/local-ci.mjs sweeps, and only roots owned by its private TMPDIR. /tmp on this host held 48,175 ab-rstest-* directories (created since 2026-09-03 04:50 ≈ 44 h; 47,105 at audit start, so the count grows by ~1k/h under normal agent traffic). · Export teardown from rstest.integration.setup.ts and add a globalSetup with the same teardown to the pools that have none (unit, route-unit, projection, packed), removing roots whose owner marker names this invocation; or have rstest.setup.ts register process.on('exit') removal of its own root. · Risk: low — the owner-marker protocol already exists for local-ci.

P2 · packages/agent-bundle/src/rstest/index.ts:116 (setupFiles: [setup, ...], setup = generated route registry only) used by rstest.route-unit.config.ts:15 and rstest.projection.config.ts:18 · repo rule in rstest.worker-isolation.ts header + rstest-best-practices "isolate per-worker temp state" · The route-unit and projection pools never load rstest.setup.ts, so isolateWorkerEnvironment() is not applied: their mkdtemp(tmpdir()) calls and spawned children's TMPDIR/XDG_CACHE_HOME land in the shared /tmp, unlike the other five pools. · Prepend './rstest.setup.ts' to the helper's setupFiles (or in both configs) and extend rstest-worker-isolation.test.ts to cover the helper. · Risk: none — the module is idempotent.

P2 · rstest.config.ts:17-23 (exclude covers only fixtures/mcp/projection/route-unit/templates) vs rstest.unit.config.ts:22-34; package.json:21 (test:watch), :34,37,40-44 (test:examples:browser, test:packed:native, test:host-install, :packed, :session all run rstest --config rstest.config.ts <files>) · docs config/test/exclude.md, pool.md, test-timeout.md · The thin root config has no pool.maxWorkers cap, no testTimeout: 30_000 and no AGENT_BUNDLE_TEST_TIME_SCALE, yet it is the live config for the CI host-install-proofs job (ci.yml:272-278; dev-host-install.test.ts = 104 s of tests) and for test:watch, which discovers the ~101 integration/packed/evidence files the unit pool excludes and runs them at the 5 s default with cores−1 workers. · Point those scripts at rstest.integration.config.ts/rstest.packed.config.ts (positional filters still work), and give test:watch the unit config. · Risk: none.

P2 · rstest.unit.config.ts (no testTimeout → 5000 ms default) + packages/agent-bundle/tests/inspect-state.test.ts:49 (2.3–4.5 s per run; timed out at 5 s in run 3 of 3 locally, load ≈ 50) · rstest-debugging: a timeout within 2× of a test's normal duration is a flake, not a bound · The unit pool hosts runCli journeys that build a TypeScript program in-process; nothing in the pool reads the time scale (tests/support/time-scale.ts is imported only by integration tests), so on CI's 3 workers the margin is whatever the runner gives. · Move the runCli-driving files to the integration pool or set testTimeout: 15_000 in rstest.unit.config.ts with the slow files named in a comment. · Risk: none.

P3 · rstest.integration.config.ts:72, rstest.evidence.config.ts:22, rstest.mcp-conformance.config.ts:11 raise testTimeout (30 s / 180 s) but no config sets hookTimeout (default 10 000, docs config/test/hook-timeout.md); rstest.packed.config.ts:16-23 sets neither, so packed files survive only on per-it timeouts (packed-stdio-projection.test.ts ends }, 300_000)) · rstest-best-practices: set pool-level timeouts as floors, not per-test constants · A new beforeAll without an explicit timeout dies at 10 s while its test body has 30–180 s; a forgotten per-test timeout fails a multi-minute pack/install at 5 s. · Set hookTimeout ≥ testTimeout on the three pools and testTimeout: 120_000 on the packed pool. · Risk: none.

P3 · packages/agent-bundle/tests/mcp-conformance.test.ts:14 (expect(report.failed).toBe(report.expectedFailures.length)), tests/support/mcp-conformance.ts:34,293-296 (regex captures the scenario name, only counts are kept), tests/fixtures/mcp-conformance-expected-failures.yml (23 entries); .github/workflows/mcp-conformance.yml:4 (workflow_dispatch only, 0 dispatches ever) · rstest-best-practices: assert the exact observable value · The run printed MCP conformance 0.1.16 / spec 2025-11-25: 7 passed, 23 failed and passed. A scenario that starts passing plus another that regresses keeps failed === 23; the lane cannot detect a fix or a regression, and has never run in CI. · Collect failed scenario names and expect(names.sort()).toEqual([...expectedFailures].sort()); add a weekly schedule to mcp-conformance.yml (it already uploads artifacts). · Risk: the first exact run will show how far the allowlist has drifted.

P3 · packages/agent-bundle/src/rstest/browser.ts:32,146 (headless: true hard-coded and typed), consumed by examples/mcp-app/rstest.browser-app.config.ts · docs config/test/browser.md (headless defaults to headed locally, headless in CI) · The shipped helper removes the documented local debugging default for every consumer. · headless: process.env.CI === 'true' with an explicit override option. · Risk: public helper behaviour change — needs a changeset and docs update.

2. Test-suite health (measured)

P1 · packages/agent-bundle/tests/runtime-client-surface-proxy.test.ts:1012-1035,1148-1181 wait the real upstreamRequestTimeout = 15_000 (src/dev/runtime-client-surface-proxy.ts:26, module constant, not injectable) · rstest-debugging performance-measurement.md: "tests dominates a few files → narrow, then fix the file"; rstest-best-practices: inject or fake timers, never wait wall-clock · Each case is 15.01 s; the file is 41.3 s of host span against a 43.1 s unit Duration. Runs at 16, 32 and 95 workers land at 46.2 / 43.1 / 45.5 s — the unit pool is bound by this one file, not by worker count, and CI's 1m00s carries the same 30 s. · Make the timeout an option of the proxy factory (default 15 000, asserted once) and pass ~200 ms in the two tests; the repo uses no fake timers (0 rs.useFakeTimers), so option injection is the lower-risk route. · Risk: low. Expected: unit wall drops to the next file (eval-native-mount.test.ts, 27–37 s, whose 15 fully stubbed tests each show a ~1.2 s floor — worth a --trace-narrowed look next).

P1 · packages/agent-bundle/tests/projection/contract-matrix.test.ts:172 (one describe, 21 its, each a full runContractMatrix ≈ 5.5 s, slowest 8.3 s) · rstest-best-practices: small files so the pool can parallelize; a single file bounds the pool's wall · The projection pool forks 13 processes but averages 1.2 concurrent: this file is 119.6 s of the 120.4 s worker span (CI 2m01s) while the other 12 files finish in < 8 s. · Split by concern (lifecycle/progress, state & restart durability, app-surface fixtures, runtime identity, fixture validation) — withStatefulMatrix (:35-63) already isolates per-test state. · Risk: none behaviourally; expected pool wall ≈ 30–40 s locally and on CI.

P2 · rstest.unit.config.ts:16 ("build-free, process-free") vs prebuilt-payload.test.ts:74,163,232-233 (build() from src/api ×3), test-browser-rstest.test.ts:15 (compileMcpApps → real Rsbuild build, 4.75 s) and 11 child-spawning files (classify-docs-only:149, doctor:389,444,2333,2934, entry-shell:324,711, event-ipc:70,102 (sh -c exec sleep 300), install-surface:453,670, install:103,1574-1596 (git), runtime-routes:184, uninstall:496, rsc-runtime/tests/dispatcher:420, plus stub-executable spawns in the two native-*-contract files) · manifest rule rstest.integration-tests.ts:8-11; rstest-best-practices "projects only when configurations genuinely differ" · The pool is configured (isolate: false, 95 workers) for the contract its comment states; two files run the real compiler inside it and eleven spawn children, so a compiler regression or process contention surfaces as a unit-pool flake (see the 5 s timeout above). · Move the two build files to integrationTestFiles; either reword the comment to "no builds, no long-lived processes" or also move dispatcher.test.ts/runtime-routes.test.ts (react-server Flight worker spawns). · Risk: +~10 s integration worker time.

P2 · unit pool collect = 290.6 s of 634 s worker-side (45.8 %); 107/240 files evaluate modules longer than they test (p90 collect 4.27 s; playground-orchestration-service 5.56 s vs 0.30 s tests, effect-boundary 5.23 vs 0.04, dev-log-foreground 5.11 vs 0.08 — all importing the src/dev service graph) · rstest-debugging workflow step 4 → references/dependency-bundling.md (compare default vs bundleDependencies: false/true) · With isolate: false but 95 workers for 240 files, module caches are barely reused (≈ 2.5 files/worker), so the ~4–5 s graph is paid ~100 times; on CI's 3 workers it is the bulk of the 60 s. · Run the skill's fixed comparison on dev-services.test.ts, then the full unit scope: DEBUG=rstest → dist/.rstest-temp/.rsbuild asset sizes, then bundleDependencies variants; check whether externalizing effect/typescript-5/@modelcontextprotocol/* is safe. · Risk: externalizing changes module identity (instanceof across copies) — effect-boundary.test.ts is the canary; keep only a repeatable win.

P2 · packages/agent-bundle/tests/host-adapters.native.test.ts:19 (integration pool; 29 it.skip unless AGENT_BUNDLE_NATIVE_HOST_CONTRACTS=1) and the unit-pool opt-ins host-contract.test.ts:36, native-claude-contract.test.ts:9, native-codex-contract.test.ts:8, eval-codex-home.test.ts:27 · rstest-best-practices: .skip intentionally; projects when configurations genuinely differ · Every PR run forks and collects a file that asserts nothing (~1.5 s), and when the lane is opted in, real claude/codex sessions run inside the non-isolated 95-worker unit pool next to 3 500 other tests. · Give the native lane its own manifest list (as mcpConformanceTestFiles does) with maxWorkers: 1, excluded from unit/integration, it.skipIf(!enabled) at file top. · Risk: none.

P3 · packages/agent-bundle/tests/dev-live-host.test.ts:150 (unconditional it.skip('… [missing evidence …]')), packages/rsc-runtime/tests/state-conformance.test.ts:31 (two it.skip "(durable-only)" registered against the memory driver forever) · rstest-best-practices: .skip/.todo intentionally · Three structural skips make every run report "6 skipped" / "33 skipped", desensitising readers to an accidental one. · it.todo for the wish; filter durable-only cases out for the memory driver and assert the filtered count once. · Risk: none.

P3 · Wall-clock sleeps standing in for ordering (34 sites ≥ 50 ms, inventory in Hygiene below); the large ones: mcp-app-routes.test.ts:936 (1 100 ms past a 1 s fallback timer, unit), packed-release.e2e.test.ts:1052 (1 000), mcp-probe-dev-server.test.ts:168 and projection/script-dispatch.test.ts:623 (400), script-playground-service.test.ts:264 (300), mcp-probe-service.test.ts:868,578,689,733 (250/150/120/120), projection/script-dispatch-abort.test.ts:74 (250), projection/providers.test.ts:295 (150 in a hook to force a race order); rsc-runtime/tests/notices-resource-updated.test.ts:873,933,980 race a 2 s sleep against a close signal then toBe(true) · rstest-best-practices: wait on observable state · Ordering sleeps are load-sensitive by construction (this audit reproduced a 5 s-class timeout under load); Promise.race([signal, sleep]) is await expect(signal).resolves with a worse message. · expect.poll/awaited signals; inject the 1 s fallback timer. · Risk: none.

P3 · Console noise in passing runs (counts below): unit 146 lines, 134 of them from packages/workbench/tests/runtime-app-bridge.test.ts (MCP SDK debug "Sending message", ext-apps warn) plus 10–11 React "outdated JSX transform" warns because rstest.rslib.ts lacks the tools.swc.jsc.transform.react.runtime: 'automatic' that src/rstest/index.ts:127 sets; integration 1 134 lines, mostly Rsbuild start/ready/warn from in-process builds under test and the repeated ext-apps "Critical dependency: require" warning · rstest-best-practices silent: 'passed-only' read against rstest-debugging/references/runtime-output-memory.md ("benefit may be usability rather than stable speed") · Not a time cost, but one of the warnings is a real build warning that silent would bury. · Set the automatic runtime in the shared adapter config, hand the SDK client a no-op logger in runtime-app-bridge, logLevel: 'error' on Rsbuild instances created under NODE_ENV=test (overlaps #572's fixture-fork P2 — fix there), fix the ext-apps entry; leave silent off. · Risk: none.

P3 · packages/agent-bundle/tests/support/durability.ts (zero importers — git grep finds only prose in src/core/durability-test-hook.ts:5) with its NODE_ENV save/finally-restore block copied inline in playground-service.test.ts:68-80, eval-run-store.test.ts:92-104, eval-service.test.ts:32-46; rs.stubEnv used nowhere (80 hand-rolled process.env mutations in 21 files, all restored by hand) · AGENTS.md "extract and rewire in one change" / "delete on sight" · Four copies of an env-restoring helper in a non-isolated pool is the fork-and-drift pattern the hygiene section exists to prevent; #570 turns on unstubEnvs, which only helps once tests use rs.stubEnv. · Rewire the three tests to withGlobalDurabilityValue (or delete the file) and make the helper use rs.stubEnv. · Risk: none.

3. Adapter interplay (@rstest/adapter-rslib) and stale dist

P1 · packages/agent-bundle/package.json:42-95 and packages/rsc-runtime/package.json:34-70 (exports are types + import → ./dist/* only; no development/source condition), tsconfig.base.json:20 (skipLibCheck: true, no paths to runtime src), package.json:13-18 (local pnpm test runs unit → route-unit → projection before test:integration's pnpm build), rstest.integration.setup.ts:11-13 → packages/workbench/tests/helpers/runtime-example-payload.ts:27-30 (ensureRuntimeExamplePayload returns if examples/rsc-agent-runtime/dist/{app,runtime} exist; no freshness check), AGENT_BUNDLE_WORKBENCH_PREBUILT=1/PACKAGE_PREBUILT=1 (package.json:18) skip rebuilds after that · rstest-best-practices: reproduce required resolution explicitly; rslib-best-practices: validate the artifact, not the config · Unit (~8 files), route-unit (7 of 9) and projection (5 of 13) live-import @agent-bundle/runtime from dist; typecheck types against the same dist .d.ts; integration/evidence read packages/{workbench,rsc-runtime}/dist and the example dist through helpers. No pool has a freshness guard — the failure mode that already bit this repo (typecheck green against stale runtime dist) and recurred during #570's gate (ERR_MODULE_NOT_FOUND dist/cli-runtime.js from a concurrent build). · Have pnpm build write a stamp (git rev-parse HEAD + per-package build id) and assert it in a globalSetup shared by unit/route-unit/projection/integration (and in typecheck); make ensureRuntimeExamplePayload compare source mtimes/stamp before short-circuiting; or add an exports development → src condition used only by workspace tests via resolve.conditionNames. · Risk: stamp noise on dirty trees; a development condition must never publish.

P2 · packages/agent-bundle/tests/projection/mcp-lineage.test.ts:3 (import … from 'agent-bundle/test' → package exports → dist) vs siblings contract-matrix.test.ts:25, cli-dispatch.test.ts:4 (../../src/test/*) · rstest-best-practices: one module graph per pool; do not mix built and source copies of the same module · The only dist-entry import in the projection pool; the same pool also loads @agent-bundle/runtime dist, so two copies of the agent-bundle test surface can coexist with divergent identity, silently. · Import ../../src/test like the rest. · Risk: low mechanically; high if left.

P2 · rstest.rslib.ts:8-17 calls withRslibConfig without libId; adapter dist/index.js:53-61,72-96,109 then sets libConfig = {} and merges only the shared rslib.config.ts fields — lib-only format: 'esm' (packages/agent-bundle/rslib.config.ts:51) is never read, output.module is ESM only because 'cjs' !== undefined (index.js:95), shared output.target: 'node' → testEnvironment: 'node' (index.js:109); ignored entirely: dts, source.entry, redirect, output.externals/autoExternal, cleanDistPath, shims/banner · adapter index.d.ts libId option; rstest-best-practices: reuse build config only when compatible · Today's result is right for the published ESM/node lib by accident; adding a second lib entry (CJS or web) without libId changes every pool's test build, and autoExternal does not apply to tests — effect/react are bundled by Rstest's resolver, not externalized as in production. · Give the lib an id and pass libId; document in rstest.rslib.ts what is mapped vs ignored; if externals parity matters set resolve/externals explicitly in modifyLibConfig. · Risk: none today; guards the next lib entry.

P3 · packages/agent-bundle/src/rstest/index.ts:20-36,112-128 and src/rstest/browser.ts:143-162 build their own config (react-server execArgv + conditionNames, agent-bundle/meta alias, SWC automatic JSX) and do not compose withAgentBundleRslibConfig() — so route-unit/projection get no __AGENT_BUNDLE_VERSION__ source.define and none of the adapter's tools/resolve · rstest-best-practices: reproduce aliases/defines the adapter cannot derive, once · Define/resolve drift between the two config families; harmless until code under route-unit reads the define. · Compose extends: withAgentBundleRslibConfig() with the helper overlay, keeping react-server bits in the helper. · Risk: low.

P3 · packages/rsc-runtime/rslib.config.ts:30,42,57,73,93,102 — root lib cleanDistPath: true, secondary libs false (multi-entry stitch) · rslib-best-practices: validate the artifact, not the config · Moving an entry without a clean root wipe leaves orphan files in packages/rsc-runtime/dist that state-packaging.test.ts reads. · Covered by the stamp guard above; until then keep the root clean and add a dist listing assertion. · Risk: low.

4. Flake risk beyond the known two

P1 · packages/workbench/tests/mcp-app-real.e2e.test.ts:432-441 (waitForRequest 120 s timeout in CI) · guide/advanced/playwright.md: wait on the state the action produces; rstest-debugging: a wait longer than the test's own budget is a hang, not a bound · Mechanism (from the client): the Workbench sends the graceful POST …/close only after the frame relay has seen the proxy's ready postMessage (packages/workbench/src/mcp/mcp-app-frame.tsx:209-212); before that, close() takes the forceClose → DELETE path (mcp-app-frame.tsx:229-231, mcp-app-client.ts:1252). The test infers readiness from a count of ui/notifications/initialized HTTP notifications across both previews (:432-435, toBe(2)) — a different channel from the relay's #resourceProvided flag — then clicks "Close MCP session" and awaits waitForRequest('/close', { timeout: 30_000 * timeScale }) (120 s in CI, :436,440). When the relay has not flipped, no /close is ever sent; the 120 s wait is 89 % of the observed 136 s failure. · (1) Filter appRequests by the reopened binding id (from the second /apps POST response) instead of counting to 2; (2) expose relay readiness on the DOM (e.g. data-relay-state="open" on the preview wrapper when #resourceProvided flips) and await expect(outerFrame).toHaveAttribute(...) before clicking; (3) Promise.race the /close wait against a DELETE …/apps/<id> wait so a force-close fails in milliseconds with a precise message; (4) drop the waitForRequest timeouts from 30_000 * timeScale to browserTimeout (click→request latency is milliseconds). · Risk: (2) touches production markup; the others are test-only.

P2 · packages/workbench/tests/support/workbench-e2e.ts:15-20 (e2e = test.extend({ playwright: { launchOptions, contextOptions } }), no trace), .github/workflows/ci.yml (no upload-artifact step anywhere; the only upload in the repo is mcp-conformance.yml:31) · guide/advanced/playwright.md "Trace debugging": trace: 'retain-on-failure' / RSTEST_PLAYWRIGHT_TRACE writes .rstest/playwright-traces/**/trace.zip + debug.md · Every *.e2e.test.ts failure in CI leaves only reporter text; the 120 s hang above had to be diagnosed from source. Overriding the playwright fixture "replaces its value instead of merging", so an env var alone may not reach the shared override. · Add trace: process.env.CI ? 'retain-on-failure' : 'off' to the shared e2e fixture, RSTEST_PLAYWRIGHT_TRACE_OUTPUT_DIR: ${{ runner.temp }}/rstest-traces, and actions/upload-artifact with if: failure(), if-no-files-found: ignore, retention-days: 7. The raw-playwright suites (mcp-page-app-browser.test.ts:10, mcp-app-preview-browser.test.ts:10 launch chromium directly) are outside the fixture and need onTestFailed screenshots. · Risk: none (traces recorded only for failing tests).

P1 · packages/workbench/tests/mcp-app-preview-browser.test.ts:332 (same construct at :354, :476; 8 toEqual sites) · Playwright expect.poll (in-repo pattern mcp-app-frame.test.ts:729); rstest-best-practices: wait on observable state · expect(fixture.bootstrapRequests).toEqual(['/runtime-bootstrap', '/runtime-bootstrap']) reads a Node-side request log synchronously the moment an in-page waitForFunction resolves; the iframe's second bootstrap fetch is a separate network round-trip. This is the only flake that hits the PR (Node 24) leg — twice on main in 36 h (33708273477, 33888345132), green on the next push each time. · await expect.poll(() => fixture.bootstrapRequests.length).toBe(2) then the exact toEqual. · Risk: low; keep the exact array so an over-fetch still fails.

P1 · packages/agent-bundle/tests/dev-host-install.test.ts:317 (new DevCoordinator({...}) with no createWatcher) → assertion :370 (toBe(markerBeforeFailure)) · rstest-debugging references/runtime-output-memory.md (order-dependent state); in-repo pattern dev-coordinator.test.ts (stub watcher) · Without createWatcher the coordinator gets the real ProjectWatcher (src/dev/coordinator.ts:244, 100 ms debounce), so the test's own writeFiles schedule a second rebuild that can land after manager.settled() (:368) and rewrite DEV_INSTALL_MARKER (expected '{"epochId":"6a887b4c-…' to be '{"epochId":"4c52437e-…'). Failed the Host install proofs job on 33910837009 a1, green on attempt 2 of the same SHA. · Inject a stub createWatcher as dev-coordinator.test.ts does, or drain the coordinator before reading the marker. · Risk: low; watcher behaviour is covered by dev-watcher.test.ts.

P2 · packages/workbench/tests/support/packed-outage-ledger.ts:155,216-222,254-256 (stamps in packed-release.e2e.test.ts:222,969,974) · Playwright network events are delivered asynchronously; rstest-best-practices: derive windows from observed events, not Date.now() · The "quiet fence" compares Date.now() taken around clicks with Date.now() taken when Playwright delivers requestfailed, and allow-lists a /api/logs/replay abort only if a 2xx status was already seen; two legitimate cancellations (pre-header navigation abort; stream reconnect after=0→5) failed Release gates on two PRs (33849493532 a1, 33829634707), green on rerun/next push. · Derive windows and the fresh-B session id from the ledger's own wire entries (DELETE/POST), accept status === undefined client aborts for replay, waitForResponse('**/api/logs/replay*') before navigating away from Logs. · Risk: medium — loosen only those two shapes; the fence is a deliberate regression guard.

P2 · .github/workflows/ci.yml:130 (PRs: Node 24 only; main: 22.19.0 / 24 / 26) · rstest-best-practices CI: the gate must run what main runs · In the 200-run window push to main failed 24 / 26 (92 %) while pull_request failed 3 / 100 completed; 21 of the 24 push failures were on legs no PR had exercised (20 Node-26-only, 1 Node 26 + 22.19). A deterministic Node 26 break (script-dispatch.test.ts, fixed in #554) stayed red for 101 consecutive main pushes, and a Node 22.19-only timing flake (mcp-app-real.e2e) lost 11 of 73 main pushes — main was red for ~32 h with zero PR signal about either. merge_group (ci.yml:13) would be the natural place for the full matrix but is dormant on a user-owned repo. · Run the cheap pools (unit + route-unit + projection, ≈ 3.5 min) on Node 22.19 and 26 for PRs as a second fast leg (see the sharding finding), or gate merges on a workflow_dispatch full-matrix run for PRs that touch src/**. · Risk: +7 runner-minutes per PR.

P3 · 33 per-test { retry: 2 } sites in 7 files: generated-route-server.test.ts:58,312,429,522,589,688,741,807,… (12), rsc-runtime/tests/dispatcher.test.ts:464… (16), agent-document.test.ts:243, layout-build.test.ts:151,249, cli-routes-build.test.ts:49, artifact-cli-bin.test.ts:190,293, examples-contract.test.ts:292 · docs config/test/retry.md; rstest-best-practices: retries hide flakes · No config sets retry, but these tests retry twice and Rstest only prints ✓ … (retry xN) when a retry passes — a marker nothing in CI inspects. The 75 dumped CI logs contain zero such markers, so the retries hide nothing today, yet they sit on exactly the warm-runtime/socket-handover tests most likely to flake next. · Drop retry (fail loudly) or add a CI grep that fails on (retry x in test output. · Risk: low; may surface latent flakes in the 60–90 s generated-route-server cases.

P3 · .github/workflows/ci.yml:107,133,201,242,289,324 (pnpm/setup@v2, no retry) · external network · Eight runs on 09-04 (five in a 70-minute burst) failed before any test ran on Could not download https://registry.npmjs.org/@pnpm/exe/11.23.0: 504 / fetch failed, each needing a manual rerun. · Cache the pnpm/Node binaries with actions/cache keyed on the pinned versions, or wrap setup in a retry step. · Risk: low.

5. CI shape

P1 · packages/create-agent-bundle/tests/scaffold-packed-matrix.e2e.test.ts:50,53,98 (expect(...).toContain('"failedTests": 0')), run by pnpm check:release → test:packed:release in the nightly packed-matrix job (.github/workflows/ci.yml:282-307) and by release.yml:91 · rstest-best-practices: assert structured results, not reporter text · The default reporter never prints "failedTests", so the nightly has been red every night it ran in the window (33504006310, 33624486037, 33749241855, 33867802723), pnpm test:evidence (ci.yml:307, no if: always()) has never executed in CI, the mcp-server/cli-tool template configs are verified nowhere, and a release run would fail the same way. · Have the scaffolded project's check emit JSON and assert on the parsed object (or on exit code + the Test Files … passed line); give test:evidence its own step with if: ${{ !cancelled() }}. · Risk: none.

P1 · .github/workflows/ci.yml:158-161, package.json:13,18 · guide/basic/cli.md#sharding-tests (--shard index/count, path-sorted, contiguous slices; blob reporter + merge-reports); #566 §3 flagged sharding without numbers — these are the numbers · One Verify leg serialises unit 60 s → route-unit 13 s → projection 121 s → second pnpm build 19 s → integration 549 s (2 workers) = 769 s of the 827 s leg; integration alone is 66 % of the PR critical path and branch protection is strict: true, so every rebase re-pays it. projects/--project cannot replace this: ProjectConfig omits pool and isolate (docs config/test/projects.md). · Add a leg matrix dimension: fast = build + typecheck + lint + unit + route-unit + projection (≈ 245 s); integration-1/2 = build + pnpm test:integration:run -- --shard N/2 (≈ 362 / 271 s). Predicted Verify wall 6.0 min vs 13.8 (−56 %), PR runner-minutes 14.6 vs 13.8 (+6 %), main ≈ 45 vs 42.6. verify-gate (ci.yml:171-189) stays the required context. Shard membership is contiguous by path: 1/2 = 45 packages/agent-bundle/tests/* files (650 s of work, incl. the 104 s dev-host-install), 2/2 = 12 agent-bundle + 5 rsc-* + 28 workbench (456 s) — 59/41 imbalance, ≈ 45 s worse than an ideal split; 3 shards give 4.2 min wall at +10 % minutes but ≈ 19 concurrent jobs on main. Alternative with no extra minutes: AGENT_BUNDLE_INTEGRATION_MAX_WORKERS=3 burn-in on a dispatch (rstest.integration.config.ts:30) — bound ≥ 10.6 min, contention unknown. · Risk: globalSetup builds the example payload in both shards though only shard 2 uses it; add --reporters=blob + a merge job only if a single summary is wanted.

P2 · .github/workflows/ci.yml:146-152 (reuse preinstalled google-chrome), packages/workbench/tests/support/workbench-e2e.ts:17 and packages/agent-bundle/src/rstest/browser.ts:34,148 (channel: 'chrome') · rstest-best-practices: configure the environment intentionally; guide/advanced/playwright.md launchOptions · Every browser test binds to whatever Chrome the ubuntu-latest image ships that week (152.0.7977.64 in the green run) while Playwright is pinned at 1.62.1 — the browser under test changes with no commit, a red main after an image refresh is not bisectable, and local Chrome ≠ CI Chrome. · Cheapest: pin runs-on: ubuntu-24.04 and write google-chrome --version to $GITHUB_STEP_SUMMARY; deterministic: CI-only playwright install chromium + actions/cache on ~/.cache/ms-playwright keyed by the Playwright version (the only thing that would make that cache meaningful). · Risk: branded Chrome is deliberate (ci.yml:139-145) and ships to consumers via agentBundleBrowserRstest, so a switch must be env-gated or a documented public change.

P2 · examples/mcp-app/package.json:8,10 (check = validate + build + typecheck; test:browser-app = the shipped agentBundleBrowserRstest config, not in check), scripts/run-examples-check.mjs:19-24 (examples-check runs check only) · AGENTS.md "a module with no production importer is not delivered"; examples are user-facing product · With #570 deleting the root browser-mode config, examples/mcp-app is the only @rstest/browser consumer left in the repo and it never runs in CI; the helper (browser.ts:93) is exercised by no pool. · Append && pnpm test:browser-app to the example's check and add the "Ensure branded Chrome" step to examples-check. · Risk: the suite has not run in CI since its config last changed and may have rotted.

P3 · .github/workflows/ci.yml:158,161, package.json:17-18,22 · one build per job · Verify runs pnpm build (20 s) and then test:integration → pnpm build again (19 s): 19 s per PR, 57 s per main push, for nothing; pnpm check (package.json:22) already uses test:integration:run after one build. · Add test:ci (unit → route-unit → projection → test:integration:run) and call it at ci.yml:161 (subsumed by the leg split). · Risk: none.

P3 · .github/workflows/ci.yml:122 (Verify timeout-minutes: 45 vs 13.8–14.9 min measured), :104 (examples-check 25 vs 4.3), :198 (release-gates 30 vs 2.8), :239 (host-install-proofs 30 vs 3.9), :321 (micro-eval 15 vs 0.8); tests/support/time-scale.ts:5-6 relies on "workflow-level timeout-minutes still bounds real hangs" · GitHub Actions timeout-minutes · A hung Chrome or dev server burns 45 min × legs before failing. · Verify 25 (15 per leg after sharding), examples-check 12, release-gates 12, host-install-proofs 15, micro-eval 8; leave packed-matrix at 40 until it has completed once. · Risk: cold caches add ≈ 2 min; the values keep ≥ 1.7× headroom.

P3 · .github/workflows/ci.yml:130 × rstest.integration.config.ts · rstest-best-practices: multiple environments only when they genuinely differ · On main the integration pool runs 3× (Node 22.19 / 24 / 26, 549 s each) and the Chrome half is identical across legs; only the Node-side harness differs. · After sharding, exclude: the browser-dominated shard 2/2 on Node 26 (keep 22.19 as the engines floor): ≈ 4.5 min saved per main push. · Risk: packages/rsc-runtime/tests/*-sqlite-cross-process.test.ts land in shard 2/2 and are Node-version-sensitive — pin them by filename filter if adopted.

P3 · .github/workflows/native-host-smoke.yml:54 (pnpm test -- ${{ matrix.source_tests }}) with package.json:13 · guide/basic/cli.md positional filters apply to one rstest invocation · -- appends the filters to the last command of the && chain only, so the step runs unit, route-unit and projection unfiltered (~3.5 min) and filters only the integration pool; on the codex leg both target files live in the unit pool, so the integration run matches nothing and exits 1 (passWithNoTests false). Never surfaced: 0 dispatches ever. · Call pnpm exec rstest --config rstest.unit.config.ts <files> / --config rstest.integration.config.ts <files> per host. · Risk: none (manual, self-hosted workflow).

6. Vitest/Jest leftovers

None. Every category in migrate-to-rstest came back at zero (patterns and counts in "Already compliant").

Measurements

Per-pool wall time and counts

Lifecycle split from the traces (worker-side seconds; do not add to wall):

CI Verify (Node 24) step split: setup 9 s · install 4 s (store cache hit) · Chrome 2 s · build 20 s · typecheck 9 s · lint 6 s · pnpm test 769 s (unit 60 · route-unit 13 · projection 121 · second build 19 · integration 549). Legs: 891 / 827 / 839 s for Node 22.19 / 24 / 26.

Slowest 15 per pool (worker-side seconds; bare names are packages/agent-bundle/tests/*, wb/ = packages/workbench/tests/)

unit (107/240 files have collect > tests; collect median 0.26 s, p90 4.27 s, max 5.56 s)

Collect-only leaders (all import the src/dev service graph; collect s / tests s): playground-orchestration-service 5.56/0.30, effect-boundary 5.23/0.04, eval-run-store 5.14/0.76, dev-log-foreground 5.11/0.08, eval-codex-home 4.97/0.17, dev-server 4.95/0.58, runtime-mcp-registry 4.91/0.84, dev-events.contract 4.57/0.02, rule-config 4.55/0.16.

route-unit (all 9): lifecycle-replay 14.20 s (tests 12.29), render-route 3.30, workbench-surface-rendered-skill 3.20, event-project 1.47, lineage-notices 1.35, thrown-route-error 1.25, mount-test-state 1.15, published-notices 1.04, load-route-module 1.02 — collect exceeds tests in 4 of 9.

projection (all 13)

integration (collect is a flat 1.5–2.2 s/file under isolate: true; 11/91 files have collect > tests, none in the top 15)

Slowest single cases: route-register-typegen "types every route-aware public surface…" 50.9 s; mcp-tasks.e2e 40.7 s; runtime-playground-hmr.e2e 38.5 s; dev-host-install Claude/Codex/Cursor variants 30–32 s each.

Skip inventory

.only 0, it.todo 0, skipIf 1 (mcp-conformance.test.ts:7, the correct form); every other gate is a cond ? it : it.skip ternary; examples/rsc-agent-runtime/tests/dev-invocation.integration.test.ts:28 skips permanently off Windows. Observed: unit 6, integration 33 + 1 whole file, packed 1.

Console noise in passing runs

Hygiene greps (468 test/support files; repos/, node_modules/, dist/, templates excluded)

• test.concurrent: 2 sites, both scaffold-packed-matrix.e2e.test.ts:33,80 (share one memoized fixture promise and one npm cache — documented :28-32).
• Snapshots: 0 toMatchSnapshot / toMatchInlineSnapshot / toMatchFileSnapshot, 0 __snapshots__, 0 .snap.
• os.tmpdir(): 171 files / 684 sites, virtually all mkdtemp(join(tmpdir(), …)) — compliant where rstest.setup.ts runs (TMPDIR rewritten per worker before modules evaluate); not in route-unit/projection (finding 1-P2). No hard-coded /tmp writes.
• process.cwd(): 100+ sites; ~30 integration/packed/support files set workspaceRoot = process.cwd() (cli.test.ts:16, overview.e2e.test.ts:25, workbench-e2e.ts:12, …) — works only because every pool launches from the repo root.
• process.env mutation: 21 files / 80 sites, all save-then-finally-restore by hand; rs.stubEnv 0. Unit (shared worker): playground-service.test.ts:72, eval-run-store.test.ts:97, rsc-runtime/tests/plugin-root.test.ts:66. process.chdir: only the excluded mcp-session-service.test.ts:399,415.
• Fixed ports: none. 137 listen(0)/port: 0 sites; numeric literals are stub results/fixture URLs only (dev-workbench.test.ts:1860, cli.test.ts:846,882,959, CSP/Origin fixtures).
• Wall-clock sleeps ≥ 50 ms (excluding mcp-session-service, state-sqlite-cross-process): 34 sites; the 12 largest are in finding 2-P3, the rest are 50–100 ms ordering sleeps in examples/rsc-agent-runtime/tests/{dev-provider.integration,environment-checkpoint-store}.test.ts, artifact-validator:1639, overview.e2e:1379, hook-playground-service:653,675, mcp-entry:204, eval-service:734, generated-route-server:612, host-mcp-proxy:154, projection/{mcp-in-memory:721,script-dispatch:221}, rsc-runtime/tests/effect-boundary:142, packed-release.e2e:273; two 50 ms poll intervals in support/{mcp-conformance:109,packed-release-harness:39} are fine.
• Fake timers: rs.useFakeTimers 0 (so no restore mismatch possible). One hand-rolled interception: mcp-app-routes.test.ts:984-994 replaces globalThis.setTimeout and restores in finally — what rs.useFakeTimers({ toFake: ['setTimeout'] }) + rs.getTimerCount() would express without patching a global in a non-isolated worker.
• Mocks: rs.mock/rs.doMock 0 (no hoisting pitfalls possible); rs.spyOn 0; rs.fn 3 (hooks.test.ts:252,351,477); rs.stubGlobal/rs.stubEnv 0. Never-restored Object.defineProperty(globalThis, '__AGENT_BUNDLE_VERSION__') in cli.test.ts:49,802,903, eval-cli:21, install:1456,1777, prepack:134, route-graph:791, uninstall:1252 — test: guard the Rstest pool lists, strip publish-only plugins from pools, restore state between tests (#566 §3) #570 shows the stub is dead (source.define replaces the identifier at compile time); delete them all.

Flake table

Windows: the last 40 ci.yml runs (2026-09-04 22:33Z → 09-05 00:35Z) plus a 200-run window (09-04 08:14Z → 23:58Z) and per-job logs back to 09-03 02:36Z. Run links are actions/runs/<id>; a1/a2 = attempt. Rstest console interception (stderr | file > test blocks such as "Request timed out" from runtime-app-bridge.test.ts or "Syntax Error: Unexpected token" from build-diagnostic negative tests) was excluded — those are passing tests' output, not failures. retry is set in no config; 33 per-test { retry: 2 } sites exist (finding above) and the (retry xN) marker appears 0 times in 75 dumped logs.

Flakes (failed, then passed on rerun or the next run of the same branch/SHA)

Already resolved before the audit (listed so they are not re-chased): release-audit.test.ts — npm audit signatures → EATTESTATIONVERIFY/120 s timeout on five main Release-gates runs (33771541855 … 33823120338); external network; gate removed by #487 — if reinstated, use a vendored attestation fixture offline. packed-release.e2e "invalid edit retains stale epoch B" (33779266923) and mcp-page-app-browser.test.ts:507 (33735363205) — watcher-debounce / request-log timing, both fixed by #435. Excluded per brief (#567): state-sqlite-cross-process.test.ts (33794981569 main 22.19; PR attempt-1s 33910837009, 33926598550) and mcp-session-service.test.ts (33841695068, 33915324117 main 24; PR attempt-1s 33885396000, 33913486168). runtime-client-surface-proxy.test.ts, public-api.test.ts and hook-handler-contract.test.ts, suspected earlier, are not flakes (the first only ever passes; the other two were branch breaks, below).

Deterministic failures, not flakes (for dedupe)

• projection/script-dispatch.test.ts on Node 26 — 101 of 101 main pushes red 09-03 15:15Z → 09-04 23:26Z (33927566698 is the last); fixed by fix: Node 26 transform flag; operator .env precedence + import order; MCP negotiation test; provider-view cross-process coverage (integration) #554 (d88cc10be), 4 green pushes since.
• scaffold-packed-matrix.e2e.test.ts — every nightly (33867802723 and the 3 before); asserts "failedTests": 0, which the default reporter never prints (finding 5-P1).
• Branch breaks, not flakes: public-api.test.ts:224 (33857773495, effect types in the public declaration graph); hook-handler-contract.test.ts (33906580093, Cannot find package 'agent-bundle' from a /tmp wrapper importing launchEnvLayerSpecifier, fixed on the branch); four host-install files with AB5000 … outside output root (33846398648, the PR's own build change); packed-consumer:369/public-api-packed:138 EEXIST … node_modules/@types (33829634707, fixed by feat(api): stage temporary artifacts and Codex schema output in Effect scoped temp directories (FileSystem phase 1, module 2) #508); native-playground-service.test.ts snapshot (33715395180); three pnpm typecheck reds after back-to-back merges (33838306750 …).

Infrastructure

Rates

• 200-run window: push to main 24 / 26 failed (92 %); pull_request 3 / 100 completed (3 %); of the 24 push failures 21 were only on legs PRs never run (20 Node-26-only, 1 Node 26 + 22.19); 3 also failed Node 24 (one infra, mcp-app-preview-browser, mcp-session-service). Counting attempt 1, 6 more PR runs failed then passed on rerun → PR first-attempt failure ≈ 9 / 100 (2 sqlite, 2 mcp-session-service, 1 mcp-app-real, 1 dev-host-install; 3 involved pnpm/setup).
• Last 40: push 1 / 4 failed (the last pre-fix: Node 26 transform flag; operator .env precedence + import order; MCP negotiation test; provider-view cross-process coverage (integration) #554 push, Node 26 only); PR 0 / 10 completed, 1 needed a rerun (sqlite).
• Legs PRs never run: Node 26 red on 101 consecutive main pushes (deterministic); Node 22.19 lost 11 of 73 main pushes to mcp-app-real alone. Main was red for ~32 h with zero PR signal about either.

Already compliant (do not re-audit)

• Config options at their documented defaults, correctly: retry 0, update unset (no -u anywhere), passWithNoTests false, globals false (every test imports @rstest/core/effect-rstest), bail 0, snapshotFormat default, slowTestThreshold 300, maxConcurrency 5, printConsoleTrace false, detectAsyncLeaks off.
• Pool shape: pool.type: 'forks' explicit where execArgv is needed (src/rstest/index.ts:114); testEnvironment: 'node' everywhere (browser proofs run under @rstest/playwright in Node, not jsdom); rstest.setup.browser.ts intentionally empty; AGENT_BUNDLE_TEST_TIME_SCALE 4 in CI instead of retries (tests/support/time-scale.ts:14-16); no config sets reporters — 0.11.10 already defaults to ['default', 'github-actions'] under GITHUB_ACTIONS.
• Why projects is not used for the CI legs is legitimate: ProjectConfig omits pool/isolate/bail/coverage/reporters; route-unit needs pool.execArgv and unit vs integration differ on isolate, so separate rstest --config processes are required. test: guard the Rstest pool lists, strip publish-only plugins from pools, restore state between tests (#566 §3) #570's follow-up plan (directory-based pools + one projects config for the rest) is the right shape.
• Adapter: source.define.__AGENT_BUNDLE_VERSION__ flows into adapter pools (rslib.config.ts:92-94 → adapter index.js:86); single-lib package so no redirect rewrite applies; agentBundleRstest correctly owns the react-server condition the adapter cannot derive; packed path always builds first (scripts/run-packed-tests.mjs:38-39); CI order build → typecheck → test (ci.yml:158-161).
• Test discovery guard: test: guard the Rstest pool lists, strip publish-only plugins from pools, restore state between tests (#566 §3) #570's rstest-pool-lists.test.ts verifies every manifest path exists, every glob matches, no duplicates, and no *.{test,spec}.* file (any extension, including hidden paths) is outside a live pool — this covers the *.test.ts-only include glob.
• Vitest/Jest leftovers (all zero): vi.*/jest.* calls, vitest/@jest/globals/@playwright/test imports, foreign config files, vitest|jest in any package.json field or the lockfile, __mocks__/, globals: true, Jest config keys, expect.extend, done-callbacks, legacyFakeTimers, .toBeCalled(, expect.assertions(. Docs and create-agent-bundle templates instruct rstest only; the sole vitest string is docs/effect-conventions.md:720 banning @effect/vitest.
• Isolation and ports: per-worker TMPDIR rewrite happens before any test module evaluates (rstest.setup.ts → isolateWorkerEnvironment()); no fixed ports; no process.chdir outside the excluded file; all process.env mutations restored in finally.
• CI: pnpm store cached and hit (install 4 s); not caching dist is right (20 s build, stale-dist risk); Chrome reuse with fallback (ci.yml:146-152); host CLI cache keyed by pins (ci.yml:255-263); no watch mode; no continue-on-error; fail-fast: false right for the main matrix; concurrency cancels superseded PR runs only; verify-gate matches the required contexts; fixture child pools pass --reporter=default (rstest-meta-consumer.test.ts:28); mcp-conformance.yml:29-36 already uploads artifacts if: always() — the pattern Verify should copy.

Not assessed / caveats

• Local timings were taken at load 40–125 on a shared 96-core host; the unit-pool 5 s timeout and the run-to-run spread (unit 41.6–54.3 s) are partly that load. CI numbers are from one green main run.
• scripts/run-packed-tests.mjs and the nightly template matrix (test:packed:release) were not timed separately; test:evidence and mcp-conformance have no CI history to compare against.
• The @rstest/browser screenshotFailures/isolation options requested in the brief do not exist in the installed 0.11.10 BrowserModeConfig; with test: guard the Rstest pool lists, strip publish-only plugins from pools, restore state between tests (#566 §3) #570 removing the root browser-mode config, the only remaining Browser Mode consumer is examples/mcp-app (finding 5). Whether 0.11.12 adds browser options worth adopting was not checked.
• Not measured: adapter-pool externals vs Rslib autoExternal (needs DEBUG=rstest), the bundleDependencies comparison, 3–4-worker contention on a 4-vCPU runner (needs a dispatch burn-in), whether RSTEST_PLAYWRIGHT_TRACE reaches a test.extend override that omits trace, and the account's concurrent-job cap (assumed 20).
• rsc-runtime secondary-lib cleanDistPath: false orphan check was not run (no build outside the worktree).
• Cross-references: Workbench browser-fixture forks (8 files re-owning createRsbuild + pluginReact) are Rsbuild audit: MCP Apps compiler path, Workbench, docsite builder #572's P2; the runtime-playground orphan test/configs, restoreMocks-family flags, publish plugins in pools and the github-actions reporter are test: guard the Rstest pool lists, strip publish-only plugins from pools, restore state between tests (#566 §3) #570; state-sqlite-cross-process and mcp-session-service sleeps are fix(runtime): retry the sqlite WAL switch under the busy timeout; await MCP session admission in tests #567.