Skip to content

Commit dd3f550

Browse files
ci: split Verify into a fast leg + 2 integration shards behind the required gate; pin the browser; retry setup; nightly pools (#576) (#583)
* test(workbench): select the Playwright browser via AGENT_BUNDLE_PLAYWRIGHT_CHANNEL and keep traces on CI failures Every Workbench browser suite hard-coded `channel: 'chrome'`, so CI tested whatever Google Chrome the runner image shipped that week while Playwright stayed pinned (#576). All 31 sites now read one shared value: - tests/support/browser-launch-options.ts (new leaf) exports `browserLaunchOptions`, selected by AGENT_BUNDLE_PLAYWRIGHT_CHANNEL: unset/empty or `chrome` -> `{ channel: 'chrome' }` (local default, unchanged); `chromium` -> `{}` (Playwright's bundled Chromium for the installed Playwright version; CI sets this); anything else throws at module load. It is a leaf because capture-runtime-playground.mjs cannot load workbench-e2e.ts (`test.extend` needs a running Rstest worker). - tests/support/workbench-e2e.ts re-exports it, uses it in the shared `e2e` fixture, and adds `browserTrace`: `retain-on-failure` when CI=true, else undefined so RSTEST_PLAYWRIGHT_TRACE keeps working (@rstest/playwright reads that variable only when `trace` is undefined). Failed tests land in `.rstest/playwright-traces/<test-name>-<hash>/{trace.zip,trace-summary.json,debug.md}` relative to the Rstest root (the repo root); `.rstest/` is now gitignored. - The 10 `test.extend` forks swap the literal for `browserLaunchOptions` and add `trace: browserTrace`; the 11 raw `chromium.launch` suites (20 calls) and the nightly capture script pass `browserLaunchOptions`. Note for the `chromium` mode: rstest.worker-isolation.ts points XDG_CACHE_HOME at an empty per-worker directory and Playwright derives its browsers directory from that variable on Linux, so workers only find the bundled build when PLAYWRIGHT_BROWSERS_PATH names the real install (~/.cache/ms-playwright). Branded Chrome is a system install and unaffected. * ci: split Verify into a fast leg and two integration shards behind the required gate; pin the browser; retry setup; schedule the nightly pools (#576) Verify used to run the four Rstest pools serially per Node version (13.8 min on a PR, 66% of it the integration pool). It is now a `leg` × `node-version` matrix — `fast` (build, typecheck, lint, unit, route-unit, projection) plus `integration-1|2` (`rstest --shard N/2` over rstest.integration.config.ts) — fanned into the unchanged required context "Verify gate", which now fails on a failed, cancelled, or non-docs-only skipped matrix. The fast leg runs scripts/verify-rstest-shards.mjs, which proves the shards are disjoint and cover every file `rstest list` reports for the pool. The shard step passes `--shard` without a `--` separator (rstest's parser drops everything after one, so the whole pool ran on both shards) and fails unless rstest prints its "Running shard i of N" banner. PRs and merge-queue entries run every leg on Node 24 plus the fast leg on Node 26; main pushes and manual dispatches run every leg on 22.19.0/24/26. .github/actions/setup-workspace replaces every pnpm/setup + install block: pnpm/setup@v2 is retried once (its first network call is the registry fetch that returned 504 on eight runs on 2026-09-04), `pnpm install --frozen-lockfile` up to three times, and browser-driving jobs get `pnpm exec playwright install chromium`. Workbench browser suites select the browser through AGENT_BUNDLE_PLAYWRIGHT_CHANNEL (chromium in CI, branded Chrome locally) and keep Playwright traces on failure under CI, uploaded as artifacts by every browser-driving job. rstest.worker-isolation.ts pins PLAYWRIGHT_BROWSERS_PATH to the registry Playwright resolves before it redirects XDG_CACHE_HOME, so the bundled build is found inside workers. nightly.yml (schedule + dispatch) hosts the packed release matrix, the evidence pool, and MCP conformance as independent jobs; the manual-only mcp-conformance.yml is folded into it. examples-check runs examples/mcp-app's browser-app suite. native-host-smoke runs each pool's script with its own files instead of `pnpm test -- <files>`, which filtered nothing. * ci: fail the shard step on a failed rstest (pipefail) and match the coloured shard banner; launch the pinned browser as channel 'chromium' - The shard step's `run:` used the default `bash -e` shell, so `pnpm … | tee` returned tee's status and a failed rstest fell through to the banner check, which then misreported the failure as a dropped --shard flag. The step now sets `pipefail` explicitly and strips ANSI codes before matching the banner rstest prints in colour under GITHUB_ACTIONS. - `AGENT_BUNDLE_PLAYWRIGHT_CHANNEL=chromium` launches `channel: 'chromium'` (the full Chromium build in the new headless mode branded Chrome uses) instead of the chromium_headless_shell a bare headless launch picks: the shell reports the MCP App sandbox's srcdoc frames differently and both mcp-app-real.e2e frame-URL polls time out under it (first CI run of #583). - examples-real.e2e's error ledger now allows the aborted GET /api/artifacts/epochs/<id>: the MCP page inspects the active epoch under an AbortController it aborts on unmount, and Chromium 151 observes that designed cancellation where Chrome 152 never did. - The local gate's Verify legs run test:route-unit and test:projection too, as the hosted fast leg does (self-review finding). * ci: check the shard banner with one grep — under pipefail, grep -q exiting early gave sed SIGPIPE (141) and read as a missing banner
1 parent b75073b commit dd3f550

41 files changed

Lines changed: 1074 additions & 272 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# Set up workspace: pnpm + Node.js via pnpm/setup@v2, then
2+
# `pnpm install --frozen-lockfile`, each with a bounded retry.
3+
#
4+
# Why this exists (#576, flake table): on 2026-09-04 `pnpm/setup@v2` failed
5+
# eight CI runs before any repository code ran —
6+
# `Could not download https://registry.npmjs.org/@pnpm/exe/11.23.0: 504
7+
# Gateway Timeout` / `fetch failed` — and every one was green on rerun. The
8+
# action downloads pnpm's executable from the npm registry as its very first
9+
# network call and exposes no retry input (its inputs are `version`, `dest`,
10+
# `runtime`, `cache`, `cache-dependency-path`, `working-directory`,
11+
# `package-json-file`, `require-lockfile`, `install`, `token`), so the retry
12+
# lives here instead of in every job:
13+
#
14+
# 1. `pnpm/setup@v2` with `continue-on-error`; on failure, wait 15 s and run
15+
# it once more. The second attempt is not `continue-on-error`, so a
16+
# second failure fails the job. (`steps.<id>.outcome` is the result
17+
# before `continue-on-error` is applied, so it reads `failure` for a
18+
# failed first attempt even though the step's conclusion is `success`.)
19+
# 2. `pnpm install --frozen-lockfile` in a 3-attempt loop (10 s, then 20 s
20+
# back-off), for the same registry 5xx shape during package fetches.
21+
#
22+
# Both retries are bounded: a persistent registry outage still fails the job
23+
# within roughly a minute of extra wall time (15 s + 10 s + 20 s of sleeps)
24+
# instead of looping. A failed first `pnpm/setup` attempt leaves a harmless
25+
# post step behind — it finds no cache state and logs that it is not saving.
26+
#
27+
# Jobs whose pools launch a browser also pass `playwright-browser` (e.g.
28+
# `chromium`): the action then runs `pnpm exec playwright install` for that
29+
# browser, so the build the tests drive is the one pinned by the `playwright`
30+
# version in pnpm-lock.yaml, not whatever Chrome the runner image ships that
31+
# week (#576). No `--with-deps`: the ubuntu-latest image ships Google Chrome,
32+
# whose shared libraries are the ones the bundled Chromium needs, and
33+
# `--with-deps` would add an apt round-trip (one more network dependency per
34+
# job) to install what is already there. If a future image drops Chrome, the
35+
# launch error names the missing library — add `--with-deps` then.
36+
# Playwright's installer retries its own downloads.
37+
#
38+
# Every job that starts from a checkout uses it as
39+
#
40+
# - uses: actions/checkout@v7
41+
# - uses: ./.github/actions/setup-workspace
42+
# with:
43+
# node-version: ${{ matrix.node-version }} # or a literal, e.g. 22.19.0
44+
# playwright-browser: chromium # only for browser-driving jobs
45+
#
46+
# A `./` action path resolves against the runner workspace, so
47+
# `actions/checkout` must run before it.
48+
name: Set up workspace
49+
description: >-
50+
Install pnpm and Node.js with pnpm/setup@v2 (retried once on failure), run
51+
`pnpm install --frozen-lockfile` (up to three attempts), and optionally install a
52+
Playwright browser.
53+
54+
inputs:
55+
node-version:
56+
description: Node.js version for pnpm/setup@v2's `runtime` input, e.g. `22.19.0`, `24`, `26`.
57+
required: true
58+
playwright-browser:
59+
description: >-
60+
Playwright browser to install with `pnpm exec playwright install` (e.g.
61+
`chromium`), at the Playwright version pinned in pnpm-lock.yaml. Empty skips
62+
the install.
63+
required: false
64+
default: ''
65+
66+
runs:
67+
using: composite
68+
steps:
69+
- name: Set up pnpm and Node.js
70+
id: setup
71+
uses: pnpm/setup@v2
72+
continue-on-error: true
73+
with:
74+
cache: true
75+
install: false
76+
runtime: node@${{ inputs.node-version }}
77+
78+
- name: Wait before retrying pnpm/setup
79+
if: steps.setup.outcome == 'failure'
80+
shell: bash
81+
run: |
82+
set -euo pipefail
83+
echo "::warning::pnpm/setup@v2 failed on the first attempt (registry 5xx or network error?); retrying in 15 s."
84+
sleep 15
85+
86+
- name: Set up pnpm and Node.js (retry)
87+
if: steps.setup.outcome == 'failure'
88+
uses: pnpm/setup@v2
89+
with:
90+
cache: true
91+
install: false
92+
runtime: node@${{ inputs.node-version }}
93+
94+
- name: Install dependencies
95+
shell: bash
96+
run: |
97+
set -euo pipefail
98+
delays=(10 20)
99+
for attempt in 1 2 3; do
100+
if pnpm install --frozen-lockfile; then
101+
echo "pnpm install --frozen-lockfile succeeded on attempt ${attempt} of 3."
102+
exit 0
103+
fi
104+
if [ "${attempt}" -eq 3 ]; then
105+
echo "::error::pnpm install --frozen-lockfile failed on all 3 attempts."
106+
exit 1
107+
fi
108+
delay="${delays[attempt - 1]}"
109+
echo "::warning::pnpm install --frozen-lockfile failed (attempt ${attempt} of 3); retrying in ${delay} s."
110+
sleep "${delay}"
111+
done
112+
113+
- name: Install Playwright ${{ inputs.playwright-browser }}
114+
if: inputs.playwright-browser != ''
115+
shell: bash
116+
env:
117+
PLAYWRIGHT_BROWSER: ${{ inputs.playwright-browser }}
118+
run: |
119+
set -euo pipefail
120+
pnpm exec playwright --version
121+
pnpm exec playwright install "$PLAYWRIGHT_BROWSER"
122+
echo "Playwright $(pnpm exec playwright --version | awk '{print $2}') installed its $PLAYWRIGHT_BROWSER build" >> "$GITHUB_STEP_SUMMARY"

‎.github/workflows/changeset.yml‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -39,14 +39,10 @@ jobs:
3939
with:
4040
fetch-depth: 0
4141
persist-credentials: false
42-
- uses: pnpm/setup@v2
42+
- uses: ./.github/actions/setup-workspace
4343
if: github.event_name != 'merge_group'
4444
with:
45-
cache: true
46-
install: false
47-
runtime: node@22.19.0
48-
- run: pnpm install --frozen-lockfile
49-
if: github.event_name != 'merge_group'
45+
node-version: 22.19.0
5046
- name: Require a changeset for publishable package changes
5147
if: github.event_name != 'merge_group'
5248
env:

0 commit comments

Comments
 (0)