You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ct-runner runSuite: only should report deselected rows and let capability win over selection (polymorph-test #89 policy) #286
With #260 making runSuite() the only public ct-runner case loop, polymorph-test moved its browser shard worker onto it (polymorph-test#106). That surfaced one policy gap between runSuite and the JS/Rust reference runners it mirrors.
polymorph-test's docs/runner-policy.md §"Selection is not capability" (polymorph-test#89): under only, the unselected remainder of the census is reported as deselected rows (never executed, no provenance) so subset runs still fold and aggregate with the subsetting visible, and capability takes precedence — a tags-excluded case is reported not-applicable even when it also falls outside the selection. The reference runner, js/viewer/harness.mjs's runCases, and crates/component-test-runner all do this (runner.rs::only_reports_the_rest_of_the_census_deselected).
runSuite (run-suite.ts) applies only first with continue and no emit, before the tag-gating block. Observable on polymorph-test's fixture suite with missing: ["hsm"], only: "gen":
runSuite: 2 rows; fixture/hsm/attest vanishes; RunCounts has no deselected/selected (polymorph-test#106 derives selected = passed+failed+skipped+na in the worker for page-runner's empty-selection check)
The comment in run-suite.ts cites harness.mjs's ordering ("only first, then drift, then applicability") but harness.mjs emits a deselected row at that point rather than skipping. Suggested: emit {case, status: "deselected", detail: "only <filter>"} for filtered-out cases (after tag gating decides not-applicable, per the precedence rule), and add deselected to RunCounts. polymorph-test#106's selftest asserts the current runSuite behavior and would flip back when this lands. Not blocking anything.
Triage 2026-09-05: bug, p2 — a conformance gap against the runner policy runSuite claims to mirror, not a decision owed here (the policy is polymorph-test#89's; the reference runners already agree).
Scope check against the tree: runSuite (ct-runner/src/run-suite.ts) applies only with a bare continue ahead of the tag-gating block, exactly as described. The fix is two moves — reorder so tag gating decides not-applicable first, then emit {status: "deselected", detail: "only <filter>"} for the remainder instead of skipping — plus deselected (and optionally selected) on RunCounts.
Versioning note for whoever lands it: rows go out through opts.emit as JSONL, and the RunCounts addition is a new field, so this reads as caret-compatible — no breaking/ct-runner label expected. Consumers that switch exhaustively on row status will see a new value; polymorph-test#106's selftest asserts the current behavior and is expected to flip.
Activity
lannbot commentedon Sep 5, 2026
Triage 2026-09-05: bug, p2 — a conformance gap against the runner policy
runSuiteclaims to mirror, not a decision owed here (the policy is polymorph-test#89's; the reference runners already agree).Scope check against the tree:
runSuite(ct-runner/src/run-suite.ts) appliesonlywith a barecontinueahead of the tag-gating block, exactly as described. The fix is two moves — reorder so tag gating decidesnot-applicablefirst, then emit{status: "deselected", detail: "only <filter>"}for the remainder instead of skipping — plusdeselected(and optionallyselected) onRunCounts.Versioning note for whoever lands it: rows go out through
opts.emitas JSONL, and theRunCountsaddition is a new field, so this reads as caret-compatible — nobreaking/ct-runnerlabel expected. Consumers that switch exhaustively on row status will see a new value; polymorph-test#106's selftest asserts the current behavior and is expected to flip.release-bundle: RunCounts literal gains deselected/selected (#286)
ct-runner: `only` reports deselected rows; capability wins over selec…
Merge pull request #287 from polymorph-components/fix-286-deselected-…