Skip to content

ct-runner runSuite: only should report deselected rows and let capability win over selection (polymorph-test #89 policy) #286

Closed

Description

@lannbot

With #260 making runSuite() the only public ct-runner case loop, polymorph-test moved its browser shard worker onto it (polymorph-test#106). That surfaced one policy gap between runSuite and the JS/Rust reference runners it mirrors.

polymorph-test's docs/runner-policy.md §"Selection is not capability" (polymorph-test#89): under only, the unselected remainder of the census is reported as deselected rows (never executed, no provenance) so subset runs still fold and aggregate with the subsetting visible, and capability takes precedence — a tags-excluded case is reported not-applicable even when it also falls outside the selection. The reference runner, js/viewer/harness.mjs's runCases, and crates/component-test-runner all do this (runner.rs::only_reports_the_rest_of_the_census_deselected).

runSuite (run-suite.ts) applies only first with continue and no emit, before the tag-gating block. Observable on polymorph-test's fixture suite with missing: ["hsm"], only: "gen":

  • reference runners: 8 rows — 2 pass, 1 not-applicable (fixture/hsm/attest), 5 deselected; counts na: 1, deselected: 5, selected: 2
  • runSuite: 2 rows; fixture/hsm/attest vanishes; RunCounts has no deselected/selected (polymorph-test#106 derives selected = passed+failed+skipped+na in the worker for page-runner's empty-selection check)

The comment in run-suite.ts cites harness.mjs's ordering ("only first, then drift, then applicability") but harness.mjs emits a deselected row at that point rather than skipping. Suggested: emit {case, status: "deselected", detail: "only <filter>"} for filtered-out cases (after tag gating decides not-applicable, per the precedence rule), and add deselected to RunCounts. polymorph-test#106's selftest asserts the current runSuite behavior and would flip back when this lands. Not blocking anything.

Activity

  1. lannbot commented on Sep 5, 2026

    @lannbot
    CollaboratorAuthor

    Triage 2026-09-05: bug, p2 — a conformance gap against the runner policy runSuite claims to mirror, not a decision owed here (the policy is polymorph-test#89's; the reference runners already agree).

    Scope check against the tree: runSuite (ct-runner/src/run-suite.ts) applies only with a bare continue ahead of the tag-gating block, exactly as described. The fix is two moves — reorder so tag gating decides not-applicable first, then emit {status: "deselected", detail: "only <filter>"} for the remainder instead of skipping — plus deselected (and optionally selected) on RunCounts.

    Versioning note for whoever lands it: rows go out through opts.emit as JSONL, and the RunCounts addition is a new field, so this reads as caret-compatible — no breaking/ct-runner label expected. Consumers that switch exhaustively on row status will see a new value; polymorph-test#106's selftest asserts the current behavior and is expected to flip.

  2. added
    p2Minor bugs; desirable lower-priority features
    on Sep 5, 2026
  3. added 2 commits that reference this issue on Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingp2Minor bugs; desirable lower-priority features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions