Generate an Agent Bundle framework test harness on top of Rstest

Summary

Agent Bundle already uses Rstest extensively inside its own repository. The
missing product feature is a generated, consumer-facing harness that understands
compiled routes, Agent Document rendering, MCP/CLI/script projections, browser
Apps, host artifacts, and the framework's dev/build lifecycle.

Make framework testing a first-class compiler output rather than requiring every
plugin author to reconstruct internal entrypoints or protocol clients.

The current examples show the gap at two levels:

Audiobook Curator runs bare rstest tests, so it receives no generated
framework-aware project configuration or route helpers.
the full RSC runtime example carries a manual rstest.config.ts with its own
include glob, serialized worker pool, Node environment, time scaling, and
expanded timeout for real Rsbuild compiles and child processes.

Those are valid repository tests, but plugin authors should not have to learn or
copy that setup merely to test an Agent Bundle project.

Existing capabilities to preserve

The repository already has:

Rstest unit and integration configurations;
Rslib-aware compilation support;
Rstest browser and React integration;
Playwright-backed browser tests;
real MCP client/server wire tests;
packed-consumer and deleted-source artifact tests;
Workbench and MCP App browser tests;
native host smoke gates.

This issue packages and generates the useful framework boundaries for consumer
projects. It does not replace the repository's test infrastructure.

Goals

Let consumers test routes without importing generated implementation details.
Test the same compiled route graph used by build and development.
Make final documents and streaming render events directly assertable.
Provide real MCP, CLI, script, browser-App, and packed-artifact harnesses.
Make target capability and host artifact regressions visible before release.
Keep unit tests fast while preserving explicit end-to-end proof levels.

Proposed public surface

Agent Bundle supplies an Rstest integration and generated test module:

// rstest.config.ts
import { defineConfig } from '@rstest/core';
import { agentBundleRstest } from 'agent-bundle/rstest';

export default defineConfig({
  plugins: [agentBundleRstest()],
});

V1 ships agent-bundle/rstest and agent-bundle/test as explicit subpath
exports of the existing package with Rstest as an optional peer. A separate
package is unnecessary until dependency or release-boundary evidence requires
one.

Tests import stable framework helpers rather than source registries:

import {
  renderRoute,
  invokeMcpTool,
  invokeCli,
  runScript,
} from 'agent-bundle/test';

it('renders a bounded inventory result across MCP and CLI', async () => {
  const route = await renderRoute('mcp:curator:inventory', {
    source: fixtureRoot,
  });

  expect(route.final).toMatchAgentDocumentSnapshot();
  expect(route.events).toMatchAgentRenderSequence([
    'shell',
    'progress',
    'replace',
    'complete',
  ]);

  const mcp = await invokeMcpTool('curator', 'inventory', {
    source: fixtureRoot,
  });
  expect(mcp.structuredContent).toEqual(route.final.value);

  const cli = await invokeCli(['inventory', fixtureRoot, '--output', 'json']);
  expect(JSON.parse(cli.stdout)).toEqual(route.final.value);
});

Test levels

1. Route unit harness

loads the compiled route contract;
injects test capabilities, filesystem roots, environment, clock, and
cancellation;
renders through the real Agent renderer;
captures shell/progress/patch/error/complete events;
never opens a transport or builds host artifacts.

2. Projection contract harness

offers an official SDK in-memory client/server mode for fast protocol
contract proof and labels it in-memory;
separately invokes the packed generated MCP server through a subprocess/stdio
transport for process, stdout, and lifecycle proof;
invokes the generated CLI dispatcher with captured stdout/stderr;
executes plain and rendered scripts;
compares final meaning across projections without requiring byte-identical
presentation.

3. Browser harness

compiles each MCP App or Workbench surface through its exact production-owning
Agent Bundle/Rsbuild compiler function;
mounts them through Rstest browser/React facilities;
tests accessibility, bridge contracts, consent, resource binding, and render
stream presentation;
uses Playwright only where a real browser process or navigation boundary is
required.

4. Artifact harness

builds the package through the real Agent Bundle compiler;
installs or copies the packed output into a clean consumer;
reports packed-tarball install and copied-artifact proof as distinct modes;
deletes project source where applicable;
validates manifests and provenance;
exercises CLI, MCP, scripts, Apps, and selected native host layouts.

These levels must remain visibly separate. A route-unit pass is not an artifact
or host-integration receipt.

Compiler coupling

The compiler generates a test manifest alongside the route manifest:

interface AgentBundleTestManifest {
  routes: Readonly<Record<string, TestableRouteDescriptor>>;
  projections: Readonly<Record<string, ProjectionDescriptor>>;
  apps: Readonly<Record<string, BrowserAppDescriptor>>;
  targets: readonly string[];
  artifactExpectations: ArtifactExpectationSet;
}

The Rstest integration consumes this manifest to configure transforms, aliases,
test environments, generated types, and fixture lifecycles. Consumer tests do
not manually reproduce Rslib or Rsbuild configuration.

Generated fixtures and capabilities

Framework fixtures should provide:

isolated project and output roots;
strict environment allowlists;
fake clocks only when the route explicitly receives a clock capability;
bounded child-process fixtures;
real AbortSignal cancellation;
deterministic MCP client/server pairs;
render-event collection with size/rate enforcement;
explicit target capability fixtures for text, image, audio, resources, and
progress without pretending unsupported content is accepted;
explicit browser origin and bridge setup;
disposable packed-consumer directories.
request-context provider fixtures with explicit identity provenance;
production-driver conformance fixtures for optional durable state;
native event projection fixtures that never claim to be host dispatch;
notice delivery and receipt fixtures with supported/unavailable routes;

No fake transport or in-memory artifact may be presented as end-to-end proof.
Each helper states the proof level it supplies.

Matchers

Proposed matchers:

expect(document).toMatchAgentDocumentSnapshot();
expect(events).toMatchAgentRenderSequence([...]);
expect(result).toSatisfyMcpResultContract();
expect(artifact).toContainDeclaredRoute('mcp:curator:inventory');
expect(projections).toHaveEquivalentFinalValue();

Snapshots must preserve semantically important structure without pinning noisy
generated identifiers, timestamps, or bundler implementation details.

Failure diagnostics

Failures should report:

route identity and source;
config and generated-manifest provenance;
active projection and target;
last render event and boundary;
captured stdout/stderr/logs with byte bounds;
cancellation/timeout state;
packed artifact path when relevant;
a direct command for rerunning the narrow failing layer.

Migration and delivery

Extract stable helpers from existing internal Rstest suites without weakening
their assertions.
Generate a test manifest from the new route compiler.
Ship route-render and event-stream harnesses first.
Add MCP/CLI/script projection helpers.
Add Rsbuild browser-App support.
Add packed/deleted-source consumer helpers.
Convert public examples to the consumer harness.
Update create-agent-bundle templates with one route test and one compiled
projection test.

Acceptance criteria

A scaffolded project runs useful framework-aware Rstest tests without copying
Agent Bundle's internal build setup.
Route tests use the real compiler contract and real Agent renderer.
Fast MCP tests cross the official SDK in-memory transport, and process-level
MCP tests separately cross the packed subprocess/stdio boundary.
Browser tests compile through the same Rsbuild profile used by production.
Artifact tests prove behavior after source deletion.
Audio and other non-text content is projected only when the target capability
fixture advertises support; fallback and fail-closed paths are both tested.
Failure output identifies route, projection, target, and provenance.
The harness never labels a fixture or in-memory substitute as packed,
transport, browser, or host proof.

Design references

Stack position

Core route/render harness depends on: #93 routes, #95 context/providers, and #96 RSC renderer.
State, event, notice, CLI/script, host-component, and lifecycle issues add
their own conformance modules to this stable harness contract.
#104 worktree-proximity depends on those modules and is the first
cross-process composition proof.

Full meta-framework stack

Foundations: #93 routes - #94 identity - #95 context - #96 renderer
Runtime: #97 events - #98 state - #99 notices
Host: #100 components - #101 lifecycle
Surfaces and proof: #102 CLI/scripts - #103 Rstest - #104 proximity example - #105 Workbench

feat(test): ship the route-unit consumer test harness (#103 stage 1)

feat(test): ship the route-unit consumer test harness (#103 stage 1) (#…

feat(test): add the projection-contract proof levels to the consumer …

Wave 5 — #103 stage 3 starting (browser harness), scope per #107 rev 2.

#107 rev 2's verdict on this issue caps stage 3 as "browser harness" — this issue's Test level 3: compile each MCP App through its exact production-owning compiler function, mount through Rstest browser facilities, test accessibility, bridge contracts, consent, and resource binding; Playwright only where a real browser boundary is required.

Building now (branch wave5/103-s3):

Manifest app descriptors — AgentBundleTestManifest.apps from the same compiler pass (the slot the stage-2 manifest doc explicitly reserved). Additive to src/test/manifest.ts; will rebase over the in-flight manifest/cli fixes lane.
agentBundleBrowserRstest() (agent-bundle/rstest) — the browser-pool configuration: compiles the project's apps once per pool run through compileMcpApps (the production function already builds every app in one Rsbuild instance — no per-test or per-app builds) and hands workers a generated compiled-app registry, mirroring the route-unit setup pattern.
agent-bundle/test/browser (new browser-safe subpath) — mountBrowserApp mounts the compiled self-contained HTML in a sandboxed iframe and wires postMessage to the product's own bridge (createMcpAppBridge) with test-supplied binding operations, consent authority fixtures (approve/deny/unavailable), and captured host traffic. No test-only bridge reimplementation.
New proof level browser-app, honestly labeled: compiled production App HTML in a real browser page over the product bridge — NOT host embedding, NOT packed-artifact, NOT Workbench evidence.
examples/mcp-app adopts it: a browser-app pool covering mount, bridge handshake, tool-result presentation, consent approve/deny, resource binding, and a fail-closed path.

Deferred, not blocked-idle:

Workbench-surface browser helpers — depend on Add an interactive Agent Workbench with local host discovery and lifecycle replay #105 stage 1 (compiler-manifest navigation), in flight on another lane; will coordinate there rather than duplicate.
Render-stream presentation in the browser — that stage of Add an interactive Agent Workbench with local host discovery and lifecycle replay #105 (Agent Document stage) owns the surface; harness hooks follow it.
Packed/deleted-source App proof — Generate an Agent Bundle framework test harness on top of Rstest #103 stage 4.

Cost rules: no packed-pool changes at all; the browser pool adds exactly one production Rsbuild compile per run, shared by every test in the pool.

Concrete lifecycle/version-skew acceptance matrix from cargo-conductor's live regressions is now specified in #218. In particular, #103's projection harness needs open-connection queued→running→first-output→terminal replay and serialized response round-trips with additive fields; unit invocation alone would not have caught the failures.

feat(test): browser-app proof level for the consumer harness (#103 st…

Wave 5 — #103 stage 3 landed: #212, squash 34d265083.

The browser harness, per #107 rev 2's cap and this issue's Test level 3. New proof level, stamped into provenance and printed in every failure:

browser-app (MCP App HTML compiled through the production Rsbuild profile, mounted in a real browser page over the product bridge; NOT host embedding, packed-artifact, or Workbench evidence)

Delivered:

agentBundleBrowserRstest() (agent-bundle/rstest): the browser-pool configuration. One compiler pass for the manifest, then exactly one compileMcpApps invocation per pool run — the production function compiles every app in one Rsbuild instance with its self-contained-HTML invariants asserted post-build. Compiled HTML is embedded into a generated registry (bounded by the bridge's own MAX_APP_HTML_BYTES); browser workers never touch a filesystem or a second compiler.
agent-bundle/test/browser (new browser-safe subpath, zero node: imports in the emitted bundle): mountBrowserApp mounts the compiled HTML in a sandboxed iframe wired over postMessage to the product's own createMcpAppBridge — no test-only bridge fork. Tests supply binding operations, host callbacks, and consent decisions (manual, scripted, or per-challenge); the handle exposes the app document, both directions of bridge traffic, captured host callbacks, pending consent challenges, and the publish surface.
The test manifest carries collision-checked MCP App descriptors from the same compiler pass (the slot stage 2 reserved), with planCompiledMcpApps's identity rules. compileMcpApps gains a per-app target map beside the single-target form; existing callers unchanged.
examples/mcp-app adopts the pool: four acceptance tests against the real compiled status panel (which gained genuine resource-read and consent-gated refresh actions through the official @modelcontextprotocol/ext-apps SDK) — mount/handshake/accessible presentation, resource-binding round trip, consent held-then-approved-once vs denied-fail-closed (-32001, binding never called), and unavailable-operation fail-closed (-32000).

Cost rules held: packed pool untouched (no new build/pack/spawn); the browser pool adds one production compile per run shared by all tests; the harness unit test compiles the fixture app once, no pack.

Deferred, with owners: Workbench-surface browser helpers and render-stream presentation belong to #105's stages (stage 1 landed mid-flight as #211; rebased over it — no duplication). Packed/deleted-source App proof is #103 stage 4.

Coordination outcomes: rebased twice over the concurrent lanes (#206 manifest/cli fixes — one test-file conflict, resolved; #192 effect-rstest; #211 and six more). A drift pin was added: the browser registry's label copy must equal proofLevelLabel('browser-app') or the unit suite goes red.

⚠️ Two pre-existing environment notes, neither from this lane: (1) scaffold-packed-matrix.e2e.test.ts fails under npm 12.0.2 (npm pack --json now returns an object; tests/support/shared-pack.ts:84 expects the array form) — reproduced identically on clean main at c158be5f5, passes under npm 11. (2) script-playground-service.test.ts ("does not settle cancellation until its final process-tree cleanup attempt completes") flaked once on CI Node 26 and passed on re-run — timing-sensitive race for whoever owns the script playground.

feat(test): packed deleted-source proof level for the consumer harness (

Wave 6 — #103 stage 4 landed: #225, squash ae9b2fb0a. This completes the staged plan #107 rev 2 capped for this issue (stages 1–4).

The artifact harness — the packed/deleted-source proof, including the packed App proof stage 3 explicitly deferred here. New proof level, stamped into provenance and printed in every failure:

packed-deleted-source (packed tarball installed into a clean consumer, artifact built, project source removed and verified absent, generated stdio entry spawned as a real process; self-contained-artifact evidence)

Delivered:

removeProjectSource (agent-bundle/test): removes the consumer's conventional source inputs (src + root agent-bundle.config.* + caller-named extras) and returns a frozen project-relative receipt. Fail-closed everywhere: missing root, vacuous deletion (absence observed without a deletion is not evidence), path escaping the root, or any path surviving removal all refuse with the new deleted-source-unverified code.
openPackedMcpServer({ deletedSource }): verifies the receipt immediately before spawn and only then upgrades provenance to packed-deleted-source + sourceRemoved. Per a P1 caught by Codex review on the PR and fixed before merge, the receipt is also bound to the launched session: the entry must be a descendant of the receipt's project root and any explicit cwd must stay inside it — a genuine receipt from project A can no longer mint deleted-source evidence for project B's entry. Without the option, provenance stays byte-for-byte packed-stdio.
The single packed journey now runs deleted-source and proves the embedded App. One shared tarball set, one install, one build, one verified source removal, one spawned server; every prior route assertion plus: the packed server lists and serves ui://harness/panel at text/html;profile=mcp-app, and the returned HTML carries the fixture marker, an inline <script, and no external script/link references — the build's self-containment invariant re-checked on the wire with the source tree gone. The App HTML comes from the entry's inline agent-bundle/mcp-apps registry; the deleted source can play no part.
Unit coverage: receipt happy path, vacuous-deletion refusal, stale-receipt rejection before spawn, cross-project entry and foreign-cwd binding rejections, and the exact-label drift pin (same pattern as browser-app).

Cost rules held (#107): no new build, no new pack, no new spawn — the one journey gained a source rm and per-session App reads. Route-unit/projection/browser pools untouched.

Environment note (1) from the stage-3 landing is resolved: npm pack --json returns a package-keyed object on npm 12 (array on ≤11); run-packed-tests.mjs, tests/support/shared-pack.ts, and audit-packed-release.mjs now accept both forms and fail loudly on anything else. The packed pool runs under npm 12.0.2 again.

Staged plan status: stage 1 (#151: plugin/manifest/renderRoute/matchers), stage 2 (#190: mcp-in-memory/cli-dispatch/packed-stdio), stage 3 (#212: browser-app), stage 4 (#225: packed-deleted-source + App proof) — all landed. What remains on this issue's own text, so I am not closing it unilaterally:

Migration step 8 — create-agent-bundle templates still ship plain Rstest tests against their own modules (verified: no agent-bundle/rstest plugin, no agent-bundle/test imports in templates/*/tests/). One route test + one compiled projection test per template is a small, unstaged follow-up.
Target-capability fixtures — the acceptance bullet on audio/non-text content projecting only when the target capability fixture advertises support has route-level fail-closed coverage (e.g. capability-unavailable tools) but no dedicated capability-fixture surface in agent-bundle/test.
Owned elsewhere: Workbench-surface helpers and render-stream presentation → Add an interactive Agent Workbench with local host discovery and lifecycle replay #105's lanes; the projection lifecycle/version-skew replay matrix → Generated plugin contract matrix: catch source→artifact→dev→installed-host drift before release #218; copied-artifact install mode as a distinctly labeled mode and native host layout matrices → repo release/nightly per meta-framework: execution plan and sequencing #107's cost rules (packed-native-smoke, packed release matrix).

If the two open items above are re-homed to follow-up issues, this issue can close.

feat(create): scaffold the framework test harness with the mcp-server…

The two items left open by the stage-4 landing comment are done. This completes the issue's own checklist, so it closes.

Migration step 8 — create-agent-bundle template harness adoption: #244, squash 048bb8d6b.

The mcp-server template scaffolds with the testing story wired in: rstest.route-unit.config.ts + rstest.projection.config.ts from agentBundleRstest(), one route-unit test (testManifest() smoke, then renderRoute + expectDocument over tool:status/report-status, including the resultSchema-parsed result) and one compiled projection test at the mcp-in-memory level (listMcpSurface + invokeMcpTool, asserting the protocol's structuredContent equals the value the same input rendered — parity, not a copied literal). test excludes both pools; test:routes / test:projection run them; check runs all three.
cli-tool and minimal deliberately ship no harness pool — verified, not assumed: the cli-tool's src/cli.ts is claimed by the config scripts: declaration and never becomes a compiled route (its manifest has routes: []), and minimal is skills-only with zero routes; a pool in either would pass vacuously. Both READMEs document what each level would prove and the exact wiring to add with the first route module.
The release-boundary scaffold matrix now asserts positively that the scaffolded project's harness pools ran (check output names both files; explicit test:routes/test:projection runs assert zero failures) instead of trusting a silent check. Per meta-framework: execution plan and sequencing #107 cost rules the per-PR packed pool stays minimal-template-only.
Bonus pool-leak fix: rstest.unit.config.ts now excludes packedReleaseOnlyTestFiles, so the pack-and-install scaffolder matrix no longer runs inside the build-free per-PR unit pool (scoped create-agent-bundle unit run: 4 files/50s → 3 files/0.43s).

Target-capability fixtures for non-text content: #239, squash 6a4490741.

createTargetCapabilityFixture() (agent-bundle/test): an immutable fixture over the issue's capability list — image, audio, resource, progress, with text as the non-configurable always-supported baseline. Nothing defaults to supported: the caller chooses every boolean and the richContentFallback: 'text' | 'fail' policy, so unsupported content can never be accepted by omission.
projectTargetCapabilities() reprojects a real renderRouteEvents() stream through the runtime's own projectMcpRenderStream with exactly the advertised capabilities; progress rides the real progress-token/notification contract. Provenance stays honestly route-unit — real renderer, real projector, no transport/packed/host claim.
Both acceptance paths are pinned: denied + 'text' asserts the runtime's exact placeholder blocks ([image <mime>], [audio <mime>], [resource <name> <uri>]) with no rich block leaking; denied + 'fail' fails closed as AgentTestError unsupported-rich-content, naming the offending kind and printing the proof level.
expectDocument() gained field-aware toContainImage / toContainAudio / toContainResource; existing text assertions are proven not to widen. The fixture route emits text + PNG + WAV + a binary data-URI resource plus request-scoped progress.

Gates: both PRs green on CI (Verify, Release gates, examples, micro-eval). Locally: full unit/route-unit/projection pools, typecheck, rslint clean on both branches post-rebase; the templates PR additionally ran test:packed:release — both matrix templates passed with the new positive assertions.

Environment note (pre-existing, untouched by either PR): packed-consumer.test.ts :: uses only an installed tarball after source deletion still parses npm pack --json as an array and fails under npm 12's package-keyed object — the same npm-12 shape the stage-3/4 work fixed in run-packed-tests.mjs, tests/support/shared-pack.ts, and audit-packed-release.mjs but not in that one test file. The one-line fix is to mirror packOutputFromJson's object/array tolerance there.

Checklist accounting: stages 1–4 landed (#151, #190, #212, #225); migration step 8 and the target-capability acceptance bullet landed above. The remaining surfaces were already re-homed by the stage-4 comment: Workbench-surface helpers and render-stream presentation → #105, projection lifecycle/version-skew replay matrix → #218, copied-artifact install mode and native host layout matrices → repo release/nightly per #107. Closing.

Changesets: agent-bundle minor (new public test surface), create-agent-bundle minor (new scaffolded capability).

Follow-up on the pre-existing npm 12 failure noted in the closing comment: packed-consumer.test.ts still assumed npm ≤11's array-shaped npm pack --json output. PR #225 fixed that at shared-pack.ts, audit-packed-release.mjs, and run-packed-tests.mjs; this file was missed.

Fixed in #248 — packed-consumer.test.ts now reuses packOutputFromJson from shared-pack.ts.

Closed-issue re-audit (G3) — VERIFIED for stages 1–4 + the two re-homed items; two issue-text items had no owner → now #370.

On origin/main (af1c18551): agent-bundle/rstest, agent-bundle/test, agent-bundle/test/browser are real subpath exports with @rstest/core an optional peer; eight labeled proof levels (src/test/index.ts:9-18); createTargetCapabilityFixture/projectTargetCapabilities (src/test/target-capabilities.ts:69,147), removeProjectSource/openPackedMcpServer({ deletedSource }) (src/test/packed.ts:221,284), mountBrowserApp (src/test/browser.ts:257), agentBundleBrowserRstest (src/rstest/browser.ts:90); the mcp-server template ships rstest.route-unit.config.ts + rstest.projection.config.ts with one route-unit and one projection test; packed-consumer npm-12 fix (#248) is on main. Run green here: pnpm test:route-unit 35/35, test:projection 63/63, packed-stdio-projection (packed-deleted-source + App proof over the wire) ✓, scaffold-packed.e2e ✓, audiobook + worktree-proximity + rsc-agent-runtime route-unit pools ✓, examples/mcp-app browser-app pool via Examples check CI ✓.

Fell through the cracks: (1) the issue's proposed runScript helper / level-2 "executes plain and rendered scripts" never shipped in the consumer harness (scripts are proven only in the repo's own cli-routes-build packed test); (2) "Workbench-surface browser helpers" were re-homed to #105 in the stage-3 comment, and #105 closed without them. Both recorded in #370 for an explicit build-or-decline decision.

Navigation Menu

Repository navigation

Generate an Agent Bundle framework test harness on top of Rstest #103

Summary

Existing capabilities to preserve

Goals

Proposed public surface

Test levels

1. Route unit harness

2. Projection contract harness

3. Browser harness

4. Artifact harness

Compiler coupling

Generated fixtures and capabilities

Matchers

Failure diagnostics

Migration and delivery

Acceptance criteria

Design references

Stack position

Full meta-framework stack

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Generate an Agent Bundle framework test harness on top of Rstest #103

Description

Summary

Existing capabilities to preserve

Goals

Proposed public surface

Test levels

1. Route unit harness

2. Projection contract harness

3. Browser harness

4. Artifact harness

Compiler coupling

Generated fixtures and capabilities

Matchers

Failure diagnostics

Migration and delivery

Acceptance criteria

Design references

Stack position

Full meta-framework stack

Activity

ScriptedAlchemy commented on Sep 1, 2026

ScriptedAlchemy commented on Sep 1, 2026

ScriptedAlchemy commented on Sep 1, 2026

ScriptedAlchemy commented on Sep 1, 2026

ScriptedAlchemy commented on Sep 2, 2026

ScriptedAlchemy commented on Sep 2, 2026

ScriptedAlchemy commented on Sep 3, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions