Skip to content

Commit d966b88

Browse files
authored
fix(opengenui): define the ready-DOM initialization contract (#7662)
A generated calculator could render correctly while every button remained inert: its `jsExpressions` registered `DOMContentLoaded` after Websandbox had already loaded the document. This is a shared generation-contract gap, not a LangGraph TypeScript adapter failure. Centralize the React, Angular, and Vue tool description in `@copilotkit/shared` and explicitly require direct initialization against the ready DOM. Helpers belong in `jsFunctions`; the first expression initializes controls once. Existing sandbox protections and renderer execution semantics are unchanged. Historical payloads that wait for a past event must be regenerated; this change neither rewrites saved JavaScript nor redispatches readiness events. Validation: - Reproduced the captured failing calculator in real Chromium: clicking `2` left `0`. - Five real-browser tests pass against both source and exact-commit packed React/shared artifacts: failing-payload characterization, supported streaming/replay, once-only initialization, reopening, `2 + 3 = 5`, metric shortcuts, and blocked host/storage access. - Generated an unmodified `gpt-5-mini` response using the normal Calculator App suggestion and updated tool description; both streaming and replay compute `5` and insert the revenue shortcut. Captured it as a deterministic CI fixture. This is direct real-model generation plus real-renderer testing, not an end-to-end LangGraph agent transport run. - Shared, React, Angular, and Vue: builds, explicit typechecks, 3,985 unit tests, publint, and packed type-export checks pass. One pre-existing Vue test remains skipped. - Formatting, frozen lockfile, and scoped lint pass (two existing React provider warnings). Browser regression added to the Node 22 / React 19 CI lane. The pre-commit hook expanded the lockfile change to 40 projects and failed on unavailable dependencies/native toolchains outside this narrow change. Its `test-packages` tag was excluded only after explicitly running the changed-package checks above; other hooks ran normally. Fixes PNI-585. Validated hosted preview (`ba41e270e0875ddd3a45fe1b480e09ff09465a2e`): - [shared](https://pkg.pr.new/@copilotkit/shared@ba41e270e0875ddd3a45fe1b480e09ff09465a2e) - [react-core](https://pkg.pr.new/@copilotkit/react-core@ba41e270e0875ddd3a45fe1b480e09ff09465a2e) - [angular](https://pkg.pr.new/@copilotkit/angular@ba41e270e0875ddd3a45fe1b480e09ff09465a2e) - [vue](https://pkg.pr.new/@copilotkit/vue@ba41e270e0875ddd3a45fe1b480e09ff09465a2e) The downloaded hosted React/shared packages also passed all five Chromium tests. Install React and shared together. These are fresh-main artifacts; cherry-pick this commit onto the combined validator candidate to retain its other pending fixes. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Newly generated interactive UI follows updated initialization guidance for live updates and replay. * **Tests** * Added browser-based coverage for calculator rendering, interactions, replay behavior, and sandbox restrictions. * Added automated browser-test runs for eligible React 19 and Node 22 checks. * **Documentation** * Added guidance on running the browser regression suite and what it covers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Review follow-up (`289830b0bb`): convert the browser harness file URL using `fileURLToPath`, so paths with spaces and Windows filesystem paths are handled correctly. Reproduced the previous server-start failure in a directory containing spaces; all five Chromium tests pass there after the fix. This follow-up only changes test configuration; production package sources are unchanged from the validated preview above.
2 parents 98f9c15 + 5f55b2e commit d966b88

15 files changed

Lines changed: 332 additions & 42 deletions

File tree

‎.github/workflows/test_unit.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -271,6 +271,14 @@ jobs:
271271
if: matrix.node-version == '22.x'
272272
run: pnpm --dir showcase/scripts exec playwright install chromium
273273

274+
- name: Install Chromium for Open Generative UI regression
275+
if: ${{ matrix.node-version == '22.x' && matrix.react-version == '19' && contains(steps.select.outputs.projects, '@copilotkit/react-core') }}
276+
run: pnpm --dir packages/react-core exec playwright install chromium
277+
278+
- name: Run Open Generative UI browser tests
279+
if: ${{ matrix.node-version == '22.x' && matrix.react-version == '19' && contains(steps.select.outputs.projects, '@copilotkit/react-core') }}
280+
run: pnpm nx run @copilotkit/react-core:test:browser
281+
274282
- name: Run Web Inspector browser tests
275283
if: ${{ matrix.node-version == '22.x' && matrix.react-version == '19' && contains(steps.select.outputs.projects, '@copilotkit/web-inspector') }}
276284
run: pnpm nx run @copilotkit/web-inspector:test:browser

‎packages/angular/src/lib/open-generative-ui.ts‎

Lines changed: 1 addition & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -16,19 +16,7 @@ export const DEFAULT_OPEN_GENERATIVE_UI_DESIGN_SKILL = `When generating UI with
1616
- Minimal transitions (150ms) for hover/focus states only. No decorative animations.
1717
- Keep the UI focused and dense — avoid excessive padding. Use compact spacing (8–12px gaps, 10–14px padding in controls).`;
1818

19-
export const GENERATE_SANDBOXED_UI_DESCRIPTION =
20-
"Generate sandboxed UI. " +
21-
"IMPORTANT: The generated code runs in a sandboxed iframe WITHOUT same-origin access. " +
22-
"Do NOT use localStorage, sessionStorage, document.cookie, IndexedDB, or fetch/XMLHttpRequest to same-origin URLs. " +
23-
"To communicate with the host application, use Websandbox.connection.remote.<functionName>(args) which returns a Promise.\n\n" +
24-
"You CAN use external libraries from CDNs by including <script> or <link> tags in the HTML <head> (e.g., Chart.js, D3, Three.js, x-data-spreadsheet, etc.). " +
25-
"CDN resources load normally inside the sandbox.\n\n" +
26-
"PARAMETER ORDER IS CRITICAL — generate parameters in exactly this order:\n" +
27-
"1. initialHeight + placeholderMessages (shown to user while generating)\n" +
28-
"2. css (all styles FIRST — the user sees a placeholder until CSS is complete)\n" +
29-
"3. html (streams in live — the user watches the UI build as HTML is generated)\n" +
30-
"4. jsFunctions (reusable helper functions)\n" +
31-
"5. jsExpressions (applied one-by-one — the user sees each expression take effect)";
19+
export { GENERATE_SANDBOXED_UI_DESCRIPTION } from "@copilotkit/shared";
3220

3321
export const OpenGenerativeUIContentSchema = z.object({
3422
initialHeight: z.number().optional(),
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Open Generative UI browser regressions
2+
3+
Run `pnpm nx run @copilotkit/react-core:test:browser` from the workspace root.
4+
Install the matching Chromium first with
5+
`pnpm --dir packages/react-core exec playwright install chromium`.
6+
7+
The harness bundles the real React renderer and Websandbox. No iframe, script
8+
execution, or browser readiness event is mocked. It listens on localhost only.
9+
10+
- `fixtures/calculator.json` preserves the HTML and JavaScript from PNI-585.
11+
Registering `DOMContentLoaded` after sandbox readiness leaves it inert.
12+
The supported variant changes only that readiness wrapper into a directly
13+
invoked initializer; its counter detects duplicate execution.
14+
- `fixtures/normal-suggestion.json` is an unmodified gpt-5-mini-2025-08-07 tool
15+
response generated with the shared tool description on 2026-10-06. The user
16+
message was the normal Calculator App suggestion:
17+
18+
> Using the generateSandboxedUi tool, build a modern calculator with standard buttons plus labeled metric shortcut buttons that insert their values into the display when clicked. Use sample company data.
19+
20+
Both supported fixtures exercise live expression delivery and saved-content
21+
replay. Assertions cover arithmetic and metric buttons; the instrumented fixture
22+
also checks initialization counts, reopening, and denied host/storage access.
23+
The captured real-model response keeps CI deterministic and credential-free.
24+
25+
This contract change guides newly generated code. It does not rewrite historical
26+
JavaScript or redispatch document readiness events; existing inert payloads must
27+
be regenerated.
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
import type { OpenGenerativeUIContent } from "../src/v2/components/OpenGenerativeUIRenderer";
2+
import React from "react";
3+
import { createRoot } from "react-dom/client";
4+
import { OpenGenerativeUIActivityRenderer } from "../src/v2/components/OpenGenerativeUIRenderer";
5+
6+
const root = createRoot(document.getElementById("root")!);
7+
Object.assign(window, {
8+
renderContent(content: OpenGenerativeUIContent, key = "fixture") {
9+
root.render(
10+
<OpenGenerativeUIActivityRenderer
11+
key={key}
12+
activityType="open-generative-ui"
13+
content={content}
14+
message={{}}
15+
agent={{}}
16+
/>,
17+
);
18+
},
19+
});
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"html": [
3+
"<body><div class=\"calculator\" role=\"application\" aria-label=\"Metric calculator\">\n <div class=\"display\" aria-live=\"polite\">\n <div class=\"expr\" id=\"expr\">&nbsp;</div>\n <div class=\"value\" id=\"display\">0</div>\n </div>\n\n <div class=\"metrics\" id=\"metrics\">\n <button class=\"metric-btn\" data-value=\"327700\" aria-label=\"Total Revenue 327,700\">\n <div class=\"metric-label\">Total Revenue</div>\n <div class=\"metric-value\">$327,700</div>\n </button>\n <button class=\"metric-btn\" data-value=\"6\" aria-label=\"New Customers 6\">\n <div class=\"metric-label\">New Customers</div>\n <div class=\"metric-value\">6</div>\n </button>\n <button class=\"metric-btn\" data-value=\"94.2\" aria-label=\"Conversion Rate 94.2 percent\">\n <div class=\"metric-label\">Conversion Rate</div>\n <div class=\"metric-value\">94.2%</div>\n </button>\n <button class=\"metric-btn\" data-value=\"54.23\" aria-label=\"Average Order Value 54.23 dollars\">\n <div class=\"metric-label\">Avg Order Value</div>\n <div class=\"metric-value\">$54.23</div>\n </button>\n </div>\n\n <div class=\"keys\" role=\"grid\">\n <button class=\"btn secondary\" data-action=\"clear\">C</button>\n <button class=\"btn secondary\" data-action=\"back\">⌫</button>\n <button class=\"btn secondary\" data-value=\"%\">%</button>\n <button class=\"btn operator\" data-value=\"/\">÷</button>\n\n <button class=\"btn\" data-value=\"7\">7</button>\n <button class=\"btn\" data-value=\"8\">8</button>\n <button class=\"btn\" data-value=\"9\">9</button>\n <button class=\"btn operator\" data-value=\"*\">×</button>\n\n <button class=\"btn\" data-value=\"4\">4</button>\n <button class=\"btn\" data-value=\"5\">5</button>\n <button class=\"btn\" data-value=\"6\">6</button>\n <button class=\"btn operator\" data-value=\"-\">−</button>\n\n <button class=\"btn\" data-value=\"1\">1</button>\n <button class=\"btn\" data-value=\"2\">2</button>\n <button class=\"btn\" data-value=\"3\">3</button>\n <button class=\"btn operator\" data-value=\"+\">+</button>\n\n <button class=\"btn\" data-value=\"0\" style=\"grid-column: span 2\">0</button>\n <button class=\"btn\" data-value=\".\">.</button>\n <button class=\"btn equal\" data-action=\"equals\">=</button>\n </div>\n\n <div class=\"footer\">Metric shortcuts insert sample company values into the display.</div>\n</div></body>"
4+
],
5+
"htmlComplete": true,
6+
"cssComplete": true,
7+
"initialHeight": 700,
8+
"jsFunctions": "function sanitizeExpression(expr){ // remove any characters except digits, operators, parentheses, decimal and percent\n return expr.replace(/[^0-9+\\-*/().% ]/g,'');}\n\nfunction evaluateExpression(expr){ try{\n // handle percentage: replace 'number%' with '(number/100)'\n expr = expr.replace(/(\\d+(?:\\.\\d+)?)%/g, '($1/100)');\n expr = sanitizeExpression(expr);\n // avoid accidental leading operators\n if(!expr.trim()) return '0';\n // Use Function to evaluate simple arithmetic\n var result = Function('return ' + expr)();\n if(typeof result === 'number' && !isFinite(result)) return 'Error';\n if(typeof result === 'number'){\n // format with up to 6 decimals, trim trailing zeros\n return Number.isInteger(result)? result.toString() : parseFloat(result.toFixed(6)).toString();\n }\n return result.toString();\n }catch(e){return 'Error';}}\n\nfunction updateDisplay(text, exprText){\n var display = document.getElementById('display');\n var expr = document.getElementById('expr');\n display.textContent = text;\n expr.textContent = exprText || '\\u00A0';\n}\n\nfunction insertValue(val){\n var display = document.getElementById('display');\n var current = display.textContent || '0';\n // if current is '0' or Error, replace\n if(current === '0' || current === 'Error'){\n updateDisplay(val.toString());\n } else {\n updateDisplay(current + val.toString());\n }\n}\n\nfunction handleButtonClick(btn){\n var action = btn.getAttribute('data-action');\n var value = btn.getAttribute('data-value');\n var displayEl = document.getElementById('display');\n var exprEl = document.getElementById('expr');\n var cur = displayEl.textContent || '';\n\n if(action === 'clear'){\n updateDisplay('0','');\n return;\n }\n if(action === 'back'){\n if(cur.length <= 1){ updateDisplay('0',''); }\n else updateDisplay(cur.slice(0,-1));\n return;\n }\n if(action === 'equals'){\n var result = evaluateExpression(cur);\n updateDisplay(result, cur + ' =');\n return;\n }\n if(value){\n // numeric or operator\n // If current is result from last evaluation (contains '=' in expr), reset\n if(exprEl.textContent && exprEl.textContent.indexOf('=')!==-1){ updateDisplay('0',''); }\n // Insert value\n insertValue(value);\n }\n}\n",
9+
"jsExpressions": [
10+
"document.addEventListener('DOMContentLoaded', function(){\n // wire metric buttons\n document.querySelectorAll('.metric-btn').forEach(function(btn){\n btn.addEventListener('click', function(){\n var v = btn.getAttribute('data-value');\n // Insert with parentheses if it's a large number to allow arithmetic\n insertValue('('+v+')');\n });\n });\n\n // wire calculator buttons\n document.querySelectorAll('.btn').forEach(function(btn){\n btn.addEventListener('click', function(){ handleButtonClick(btn); });\n });\n\n // keyboard support\n document.addEventListener('keydown', function(e){\n var allowed = '0123456789+-*/().%';\n if(allowed.indexOf(e.key) !== -1){ insertValue(e.key); e.preventDefault(); }\n else if(e.key === 'Enter'){ handleButtonClick(document.querySelector('[data-action=\"equals\"]')); e.preventDefault(); }\n else if(e.key === 'Backspace'){ handleButtonClick(document.querySelector('[data-action=\"back\"]')); e.preventDefault(); }\n else if(e.key === 'Escape'){ handleButtonClick(document.querySelector('[data-action=\"clear\"]')); e.preventDefault(); }\n });\n\n // initialize\n updateDisplay('0','');\n});"
11+
],
12+
"generating": false
13+
}

0 commit comments

Comments
 (0)