Repository navigation
Commit 2aff26d
authored
feat: native generative UI (createGenerativeUI, tool and markdown transports, standard catalog, inspector pane) (#7819)
Try it: run `examples/v2/generative-ui-shop` (setup in its README) and
ask the chat for "Top rated" products. The home row on the page changes
to what the agent picked. This PR makes generative UI one feature: an
agent streams React components into the chat or into named areas of the
page, with the `render_ui` tool (tool transport) or with a2ui code
fences in its reply (markdown transport). It includes a standard catalog
of 69 components (the A2UI 1.0 basic catalog, from #7879, with the
follow-ups of #7888, #7890 and #7898), the work of #7829 and #7848, two
new examples, and a Generative UI pane in the inspector.
## What does this PR do?
### Catalog (`@copilotkit/react-core/v2/generative-ui`)
- `createGenerativeUI({ id, extends, state, functions, overrides,
components })` declares the components, the app state, and the
functions. Schemas can be any Standard Schema (zod 3, zod 4.2+, Valibot,
ArkType). Props need a JSON Schema, and props and state must validate
synchronously.
- Binding: a prop with the name of a state key, or a `bind` selector,
gets its value from app state. An event with the name of a function (or
`onX` for `x`) calls it. An event that no function matches becomes a
visible user message.
- `writable: ["value"]` gives a component a typed `setValue`. A path
that the user wrote becomes user-owned: later agent writes to it are
ignored.
- `extends` merges catalogs, with full types for every component.
`overrides: { Card: { component: MyCard } }` replaces the React
component of a base component and keeps what the model sees.
`standardUI.omit(["Map"])` drops components, also from the prompt.
- `useGenerativeUIState(ui, key, value)` and
`useGenerativeUIFunction(ui, name, handler)` connect app state and
handlers, with types from `ui`.
### Surfaces
- `<GenerativeUISurface id description ui components fallback />` is a
named area of the page. The fallback shows until the agent renders into
it.
- The `ui` prop types `components`, so a wrong component name is a `tsc`
error. `components` needs `ui`.
- `components` limits the area to those components. Other components are
rejected, and the model gets the reason.
- The page scrolls to a surface when the agent starts to render into it,
unless the surface is already on screen. `autoScroll={false}` turns this
off for one surface.
### Standard catalog and theme
- `standardUI`: 69 components for layout, text, inputs, feedback, data,
media, and composites (the a2ui-renderer catalog, ported in #7848).
Chart uses `recharts`. Map loads `maplibre-gl` (an optional peer).
- Built-in dark mode: the components switch to dark values under a
`.dark` class, like the CopilotKit chat. The map uses a dark basemap.
- `theme` keys are the exact `--a2ui-*` variables, so a wrong name is a
type error. Top-level values apply in both modes, and `dark: { … }`
replaces them in dark mode. The theme is a stylesheet on the surface
root.
### Wire and runtime (`@copilotkit/shared`, `@copilotkit/core`, runtime)
- A2UI v1.0 only, in both transports: `createSurface` with inline
`components` and `dataModel`, `{"@path"}` bindings, `{"@call"}` function
calls in events, `{"@call":"@index"}` in templates, `userMessage` on
events, and `"value": null` deletes. `catalogId` is optional.
- `supportedCatalogIds` lists the id of the UI first, then the id of
every catalog in its `extends`. An operation can name any of these ids
in `catalogId`, and the surface keeps that id. An unknown id on a
component or a function call is rejected with a reason. An unknown id on
`createSurface` is replaced with the id of the UI.
- `callRendererFunction` is off for now (#7890): the prompts do not
mention it, and the validator rejects it with a reason. The code and its
tests stay behind `GENERATIVE_UI_AGENT_FUNCTION_CALLS`.
- The manifest goes to `forwardedProps.copilotkit.ui` (format 3, the
v1.0 renderer capabilities shape), only to the agents in `agentIds`. The
guide sends only the rules that the catalog needs (about 575 tokens for
a small catalog, 930 with all rules).
- `CopilotRuntime({ generativeUI: true })` gives tool-transport runs
`render_ui` through `GenerativeUIMiddleware`. A markdown-transport run
never gets it.
- Markdown transport: the fence parser from #7829 streams operations
component by component. A fence with `target=<id>` renders in that
surface, and the v1 chat (`@copilotkit/react-ui`) renders a2ui fences
too.
### Inspector (`@copilotkit/web-inspector`)
A new **Generative UI** pane, with three tabs:
- **Surfaces**: each surface with its status, target, rejected
operations, data model, and user values. A tree shows the rendered
components and their props. It follows `slots` and the `child()` /
`childList()` refs at any depth. Hover over the card of a page surface
to outline it on the page and scroll to it.
- **Catalog**: the components, split by the catalog that declared them.
The provider catalog comes first. A base catalog from `extends` has an
"extended" tag and starts closed. A small search filters components and
functions.
- **Prompt**: what the last run sent and what the next run will send:
the envelope and the `render_ui` guide. The large text is drawn only
when you open it.
React-core gives core an internal debug source
(`ɵGenerativeUIDebugSource`), and the pane reads it. The prompt builder
moved from runtime to `@copilotkit/shared`, so the pane shows the same
guide that the runtime builds.
### Examples
- `examples/v2/generative-ui-shop` (new, React Router): the shop from
the AI-powered apps workshop, with Prisma and SQLite. A TanStack AI
agent with server tools fills four page surfaces: the home row, the
product recommendations, the cart items, and the cart suggestions.
- `examples/v2/a2ui-markdown` (Next.js): both transports, page surfaces
with pills, and `/catalog` with every standard component.
- `examples/v2/react/demo` `/generative-ui`: a storefront with `bind`,
`writable`, and a limited surface.
### Known gaps
- Agent-called functions (`callRendererFunction`) are off. Their result
goes back as a developer message, and not every backend keeps that
message where the model reads it in order.
- In markdown mode, a surface cannot be created again after
`deleteSurface`.
- `HtmlView` content stays light-styled in dark mode: it is a sandboxed
iframe that the theme cannot reach.
- From A2UI v1.0, not supported yet: catalogs by id (without an inline
catalog), and `sendDataModel`. `extends` merges catalogs into one
namespace, and overrides replace by name.
- Several catalogs in one surface work only in part. A component is not
checked against the catalog that the surface names, and the prompt does
not tell the agent to set `catalogId`.
- `callAgentFunction` and `agentFunctionResponse` are not implemented. A
user event starts a new run instead.
- `@a2ui/web_core` stays 0.10.4: the engine uses its `v0_9` surface and
data models. Version 0.14 has `v1_0` models, and the move is a separate
integration task.
- `@ag-ui/a2ui-toolkit` 0.0.5 writes v0.9 operations, so agent tools
built with it do not work with this engine. The `render_ui` route does
not use the toolkit, and it works with a LangGraph Python agent (see
Testing).
- The standard catalog keeps each React component and its schema in one
entry. `exportA2UICatalog` gives the portable JSON, but definitions
without React are not split out yet.
## Related PRs and Issues
- Contains the work of #7829 (A2UI markdown mode), ported to this
engine. #7829 itself is not changed.
- Contains #7848 by @mme: the a2ui-renderer standard catalog and
markdown-mode behavior on `createGenerativeUI`.
- Contains #7879: the A2UI 1.0 basic catalog names, typed child
references, dynamic values, checks, and `calc`.
- Contains #7888: A2UI 1.0 follow-ups: action events, function
permissions, the component envelope, checks severity, Video,
`exportA2UICatalog`, and an adoption guide.
- Contains #7890: no agent-called renderer functions for now, the A2UI
default for `allowedCallers`, the experimental callout, and two prompt
rules.
- Contains #7898: drops `Loading`, `LoadingIndicator`, `LoadingBlock`
and `Celebration`.
## Checklist
- [ ] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [x] If the PR changes or adds functionality, I have updated the
relevant documentation
- [ ] If the PR contains a breaking change, the title has `!` (for
example `feat(react-core)!: ...`) and the description has a `BREAKING
CHANGE:` line with migration steps. See
[VERSIONING.md](https://github.com/CopilotKit/CopilotKit/blob/main/VERSIONING.md)
- [ ] "Allow edits by maintainers" is checked (lets us help iterate on
your PR directly — faster turnaround for everyone)
Docs: a new "Components and surfaces" section under Generative UI, in
the Declarative group
(`showcase/shell-docs/src/content/docs/generative-ui/components/`), with
9 pages, and an adoption guide from the `a2ui` renderer is one of them.
Its support matrix lists these limits. The quickstart has a "When
something does not render" table that points to the tab of the inspector
pane to open. The Inspector page, `surfaces.mdx`, and `catalogs.mdx`
link to the pane too. Reference pages for `createGenerativeUI`,
`GenerativeUISurface`, and the hooks come in a later PR. There is no
breaking change for users: none of these APIs is on `main`.
## Testing
**Commands run**
- `tsc --noEmit` passes in react-core and in the three examples
(`generative-ui-shop`, `a2ui-markdown`, `react/demo`), after a rebuild
of react-core.
- The generative UI tests of react-core pass: 457 tests in 29 files.
They include tests for Standard Schema with a non-zod schema, the theme
stylesheet, overrides, `omit`, and the type rules.
- `pnpm install --frozen-lockfile --lockfile-only` passes for all 86
workspace projects.
- The lefthook pre-commit hook ran on every commit, except the package
test step of three lockfile commits (`032d695e21`, `89ec79e649`,
`ae485ca8bf`). A lockfile change makes that step run the Go and Ruby
runtime tests, which cannot run on the author's machine. CI runs them.
- Inspector pane and its merge with #7879: builds of shared, core,
react-core, and web-inspector, then `tsc --noEmit` in those four and in
runtime. These tests pass: `generative-ui-view.spec.ts` and
`hud-conformance.test.ts` (69), `use-generative-ui-root.test.tsx` and
`catalog.test.tsx` (46), `generative-ui-debug.test.ts` (2),
`generative-ui-middleware.test.ts` (44), and `shared/src/generative-ui`
(76).
- The inspector commits skipped the package test step of the pre-commit
hook. In the full hook run, 3 react-core chart tests and 1 vue SSR test
timed out under load. They are not in changed files, and all 4 pass
alone. CI runs the full suites.
- Merge of #7888 into this branch: builds of shared, core, react-core,
and web-inspector, `tsc --noEmit` in web-inspector, and the 69 inspector
tests pass. The docs commit has no code.
- Real backend: the LangGraph Python agent of
`examples/v2/a2ui-markdown` (`CopilotKitMiddleware`, `uv sync` from its
`uv.lock`) with an OpenAI model, on both transports. With the tool
transport, it rendered a dashboard into a page target, then updated the
same surface in place. A sign-up form showed its checks, and Create
account stayed disabled until every field was valid. The submit event
started a new run, and the agent repeated the 4 values. The inspector
showed 2 surfaces and no rejected operations. With the markdown
transport, a fence rendered a checklist in the `notes` target, and no
fence text appeared in the chat. The agent and the runtime logged no
errors.
- The catalog ids of `extends` (`2ac89b6e67`), after a rebase on #7898:
builds of shared and core, then `tsc --noEmit` in shared, core,
react-core and runtime. The touched test files pass:
`validate-operation.test.ts` and `a2ui-envelope.test.ts` (102),
`a2ui-markdown-operations.test.ts` (74), `catalog.test.tsx`,
`markdown-source.test.ts`, `surface-manager.test.ts` and
`use-generative-ui-root.test.tsx` (130), and
`generative-ui-middleware.test.ts` (42). The full pre-commit hook
passed. In one earlier hook run, the react-core `Chart` test in
`standard/__tests__/data.test.tsx` timed out under load. It passes
alone.
- Not run locally: full package suites and e2e. CI runs them.
**Manual test**
1. Follow the setup in `examples/v2/generative-ui-shop/README.md`, then
run `pnpm example-dev` and open http://localhost:3012.
2. On the home page, click **Top rated**. The Featured Products row
changes, and the page scrolls to it. Click **Featured again** to bring
the original row back.
3. Open a product and click **Cheaper alternatives**. The "You Might
Also Like" row changes.
4. Add a product to the cart, open the cart, and click **Show my cart
differently**. The cart items change to detailed cards.
5. Add the `dark` class to `<html>` in the devtools of
`examples/v2/a2ui-markdown` `/catalog`. Every standard component
switches to dark.
6. In the shop, open the CopilotKit inspector and click **Generative
UI**. Open **Catalog**, and type `button` in the search: the standard
catalog opens with its matches. On **Surfaces**, hover over a page
surface card: the surface gets an outline on the page.
7. Run the LangGraph agent of `examples/v2/a2ui-markdown` (see
`agent/README.md`), then open
http://localhost:3011/?agent=langgraph&transport=tool. Click **Dashboard
on the page**, then **Update the dashboard**: the same surface changes.
Click **Sign-up with validation**, fill the form, and submit it: the
agent repeats what you entered.
**How this PR makes testing easy**
- Unit and type tests in
`packages/web-inspector/src/components/generative-ui-view.spec.ts`,
`packages/react-core/src/v2/generative-ui/**/__tests__`,
`packages/shared/src/__tests__/a2ui-*.test.ts`,
`packages/core/src/__tests__/a2ui-markdown-*.test.ts`, and
`packages/runtime/src/v2/runtime/__tests__/generative-ui-*.test.ts`.
- Three example apps (see Examples above). The shop and a2ui-markdown
were tested with a real model, and `/catalog` renders with no model.
- The docs section walks through each feature with copyable code:
surfaces, catalogs, the standard catalog, state and actions, theming,
transports, markdown mode, and the adoption guide from A2UI.
## Risk / rollback
Low risk for current users. All new APIs are experimental and opt-in,
and the `a2ui` prop path does not change. react-core gets `recharts` as
a dependency and `maplibre-gl` as an optional peer; apps that do not
import the subpath do not bundle them. The new shop example adds Prisma,
TanStack AI React, and Faker to the lockfile. To undo, revert the PR.
## Public API change
New experimental exports. Nothing that exists on `main` changes.
**Before**
```tsx
<CopilotKitProvider runtimeUrl="/api/copilotkit" a2ui={{ catalog }}>
<CopilotChat />
</CopilotKitProvider>
```
**After**
```tsx
import { createGenerativeUI, GenerativeUISurface, standardUI } from "@copilotkit/react-core/v2/generative-ui";
const ui = createGenerativeUI({
id: "acme/app@1",
extends: [standardUI.omit(["Map"])],
overrides: { Card: { component: MyCard } },
components: { ProductCard: { component: ProductCard, props: productSchema } },
});
<CopilotKitProvider
runtimeUrl="/api/copilotkit"
generativeUI={{ ui, theme: { "--a2ui-color-accent": "#d97706", dark: { "--a2ui-color-panel": "#1f2937" } } }}
>
<GenerativeUISurface id="featured" ui={ui} components={["ProductCard"]} fallback={<Featured />} />
<CopilotChat />
</CopilotKitProvider>
// Server
new CopilotRuntime({ agents, generativeUI: true });
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added v2 Generative UI with interactive surfaces, custom catalogs,
state bindings, actions, theming, validation, charts, maps, media, and
responsive standard components.
* Added tool and Markdown transports with streamed `a2ui` content,
targeted surfaces, inline rendering, and chat integration.
* Added Generative UI inspection in the web Inspector, showing catalogs,
surfaces, prompts, state, and rejected operations.
* Added generative UI examples, including a Markdown demo, storefront,
catalog gallery, shopping assistant, cart, and product comparison
experience.
* **Documentation**
* Added setup and reference guides for catalogs, surfaces, transports,
theming, state, actions, and Inspector workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->276 files changed
Lines changed: 61102 additions & 500 deletions
File tree
- .github
- examples/v2
- a2ui-markdown
- agent
- src
- app
- api/copilotkit/[[...slug]]
- catalog
- public
- generative-ui-shop
- app
- components
- contexts
- domain
- generative-ui
- hooks
- prompts
- routes
- _landing
- +/index
- products
- +/index
- schemas
- tools
- utils
- prisma
- migrations
- 20251002111449_init_project
- 20251003085813_add_comments_to_product_review
- 20251004101747_add_category_images_to_categories
- public
- react/demo/src/app
- api/copilotkit-generative-ui/[[...slug]]
- generative-ui
- packages
- a2ui-renderer/src/react-renderer
- core
- core/src
- __tests__
- fixtures/a2ui-markdown
- a2ui-markdown
- core
- __tests__
- react-core
- src/v2
- __tests__
- components/chat
- generative-ui
- __tests__
- markdown
- standard
- __tests__
- fixtures
- providers
- react-ui
- src/components/chat
- messages
- runtime/src/v2/runtime
- __tests__
- core
- generative-ui
- handlers
- web-inspector/src
- components
- lib
- __tests__
- scripts/release/public-api
- showcase/shell-docs/src/content
- docs
- generative-ui
- a2ui
- components
- skills/inspector-docs/references
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
80 | 80 | | |
81 | 81 | | |
82 | 82 | | |
| 83 | + | |
| 84 | + | |
83 | 85 | | |
84 | 86 | | |
85 | 87 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
Whitespace-only changes.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
0 commit comments