Inspired by page-agent, but built on the emerging WebMCP standard via a polyfill.
- 📦 Just an npm package.
npm i @aralroca/gui-agentandimport. No script injection, no browser extension, no headless browser.
에이전트 강화
게시자의 자체 명령입니다. PlugMCP 엔드포인트가 아닙니다.
Host a copy with our keys and logs.
npx -y @aralroca/[email protected]Claude Code에는 로컬 명령용 설치 링크가 없습니다. 아래 명령을 복사해 터미널에서 실행합니다.
claude mcp add aralroca-gui-agent -- npx -y @aralroca/gui-agent@0.7.0터미널에서 실행합니다.
{ "mcpServers": { "aralroca-gui-agent": { "command": "npx", "args": [ "-y", "@aralroca/[email protected]" ] } }}claude_desktop_config.json
원격 URL인가요, 내 컴퓨터에서 실행하는 명령인가요? 두 설치 방식의 차이
PlugMCP에서 실행
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
실행게시자 제공
Inspired by page-agent, but built on the emerging WebMCP standard via a polyfill.
npm i @aralroca/gui-agent and import. No script injection, no browser extension, no headless browser.An open-source WebMCP-powered in-page GUI agent. Let a natural-language agent live inside your web app and drive its UI — no script injection, no browser extension, no headless browser.
document.modelContext (WebMCP). The polyfill (@mcp-b/webmcp-polyfill) installs it where browsers don't natively support it yet.@aralroca/gui-agent/ui).⚠️ Early/experimental. WebMCP is a moving W3C draft; APIs may change.
⭐ If gui-agent is useful to you, starring the repo helps other developers find it.
Thinking is cheap. But what if your agents didn't have to think so much—and instead had TOTAL CONTROL of your app (AUTOMATICALLY)? Take a look Janux Framework for more than gui-agent lib
npm i @aralroca/gui-agent
# optional peers, depending on what you use:
npm i ai zod # for the AI SDK adapter / Zod schemas
import { defineTool, GuiAgent } from "@aralroca/gui-agent";
import { createAiSdkLlm } from "@aralroca/gui-agent/ai-sdk";
// 1. Expose your app's actions as tools (producer side, optional).
defineTool({
name: "go_to_tab",
description: "Switch the console to a tab.",
inputSchema: { type: "object", properties: { tab: { type: "string" } }, required: ["tab"] },
annotations: { readOnlyHint: true },
execute: ({ tab }) => router.push(`/${tab}`),
});
// 2. Run the agent. It discovers your tools + synthesizes DOM tools (click/fill/read).
const agent = new GuiAgent({
llm: createAiSdkLlm({ model: "anthropic/claude-opus-4-8" }),
confirm: async (call) => window.confirm(`Allow "${call.name}"?`), // HITL gate
});
await agent.run("Invite [email protected] to the team as admin");
Importing the package installs the WebMCP polyfill automatically — no <script> tag.
gui-agent unifies two approaches:
defineTool(...) to register reliable, structured actions on document.modelContext. Any WebMCP agent — including a browser's native one — can use them.e7) and gets synthesized read_page / click / fill / select_option / drag / wait_for_text tools. No screenshots, no multimodal model needed.The built-in loop discovers all available tools, asks your LLM what to do, runs the calls (gated by your optional confirm), feeds results back, and repeats until done.
For each step the agent prefers a purpose-built WebMCP tool if one fits, and only infers the DOM when nothing is exposed for the job:
flowchart TD
Goal["🗣️ Goal in natural language"] --> Loop["🧠 Agent loop · bring your own LLM<br/>(AI SDK adapter · remote LLM · React bindings)"]
Loop --> Discover["🔍 Discover the available tools"]
Discover --> Decide{"Is there a purpose-built<br/>WebMCP tool for this step?"}
Decide -- "Yes · preferred" --> WebMCP["⚙️ Call the WebMCP tool<br/>structured & reliable"]
Decide -- "No · fallback" --> DOM["🖱️ Infer from the DOM<br/>read_page → click / fill / select_option / drag / wait_for_text"]
WebMCP --> Gate{"Does the tool have<br/>readOnlyHint?"}
DOM --> Gate
Gate -- "No · write action" --> Confirm["🙋 Human-in-the-loop confirm()"]
Gate -- "Yes · read-only" --> Run["Execute"]
Confirm -- "approved" --> Run
Confirm -- "denied" --> Feed
Run --> Feed["Feed the result back to the LLM"]
Feed --> Loop
Loop -. "no more tool calls · maxSteps" .-> Done["🏁 Done"]
subgraph MC["document.modelContext · WebMCP (polyfilled — no server, no browser extension)"]
direction LR
App["Your app's own actions<br/>defineTool() / useTool()"]
Ext["Tools registered by other code<br/>discoverExternalTools() · opt-in"]
end
Discover -. reads .-> App
Discover -. reads .-> Ext
The agent is handed a single merged tool list and is instructed to prefer your app's WebMCP tools; the synthesized DOM tools are the always-on fallback (a same-named app tool wins). WebMCP tools live on
document.modelContext— your app registers them (so any WebMCP agent, including the browser's native one, can call them) anddiscoverExternalTools()can pull in tools other code registered there.upload_fileandnavigateDOM tools are opt-in.
gui-agent is inspired by page-agent and shares its core idea — a natural-language agent that drives a real web page from a text-based DOM snapshot (no screenshots, no multimodal model) with a bring-your-own-LLM design. The difference is what gui-agent adds by building on the emerging WebMCP standard instead of DOM-driving alone:
When page-agent is the better fit. page-agent is excellent and has real strengths gui-agent doesn't: it drives any existing site's DOM without the app opting in, so it works on legacy or third-party apps (ERP, CRM, admin panels) you don't control — whereas gui-agent's producer tools need the app to register them (its DOM fallback still works anywhere, but the WebMCP value comes from instrumentation). page-agent also ships an optional Chrome extension for cross-tab tasks and a Node MCP server so external agents (Claude Desktop, Cursor) can drive a real browser. Both projects are MIT-licensed, TypeScript, text-based, and ship an activity visualizer.
In short: reach for page-agent to automate sites you don't own; reach for gui-agent when you do own the app and want standards-based, structured, human-gated tools — with a DOM fallback for everything else.
@aralroca/gui-agent)GuiAgentOptions.domTools forwards DomToolsOptions (root, maxNodes, allowNavigation, onTarget) to the per-run DOM tools. Just before each click/fill/select, the resolved live element is re-emitted on onStep as a tool-target step (carrying the originating ToolCall) — it's what powers the visualizer's glow.
@aralroca/gui-agent/ui)Show users what the agent is doing while it does it: a status chip per tool call ("Clicking" with a spinner → ✓, ✗ on error, blocked when denied), a "Thinking…" indicator between turns, and an animated gradient glow ring around the DOM element being acted on — while the rest of the page blurs behind a backdrop veil so the target stands out. Zero dependencies, rendered in shadow DOM so styles never leak.
import { GuiAgent } from "@aralroca/gui-agent";
import { createAgentVisualizer } from "@aralroca/gui-agent/ui";
const viz = createAgentVisualizer({
container: document.querySelector("#agent-steps")!, // where the chips go
});
// `bind` composes the visualizer into your agent options (it never replaces
// your own `onStep` — it chains it).
const agent = new GuiAgent(viz.bind({ llm }));
await agent.run("Invite [email protected] as admin");
Everything is configurable:
createAgentVisualizer({
chips: true, // action chip list
highlight: true, // glow ring on the target element
cursor: false, // simulated mouse pointer travelling to each target
showThinking: true, // "Thinking…" indicator between LLM turns
locateButton: true, // ◎ button on chips to re-flash the target
glowDuration: 1200, // ms the ring holds on the last target before fading
glowDwell: 500, // min ms per target when several highlights queue up
backdrop: { // blur/dim the page around the target (or false)
blur: 3, // px
exclude: ["chat-panel"], // element ids (or Elements) to keep sharp
},
labels: { // per-tool chip labels (string or fn)
click: "Clicking",
invite_member: (call) => `Inviting ${call.arguments.email}`,
},
theme: { // maps to --gua-* CSS custom properties
accent: "#2563eb",
chipBg: "#f4f4f5",
chipBorder: "#e4e4e7",
chipText: "#3f3f46",
font: "system-ui, sans-serif",
glowColors: ["#7c8cf8", "#f0a6c8", "#7ee0c3"],
},
});
The glow follows automatically for the DOM-fallback tools (click, fill, select_option). Producer tools act through your own code, so they opt in by calling viz.highlight(target) from their execute — call it once per element you touch and the ring tours them in order (agent actions run faster than human perception, so highlights are queued with a minimum dwell per target).
target can be an element or a CSS selector. Pass a selector when your tool creates its target asynchronously — a React Flow (@xyflow/react) node, a portal, a lazily-rendered row — and the ring waits (up to ~2s) for it to mount instead of no-op'ing on a missing element:
const id = graph.addNode({ label }); // returns synchronously…
viz.highlight(`.react-flow__node[data-id="${id}"]`); // …DOM node mounts a tick later; the ring waits for it
Set cursor: true to add a simulated mouse pointer on top of that tour: an arrow that travels to each target the ring visits and ripples where it lands, so the user sees where an action happened and not just that it did. It rides the same queue as the ring — including viz.highlight() calls from your own tools — follows the target if the page scrolls under it, and teleports without the ripple under prefers-reduced-motion. It is a drawn overlay, not the OS pointer: it does not fire :hover or real pointer events, and it never delays the action it illustrates.
Drag and drop. The DOM fallback's drag tool ({ ref, to }) drags a draggable="true" element onto a drop target; the page outline flags draggable elements and drop targets (droptarget, next to their own role) — elements with aria-dropeffect (e.g. aria-dropeffect="move" on a board column) or an ondrop handler. With cursor: true the pointer lands on what is dragged and carries an inert copy of it (no ids, handlers, frames or live custom elements) to the target, where it lets go — on the part of the target that is on screen, as the drop events are, so a tall column never takes a drop below the fold. Your own tools show the same with viz.drag(from, to) and perform it with dragAndDrop(from, to, VISIBLE_DRAG):
defineTool({
name: "move_card",
description: "Move a card to another column.",
inputSchema: { type: "object", properties: { card: { type: "string" }, column: { type: "string" } }, required: ["card", "column"] },
execute: async ({ card, column }) => {
const from = document.getElementById(String(card))!;
const to = document.querySelector(`[aria-label="${column}"]`)!;
viz.drag(from, to);
const { dropped } = await dragAndDrop(from, to, VISIBLE_DRAG);
return dropped ? `Moved to ${column}.` : `${column} doesn't take that card.`;
},
});
Known limitation: elements in the top layer (<dialog showModal>, fullscreen) paint above the overlay.
Implement the Llm interface — one async function, one turn:
import type { Llm } from "@aralroca/gui-agent";
const llm: Llm = async ({ messages, tools, signal }) => {
// call your model with `messages` + `tools`; return one turn
return { text: "...", toolCalls: [{ id, name, arguments }] };
};
@aralroca/gui-agent/ai-sdk)import { createAiSdkLlm, createRemoteLlm } from "@aralroca/gui-agent/ai-sdk";
// Run the model in-process (client key, demo, or server agent):
const llm = createAiSdkLlm({ model: "anthropic/claude-opus-4-8" });
// …or keep the model server-side and execute tools in the browser:
const llm = createRemoteLlm({ api: "/api/chat" });
// endpoint receives { messages, tools } and returns { text?, toolCalls? }
@aralroca/gui-agent/react)import { useTool, GuiAgentProvider, useGuiAgent, AgentSteps } from "@aralroca/gui-agent/react";
function UsersPage() {
// Registered while mounted; auto-unregistered (AbortSignal) on unmount.
useTool({
name: "search_users",
description: "Search users by name or id",
inputSchema: { type: "object", properties: { query: { type: "string" } }, required: ["query"] },
annotations: { readOnlyHint: true },
execute: ({ query }) => store.search(query),
});
return /* … */;
}
function Chat() {
const { run, running, steps, visualizer } = useGuiAgent();
// wire to your own chat UI; <AgentSteps /> renders the visualizer's chips
}
// Enable the visualizer on the provider (true, or AgentVisualizerOptions):
<GuiAgentProvider llm={llm} visualizer>
<Chat />
<AgentSteps />
</GuiAgentProvider>
WebMCP tools run with the user's existing session/cookies, so a tool can do real, privileged things. gui-agent gives you a confirm gate: any tool without annotations.readOnlyHint is routed through your confirm(call, tool) callback before it runs — the natural place to plug in a human-in-the-loop approval UI. Mark genuinely read-only tools with readOnlyHint: true so they don't prompt. See the WebMCP spec's security considerations.
npm run demo # opens a mini "console" you can drive in natural language
Try: "invite [email protected] as admin", "search Kenji", "change my display name to Neo" (the last one uses the DOM fallback — nothing is exposed for it), "move Fix login redirect to Done" on the Board tab (a plain HTML5 drag and drop kanban: the agent drags the card with the DOM fallback's drag tool and the pointer carries it to its column), or "build a workflow" on the Workflows tab — a React Flow canvas where the glow follows each node as the agent adds it (selector highlighting, since the nodes mount asynchronously). The demo ships with the visualizer enabled by default, so you'll see the chips, the glow tour, and the backdrop veil exactly as in the GIF above.
npm install
npm test # vitest + jsdom
npm run build # tsup → ESM + .d.ts for all entry points
npm run typecheck
Issues and pull requests are welcome — open one for bugs, ideas, or questions.
gui-agent is MIT-licensed and maintained in the open. If it saves you time:
MIT © Aral Roca
An open-source WebMCP-powered in-page GUI agent. Let a natural-language agent live inside your web app and drive its UI — no script injection, no browser extension, no headless browser.
{ "mcpServers": { "aralroca-gui-agent": { "command": "npx", "args": [ "-y", "@aralroca/[email protected]" ] } }}도구
아직 도구 목록이 없습니다. 원격 서버는 첫 상태 확인 후에 도구 목록이 생깁니다.
FAQ
이 페이지에서 Cursor, Claude Desktop, Claude Code 또는 VS Code용 stdio 설정을 복사합니다. PlugMCP 프록시가 아니라 게시자의 설치 방식을 사용합니다.
이 레코드에는 필수 환경 변수가 등록되어 있지 않습니다.
@aralroca/gui Agent은(는) stdio를 통해 로컬에서 실행됩니다.
마지막 확인: 2026-11-17. An open-source WebMCP-powered in-page GUI agent. Let a natural-language agent live inside your web app and drive its UI — no script injection, no browser extension, no headless browser.