Browser Pilot
Agent infrastructure for real browser tabs.
Observe. Execute. Verify. Trace. — No multimodal model required.
Features · Comparison · Demos · Quick Start · Tools · Architecture · Security
Features
Browser Pilot works with the tabs and authenticated sessions already open in Chrome or Edge. No headless browser, no separate profile, no multimodal model required — just structured control of real tabs.
How It Differs
Real Workflows
Each GIF shows one complete browser task: the goal, the visible action, and the verified result. Captured from an isolated Edge session using only public Browser Pilot tools.
Create and verify a support case
Find and open an overdue invoice
Sync inventory and verify the result
Quick Start
Step 1 — Install the extension
npx --yes browser-pilot-mcp@latest install
The installer copies the packaged extension to ~/.browser-pilot/extension and opens the matching Chrome or Edge extensions page. On first install, enable Developer mode, choose Load unpacked, and select the directory printed by the command.
This confirmation cannot be bypassed for an unpacked extension — it is a browser security requirement.
Use --browser edge or --browser chrome when both browsers are installed. After an npm upgrade, run the installer again and click Reload.
Run npx browser-pilot-mcp status at any time to check the install, the daemon, and the extension link; it prints the next step when something is off.
Step 2 — Configure your MCP client
[mcp_servers.browser-pilot]
command = "npx"
args = ["--yes", "--package", "browser-pilot-mcp@latest", "browser-pilot-mcp"]
On the first tool call, the MCP process starts or reuses the local daemon automatically. Artifact resources are scoped to the MCP client's first filesystem root; clients without roots fall back to BROWSER_PILOT_PROJECT_ROOT, then the process working directory.
git clone https://github.com/wklwkl115/browser-pilot.git
cd browser-pilot
npm ci
npm run build
npm run build:bridge
npm run mcp -- install
Then point the MCP client at dist/src/apps/mcp/bin.js with command = "node".
Tools
Browser Pilot exposes six composable MCP tools:
The MCP tools/list response is the public syntax authority. src/commands/commandCatalog.ts owns the public tool list; each *Command.ts module owns its schema and handler. browser_command names its core commands inline (CDP, trusted input, waits, network recorder, frames, HTML, transfers); advanced families (hook.*, intercept.*, ws.*, new-document scripts) stay public but are documented only through the browser-pilot://native-commands index and browser-pilot://native-command/<cmd> resources.
browser_observe accepts mode: "auto" | "full" | "diff" and visual: "auto" | "always" | "never". Its inline result contains decision-facing data; irreducible overflow is exposed through typed observation resources. browser_tabs returns { "tabs": [...] } (plus an effect for navigate); browser_screenshot returns capture metadata plus the image resource.
New to the vocabulary (refs, entities, frontier, collections, causal, verification)? Read docs/concepts.md first; it explains every term that appears in tool output and walks through the browser_observe result key by key.
Agent Workflow
For focused observation, use browser_observe with view: { focus: { query: "INV-2048" }, intent: "locate" }, then an observed ref and preferred field labels. Task views preserve object context, ambiguity and captured-scope limits without acting on the page. See task views.
observe -> choose a bp-ref -> execute or command -> verify -> collect evidence
- Start with the active tab. Omit
targetRefto use the selected tab. List tabs only to disambiguate; create, switch, or close only when the task requires it. - Observe when needed. Call
browser_observeonly when the task needs page understanding. Itsbp-refvalues route later actions back to the owning tab automatically. - Act through the right tool. Use
browser_tabs navigateto load a URL,browser_commandinput.reffor trusted input on an observed control (click,type,check,select,focus,hover), andbrowser_executefor page JavaScript. Combine deterministic same-page JavaScript in one call. - Wait deliberately. When the next step depends on the page settling, use
wait.loadState,wait.selector,wait.networkIdle, orwait.navigationthroughbrowser_commandinstead of polling. - Separate assertions from business outcomes.
expectverifies only the supplied assertion. Declarebusiness.success/business.failurefor business evidence; otherwise the outcome staysunknown. After uncertain execution, inspectbrowser_operationwith the returnedoperationIdinstead of resubmitting. See operation outcomes.
browser_execute provides browserPilot.refs, resolve(ref), box(ref), and setValue(target, value). Writes are serialized by target and use the extension/CDP fallback path automatically.
Successful browser_execute and browser_command calls return { "result": ..., "effect"?: ..., "verification"?: ... }; writes may add effect and verification. An expect can be a JavaScript truth expression or a structured ref/state postcondition such as { "ref": "bp-ref://control/...", "state": { "pressed": true } }. Structured verification reads the same ref before and after dispatch, fuses DOM and targeted accessibility state, and returns a target-scoped diff.
Writes also return operationId, execution, and business, independently of assertion verification. Declarative text, value, URL, captured-response and combination conditions are supported; verificationWaitMs provides a bounded observation budget. browser_operation continues stored declarative checks without executing the original write. A verified assertion may describe a failure UI, and operation IDs do not provide website-level exactly-once guarantees.
Trusted input uses command: { cmd: "input.ref", ref: "bp-ref://...", action: "type", text: "...", clear: true }; check takes checked, select takes value, label, or index. Raw CDP uses command: { cmd: "cdp", method: "Domain.method", params: {...} }. Targeting remains at the tool-level targetRef; runtime session, physical target, timeout, attach, and cleanup state are not public contract fields. Waits and transfers get a longer fixed budget than one-shot commands.
Architecture
AI agent
| MCP stdio
v
MCP process --local IPC--> Node daemon
| WebSocket bridge
v
offscreen transport
|
v
MV3 service worker
| Chrome APIs / CDP
v
Chrome or Edge tab
Source is organized by responsibility: src/apps contains the MCP server and daemon; src/bridge owns transport and extension code; src/commands owns public tool schemas and orchestration; src/browser-command-runtime prepares command execution; src/browser-page-runtime evaluates page scripts; src/browser-runtime adapts browser I/O; src/kernels stays pure. Page scanning lives in src/scan and capture-src, with observation assembly under src/commands/observe.
browser-pilot/
├── src/
│ ├── apps/ # MCP server + daemon entry points
│ │ ├── daemon/ # Local Node daemon
│ │ └── mcp/ # MCP stdio server
│ ├── bridge/ # Transport + extension
│ │ ├── extension/ # MV3 service worker, content scripts
│ │ ├── server/ # Bridge HTTP + WebSocket server
│ │ └── protocol/ # Native command schema
│ ├── commands/ # Public tool schemas + orchestration
│ │ └── observe/ # Page observation assembly
│ ├── kernels/ # Pure logic (no I/O)
│ │ ├── abml/ # Accessible model layer
│ │ ├── evidence/ # Evidence distillation
│ │ ├── refs/ # Reference resolution
│ │ └── session/ # Session lifecycle
│ ├── scan/ # Page scanning + noise rules
│ └── utils/ # Shared utilities
├── bridge/ # Bridge config + generated extension bundle (untracked)
├── docs/assets/ # Diagrams + demo GIFs
├── scripts/ # Build + dev scripts
└── tests/ # Test suites, one directory per layer
├── kernels/ # Pure kernel logic
├── runtime/ # Browser/command runtime helpers
├── bridge/ # Bridge server
├── extension/ # MV3 service worker + page scripts (chrome stubs)
├── commands/, observe/ # Tool orchestration + observation assembly
└── daemon/, mcp/ # Daemon control + MCP server
Security Model
- The WebSocket bridge accepts upgrades only from the configured Browser Pilot extension origin; command dispatch rejects stale extension builds.
- Page content is always untrusted.
- Browser Pilot does not remove page security headers or suppress page dialogs.
- Tool results, network captures, and artifacts are returned verbatim; Browser Pilot does not redact page or network content. Password field values are the one exception: they never enter observations. Treat
.browser-pilot/artifacts/as raw local evidence. - Report vulnerabilities through GitHub private vulnerability reporting. If unavailable, open a minimal public issue requesting a private contact path — never include secrets or private evidence.
Development
Requirements: Node.js 22+; Chrome or Edge for browser integration checks. mise is optional; mise.toml only pins Node and forwards to the npm scripts below.
npm run verify # Canonical gate: generated files + format + typecheck + lint + test + bridge build
npm test # Deterministic Node tests (tests/<layer>/ mirrors src/)
npm run smoke:browser # Browser integration smoke test (headless Chrome/Edge; also runs in CI)
npm run eval:browser # Repeatable task evaluation; JSON success/latency/output-size report
npx prettier --check <files> # Check changed files; use --write to format them
npx browser-pilot-mcp status # Diagnose a local install (source checkout: npm run mcp -- status)
Runtime source lives under src/ and capture-src/; dist/ and bridge/browser_pilot_bridge/ are generated outputs and are not tracked. The bridge host and port range are owned by bridge/browser_bridge_config.json — run npm run sync:config after changing it. Formatting is owned by Prettier (.prettierrc.json); the gate fails on unformatted files.
See Browser task evaluation for scenarios, metrics, and interpretation limits, and Reliability and recovery for verification and installation failure semantics.
Contributing
Contributions are welcome. Discuss unresolved requirements for major architecture, public-interface, or compatibility changes first. An existing issue or explicit task discussion is sufficient; routine fixes and documentation updates do not require a new issue.
- Fork the repository
- Create a feature branch (for example,
git checkout -b codex/amazing-feature) - Commit your changes (
git commit -m 'feat: add amazing feature') - Push to the branch (
git push origin codex/amazing-feature) - Open a Pull Request
Choose local checks by the change's impact using the repository guidelines. Documentation-only work needs content, reference, and formatting checks; behavior changes need focused regression coverage, with real-browser checks for affected browser behavior. The observe and mcp test scopes include multiple foundation layers; use node --import tsx --test tests/<layer>/<name>.test.ts for a single test file.
CI runs for pull requests and pushes to main, avoiding duplicate feature-branch push runs. Documentation-only changes take the lightweight route; other or unknown changes run full verification and browser checks on Linux and Windows. The ci result check summarizes both routes. Release checks remain mandatory. Report completed checks and any blockers; unchanged passing checks need not be rerun for every local commit. Local task completion, committing, and publishing are separate operations governed by the requested scope.
License
Licensed under Apache-2.0. See THIRD_PARTY_NOTICES for embedded dependencies.
Made with care for AI agents that need to actually get things done.
