Umbriel

Umbriel · A set of hands for your AI agent — and Playwright for the Windows desktop

Drive any Windows app through five layers — fall back to OCR and pixel-matching when there's no other way in, see and manage windows even when they're hidden, use cursor-free semantic or posted input where Windows permits it, target controls by name and role, and reach past the GUI to drive the OS itself: processes, services, the registry, scheduled tasks, the firewall, network, power, displays, disks, environment, and the event log. Built for ChatGPT/Codex, Claude, and any AI agent that speaks MCP.

What is it?

Umbriel drives the Windows desktop the way a person would — through whatever channel actually works for the app in front of it:

1. Pixels & OCR — when an app exposes no tree at all (canvas, custom-draw, games), fall back to full-screen capture, template matching, and text recognition.
2. Semantic control — read the UI Automation tree an app exposes and target controls by name and role, not coordinates. Survives the DPI, layout, and theme changes that shatter pixel scripts.
3. Sight & window control — capture the live pixels of any window (even fully GPU-composited or occluded), inspect raw HWND hierarchies, and move, raise, or size windows.
4. Input routing — semantic UIA and posted-message input for background-capable controls, with a verified SendInput fallback only when an HWND-less editor or real pointer gesture actually needs the foreground.
5. Drive the OS, not just GUIs — when the task is the machine, act on it directly: read and write the registry (values and keys), list / kill / suspend / reprioritize processes, query / start / stop / configure services, create / delete / enumerate scheduled tasks, read the firewall rules, enumerate network adapters and live connections, drive session power (lock / restart / sleep / hibernate), list volumes and free space, get and set environment variables, read the event log, and read or change the display mode — all native, no PowerShell or shelling out.

Underneath it's a few kilobytes of TypeScript on Bun's built-in FFI — no Appium server, no .NET, no node-gyp, no prebuilt binaries.

Install

bun add umbriel

That's the entire install story. No build step, nothing to compile.

Built for AI agents

This is what Umbriel is for. Register the published server with Codex (ChatGPT desktop, CLI, and the IDE share this configuration):

codex mcp add umbriel -- bunx umbriel

Or register it with Claude:

claude mcp add umbriel -- bunx umbriel

Any MCP-speaking agent can then ground itself in the accessibility tree in ~15 ms per step and use semantic, posted-message, synthetic-input, or pixel fallback as the target requires. Most classic controls remain driveable backgrounded; a no-own-HWND Chromium/WPF/WinUI editor needs a brief unlocked foreground lease for real input. Umbriel verifies that destination before injecting, can verify the resulting editor state, and restores the previous foreground window by default.

A screenshot agent burns image tokens on every step — then burns them again to screenshot its own result, because it can't read state out of pixels. Umbriel maps the whole window to plain text you cache once and reuse across steps, and reads results straight back from that map. Same grounding Microsoft's UFO2 and the OSWorld-Human benchmark recommend (structure first, vision second) — without the bill.

umbriel.tree(app, { agentProfile: true });
// → one cached round-trip → { role, name, automationId, bounds, children }

⚡ Maps a window in ~13 ms · tokens are text you cache once and reuse, not an image re-bought every step · up to ~2000× faster grounding than a 3–26 s screenshot pipeline

Umbriel figures measured on Windows 11, Bun 1.4 (14.3 ms · ~3k tokens on Calculator; single property read ~58 µs) — reproduce with bun run example/benchmark.ts. Screenshot-pipeline figures are published third-party results (OSWorld-Human; computer-use image-token costs).

The MCP server exposes 99 snapshot-first tools (76 under the default safe profile; 41 under readonly; the 23 os/fs tools need full or UMBRIEL_OS=1) behind a deployer policy you control:

UMBRIEL_PROFILE=readonly   # observe only
UMBRIEL_PROFILE=safe       # default — observe + cursor-free control + window management
UMBRIEL_PROFILE=full       # everything, including launch/run/file tools

desktop_snapshot returns a ref-keyed view — Button "Five" [ref=e49#3] — and every action replies with the smallest faithful update: a compact delta when little changed, a pruned snapshot when more did. The model re-grounds without drowning in tokens.

For chronological chats and logs, desktop_snapshot { root: "Messages in general", tail: 20, maxDepth: 2 } returns only the newest direct children. This avoids paying for hundreds of old loaded rows before reaching the messages that need a response.

When the window handle is already known, the same scope can be applied on the first call: attach { hWnd: "0x123", root: "Messages in general", tail: 20, maxDepth: 2 }. Use maxDepth/maxNodes without root to bound any other large initial tree.

For controlled web editors such as Discord's Slate composer, use the MCP type tool with method: "paste", verify: true, and submit: true (plus clear: true only after confirming an existing draft may be replaced). UIA set_value can mutate the exposed DOM value without updating the application's internal editor model, so it is not a reliable background-send path. An ambiguous submission is reported as unverified; never retry it blindly.

Use it from your code

Umbriel is also a first-class library for E2E tests and desktop automation — find by name → wait → act → assert, just like getByRole on the web:

import { ControlType, umbriel } from 'umbriel';

const app  = await umbriel.launch(['notepad.exe'], { className: 'Notepad' });
const edit = await app.waitFor({ controlType: ControlType.Document });

edit.focus().type('nothing native compiles, and it just works');
console.log(edit.text()); // → nothing native compiles, and it just works

// Drive Calculator to 5 + 3 = 8 — by name, not pixels:
const calc = await umbriel.launch(['cmd', '/c', 'start', 'calc'], { title: 'Calculator' });

for (const name of ['Five', 'Plus', 'Three', 'Equals'])
  calc.find({ controlType: ControlType.Button, name })?.invoke();

console.log(calc.find({ automationId: 'CalculatorResults' })?.name); // → "Display is 8"

Why Umbriel?

The Windows desktop-automation corner of npm is a minefield of native-addon build failures, paywalls, and abandoned daemons. There has been no zero-install, typed, in-process Windows-automation client for Node or Bun — until this.

Because Umbriel is plain TypeScript over Bun's own FFI, it:

• Can't be paywalled — there's no compiled binary to gate behind a subscription registry.
• Has no build step — no ABI matrix, no MSVC or Python toolchain, no node-gyp.
• Talks to Windows in-process — no 127.0.0.1:4723 round-trip, no Appium daemon, no Developer Mode, no WinAppDriver.

Highlights

Requirements

• Semantic first, pixels where there's no structure — custom-draw, games, and WebGL surfaces fall back to the built-in pixel + OCR layer; everything with a tree gets exact semantic targeting.
• Synthetic typing and real clicks need an unlocked desktop — invoke, reads, and setValue work even on a locked session, so prefer them.
• Minimized driving is for classic Win32 windows — invoke / setValue / toggle keep working on a minimized or fully backgrounded classic Win32/HWND window, but a UWP/WinUI store app suspends its UI thread and accessibility tree when minimized (the tree reads empty and posted actions don't land until it's restored), so restore it first.
• Windows 10 or 11, Bun ≥ 1.1 — Windows-only and Bun-only, the owned trade-off for zero dependencies and in-process speed.

Going deeper

The complete API surface lives in AI.md — written so thoroughly that an agent (or a developer) never needs to read the source. Runnable demos are in example/.

MIT Licensed · Built on bun:ffi with zero native dependencies