🖥️ Computer use for all agents
A local remote-control system for AI agents: watch your computer's screen live and send mouse/keyboard commands to it. Everything runs on your own machine — no data ever leaves it, no cloud middleman.
Power your agent
การติดตั้งจากผู้เผยแพร่ ไม่ใช่เอนด์พอยต์ของ PlugMCP
URL แบบรีโมตหรือคำสั่งบนเครื่องของคุณ? การติดตั้งสองแบบต่างกันอย่างไร
รันบน PlugMCP
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
เปิดจากผู้เผยแพร่
A local remote-control system for AI agents: watch your computer's screen live and send mouse/keyboard commands to it. Everything runs on your own machine — no data ever leaves it, no cloud middleman.
Safety-gated computer use for AI agents: screen OCR and vision, mouse, keyboard, window control.
Read AGENT_GUIDE.md first — it is written for you: endpoints, safety gates and working recipes. This README is the human-facing reference.
First ask the server what it is — GET /api/capabilities → platform.
Windows: follow the whole guide as written. Linux: check
echo $XDG_SESSION_TYPE — X11 is supported (guide →
Linux (X11) Usage); Wayland is
unsupported: the server refuses to start, HTTP 501
UNSUPPORTED_DISPLAY_SERVER (switch to an X11 session). Full routing lives
in AGENT_GUIDE.md → Find Your Environment First.
AI agents today can write code and call APIs — but they can't see or touch your desktop. Screen Control gives any agent general-purpose computer use over a clean, safety-gated HTTP/MCP interface:
One process, zero configuration, works with any language that can speak HTTP — or natively through MCP in Claude Desktop, Cursor, VS Code and cloud agents.
Screen Control is the perception and actuation layer — the eyes and hands. The effective speed and capability of any agent using it are bounded by that agent itself and by the environment it runs in:
In practice this means: the same repo makes a fast reasoning model fast and capable, and makes a slow model slow — the toolchain is not the bottleneck. Real-time or action-heavy tasks need an agent with fast inference and tight tool-loop latency; slower agents should prefer deliberate, verification-heavy tasks.
┌─────────────────────────────────────────────────────────┐
│ Browser (Web UI) │
│ ┌──────────┐ ┌──────────┐ ┌────────────────────────┐ │
│ │ Live │ │ Control │ │ Windows / Game Mode │ │
│ │ View │ │ Panel │ │ Panel │ │
│ └────┬─────┘ └────┬─────┘ └───────────┬────────────┘ │
│ │ │ │ │
└───────┼──────────────┼─────────────────────┼──────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────┐
│ HTTP API (Flask) │
│ 127.0.0.1:8745 │
│ │
│ /api/screenshot /api/mouse /api/key │
│ /api/vision/* /api/ocr /api/window │
│ /api/game /api/held /api/release_all │
│ /api/windows /api/desktops /api/desktop │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Auth Layer │ │ Watchdog │ │ OCR Engine │ │
│ │ (token) │ │ (30s auto) │ │ (RapidOCR) │ │
│ └─────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────┐
│ control.py (Core) │
│ │
│ Screen: mss (fast capture), PIL (processing) │
│ Mouse: pyautogui (absolute), SendInput (relative) │
│ Keyboard: pyautogui + SendInput+KEYEVENTF_UNICODE │
│ Windows: Win32 API (EnumWindows, SetForegroundWindow) │
│ Background: PrintWindow (capture), PostMessage (input) │
│ Virtual Desktops: pyvda │
│ Game Mode: ClipCursor + MOUSE_MOVE_RELATIVE │
└─────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
│ backends/ (pluggable) │
│ WindowsBackend │ LinuxBackend │ MacOSBackend│
│ (full) │ (X11) │ (stub) │
└──────────────────────────────────────────────┘
Per-Monitor DPI awareness. control.py calls
SetProcessDpiAwarenessContext(PER_MONITOR_AWARE_V2) at import time —
before the pyautogui import, because pyautogui touches coordinate APIs
during import and would otherwise lock the process to the interpreter
manifest's default (system-aware). With PMv2 active, every coordinate in the
system is a physical pixel end to end: mss capture, OCR bounding boxes,
pyautogui/SendInput clicks, ClipCursor. On High-DPI displays (125%/150%
scaling) nothing drifts between what OCR reports and where the mouse clicks.
Lock architecture. The server uses two independent locks instead of one global lock:
A slow OCR (3–5 s on a busy screen) no longer freezes concurrent screenshot or vision reads — reads queue behind reads, inputs behind inputs.
This system is designed for a live perceive-act loop, not pre-written command chains:
This is enforced by the expect_hwnd guard: typing is refused (409) if
the foreground window doesn't match the target.
cd screen-control
pip install -r requirements.txt
Note: The OCR package is large and may take a while to install. If it fails, everything else still works — only the OCR feature is unavailable.
Linux is supported with limits (ROADMAP Phase 5) — tested on Linux Mint with Cinnamon in an X11 session; other X11 desktop environments are expected to work. Windows remains the reference backend.
echo $XDG_SESSION_TYPE # must print: x11
sudo apt install -y xdotool wmctrl imagemagick python3-venv python3-tk python3-dev
xdotool drives input/focus, wmctrl window management, ImageMagick's
import captures window pixels (optional — an mss region fallback exists),
python3-venv creates the virtualenv and python3-dev lets evdev
(game mode) compile from source.
cd screen-control
./install-linux.sh
The script is idempotent and checks, in order: Wayland (exits with a clear
message), DISPLAY, the apt packages, the .venv virtualenv,
pip install -r requirements.txt, and finally reports whether game mode can
work.
Game mode writes to /dev/uinput. Verify access with:
[ -w /dev/uinput ] && echo "game mode: OK" || echo "game mode: unavailable"
If it is unavailable, either remediation works (spec §5.4):
sudo usermod -aG input $USER # then log out and back in
# — or a udev rule:
echo 'KERNEL=="uinput", MODE="0660", GROUP="input"' | sudo tee /etc/udev/rules.d/99-uinput.rules
sudo udevadm control --reload && sudo udevadm trigger
Everything except game mode works without this access.
./start-server.sh
# → same API as Windows: http://127.0.0.1:8745 (X-Auth-Token from .token)
Agent note: OCR is the slowest read (≈5–6 s full-screen measured). Prefer
/api/vision/frameor/api/streamfor repeated reads, use region OCR, and batch reads between actions — see AGENT_GUIDE.md → Linux (X11) Usage.
# 1. Start the server
cd screen-control
python server.py
# 2. Open in browser
# http://127.0.0.1:8745
# 3. Or control via API
TOKEN=$(cat .token)
curl -H "X-Auth-Token: $TOKEN" http://127.0.0.1:8745/api/screenshot -o screen.jpg
Returns the active backend name and what it can do. Agents should call this first (see ROADMAP.md for the multi-platform plan).
GET /api/capabilities
→ {"ok": true, "backend": "windows",
"capabilities": {"screen_capture": true, "game_mode": true, ...}}
Feature values: true (supported), false (absent), null (unknown —
stub backend), "optional" (depends on an optional dependency).
Linux X11 is implemented and supported with limits — setup, requirements and the full limitations list are in Linux (X11). Wayland fails fast with
UNSUPPORTED_DISPLAY_SERVER(501), and macOS remains a fail-closed stub: every operation returnsBACKEND_UNAVAILABLE(501) until ROADMAP Phase 7. Windows is the reference backend.† Wayland column is hypothetical capability planning only — the server never starts on Wayland (see the note above), so no Wayland cell is currently reachable.
Every request must include the X-Auth-Token header. The token is
generated on each server start and written to .token.
TOKEN=$(cat .token)
Token bootstrap (for the bundled web UI):
GET /token
→ {"ok": true, "token": "abc123..."}
The
/tokenendpoint is safe: Same-Origin Policy prevents foreign pages from reading it.
GET /api/screenshotReturns a JPEG screenshot.
curl -H "X-Auth-Token: $TOKEN" -o screen.jpg http://127.0.0.1:8745/api/screenshot
curl -H "X-Auth-Token: $TOKEN" "http://127.0.0.1:8745/api/screenshot?region=0,0,800,600"
GET /api/infoReturns screen dimensions and system state.
{"ok": true, "width": 1920, "height": 1080, "ocr_available": true,
"failsafe": true, "game_mode": false}
POST /api/mouse# Click at center of screen
curl -X POST http://127.0.0.1:8745/api/mouse -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"action":"click","x":960,"y":540,"button":"left"}'
# Right-click
curl -X POST http://127.0.0.1:8745/api/mouse -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"action":"click","button":"right","x":960,"y":540}'
# Scroll down
curl -X POST http://127.0.0.1:8745/api/mouse -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"action":"scroll","clicks":-3}'
POST /api/key# Press Enter
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"press","key":"enter"}'
# Ctrl+C
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"hotkey","keys":["ctrl","c"]}'
# Type text (Turkish characters supported)
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"type","text":"Merhaba dünya"}'
# Hold W key down (for walking in games)
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"down","key":"w"}'
POST /api/ocrConverts on-screen text to machine-readable format.
{
"ok": true,
"text": "Hello World\nFile Edit View",
"lines": ["Hello World", "File Edit View"],
"items": [
{"text": "Hello World", "x": 960, "y": 40},
{"text": "File Edit View", "x": 100, "y": 15}
]
}
# Full screen OCR
curl -X POST http://127.0.0.1:8745/api/ocr -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{}'
# Region-only (faster, ~10x for small regions)
curl -X POST http://127.0.0.1:8745/api/ocr -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"region":[0,0,800,100]}'
Three endpoints for models that can consume images:
GET /api/vision/frame# Half-size greyscale frame as base64 (for text-only models)
curl "http://127.0.0.1:8745/api/vision/frame?scale=0.5&gray=1&format=base64" \
-H "X-Auth-Token: $TOKEN"
GET /api/streamMJPEG live stream. Drop into <img src> or consume frame-by-frame.
POST /api/vision/diffText-based motion detection — no vision model required.
{
"ok": true,
"changed": true,
"changed_pct": 12.5,
"bbox": [100, 200, 400, 350],
"tiles": [
{"row": 2, "col": 4, "pct": 35.2, "center": [1000, 390]}
]
}
GET /api/windowsList all visible windows.
{
"ok": true,
"windows": [
{
"hwnd": 123456,
"title": "My Application",
"process": "app.exe",
"pid": 7890,
"focused": true,
"rect": [0, 0, 1920, 1080],
"desktop": 1
}
]
}
POST /api/window# Focus a window
curl -X POST http://127.0.0.1:8745/api/window -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"hwnd":12345,"action":"focus"}'
# Safe close (with title verification)
curl -X POST http://127.0.0.1:8745/api/window -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"action":"close","hwnd":12345,"expect_title":"Notepad"}'
# Kill process
curl -X POST http://127.0.0.1:8745/api/window -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"kill","hwnd":12345,"pid":7890}'
POST /api/launchSpawn an application, open a URL, or open a local file. Exactly one of
app | url | file. Never runs through a shell: app is a plain command
name resolved on PATH ([A-Za-z0-9._+-]+), url/file go through
xdg-open.
{"ok": true, "pid": 4321, "window": {"hwnd": ..., "title": "..."}}
With expect_title, window is the match or null (spawn succeeded, no
window yet) — the HTTP status stays 200 either way.
curl -X POST http://127.0.0.1:8745/api/launch -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"app":"firefox","expect_title":"Mozilla Firefox","timeout":15}'
GET /api/window/waitBlock until a window whose title contains title (case-insensitive
substring) appears, then return it. timeout default 10 s, max 30 s;
200 with the window, or 408 OPERATION_TIMEOUT.
curl -H "X-Auth-Token: $TOKEN" \
"http://127.0.0.1:8745/api/window/wait?title=Notepad&timeout=10"
Read and control windows without stealing focus — the user keeps working on their main desktop.
Linux (X11) note: background control there is focus-first — the target window is focused, then input is sent globally (focus stealing). Only background capture is truly focus-free. See Known limitations.
GET /api/window/captureCapture a window via PrintWindow (works even on another virtual desktop).
# Capture window as PNG
curl "http://127.0.0.1:8745/api/window/capture?hwnd=12345" \
-H "X-Auth-Token: $TOKEN" -o window.png
# Capture + OCR in one call
curl "http://127.0.0.1:8745/api/window/capture?hwnd=12345&ocr=1" \
-H "X-Auth-Token: $TOKEN"
POST /api/window/postSend input to a window, choosing the delivery path automatically.
Optional mode parameter controls routing:
# Type into a background Notepad
curl -X POST http://127.0.0.1:8745/api/window/post -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"hwnd":12345,"action":"type","text":"Hello from background!"}'
Routing rules (mode=auto):
postmessage — classic Win32 app: background PostMessage, no focus change.uia — WinUI/UWP/XAML surface (single DirectX canvas, no Win32 child
controls): posted messages are silently swallowed, so the window is focused
and the action is replayed through SendInput (client coords converted to
screen). This is the documented fallback for modern apps.focused — window is already foreground: focused SendInput path.invalid — HTTP 409; not a reachable top-level window.GET /api/window/input-modeClassify how a window receives input before posting to it. Returns one of
focused | postmessage | uia | invalid.
curl "http://127.0.0.1:8745/api/window/input-mode?hwnd=12345" \
-H "X-Auth-Token: $TOKEN"
WinUI note: New Notepad (and other XAML-hosted apps) has no classic child Edit control to post to — the whole UI is one DirectX surface.
input-modereportsuiafor these;/api/window/postthen automatically uses the focused SendInput path./api/window/childrenremains useful for classic apps with real child controls.
GET /api/window/childrenList child controls of a window (class name + title + hwnd).
curl "http://127.0.0.1:8745/api/window/children?hwnd=12345" -H "X-Auth-Token: $TOKEN"
GET /api/desktopsList all virtual desktops.
POST /api/desktopcurl http://127.0.0.1:8745/api/desktops -H "X-Auth-Token: $TOKEN"
curl -X POST http://127.0.0.1:8745/api/desktop -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"switch","number":2}'
# Start game mode
curl -X POST http://127.0.0.1:8745/api/game -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"start","sensitivity":12}'
# Look right
curl -X POST http://127.0.0.1:8745/api/game -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"move","dx":50,"dy":0}'
# Hold W to walk forward
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"down","key":"w"}'
# ... later ...
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"up","key":"w"}'
# Stop game mode
curl -X POST http://127.0.0.1:8745/api/game -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"stop"}'
GET /api/heldReturns currently held keys/buttons and watchdog status.
{
"ok": true,
"keys": ["w", "shift"],
"buttons": ["left"],
"game_mode": true,
"idle_seconds": 5.2,
"watchdog_count": 0,
"last_watchdog": null
}
POST /api/release_allEmergency: release everything (held keys, mouse buttons, game-mode cursor lock).
curl -X POST http://127.0.0.1:8745/api/release_all -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{}'
A dedicated, comprehensive guide for AI agents (LLMs, vision models, automation frameworks) is available in AGENT_GUIDE.md.
A portable SKILL.md pack lives in
skills/computer-use/ — drop it into ~/.claude/skills/ (or any agent's
skills folder) to get setup, the perceive-act loop, safety rules and a full
API reference loaded on demand; the same pack is what we list on skill
marketplaces.
It covers:
expect_hwnd) to prevent wrong-window accidentsModel Context Protocol (MCP) turns this project into a plug-and-play toolbox for any MCP-capable agent: Claude Desktop, Claude Code, Cursor, VS Code Copilot Agent mode, custom cloud agents — no custom glue code, no curl scripts. The agent discovers and calls the tools natively.
MCP agent (cloud or desktop)
│ MCP protocol (stdio or streamable-HTTP)
▼
mcp_server.py ← thin wrapper: tools → HTTP calls, token auto-read
│ REST + X-Auth-Token (localhost only)
▼
server.py ← the single source of truth:
auth, locks, watchdog, focus guard, all safety rules
mcp_server.py adds no new powers — every safety mechanism
(auth token, input/read locks, watchdog, Alt+F4 block, focus guard,
failsafe) stays enforced by server.py.
pip install mcp # optional dependency (see requirements.txt)
python server.py # start the REST server first (it writes .token)
The MCP server auto-reads the token from .token (or the
SCREEN_CONTROL_TOKEN env var) — zero configuration.
Claude Desktop — claude_desktop_config.json:
{
"mcpServers": {
"screen-control": {
"command": "python",
"args": ["C:/path/to/screen-control/mcp_server.py"]
}
}
}
Claude Code: claude mcp add screen-control -- python C:/path/to/screen-control/mcp_server.py
Cursor / VS Code: add the same entry to their MCP config files.
python mcp_server.py --http --port 8751
# MCP endpoint: http://127.0.0.1:8751/mcp
The HTTP transport is token-protected: every request must carry the
X-Auth-Token header (same token as the REST server). Query-string tokens
(?token=...) are rejected by design — URLs leak into proxy/tunnel
logs, browser history and shared links, and this token grants full desktop
control. Clients that cannot send custom headers should run a local stdio
mcp_server.py instead. Only GET /health is open, for liveness probes.
DNS-rebinding protection is disabled on this transport deliberately —
tunneled requests arrive with a foreign Host header, and the rebinding
threat is already covered by the token guard.
For a cloud agent, expose it through a tunnel:
cloudflared tunnel --url http://127.0.0.1:8751
# → prints a https://<random>.trycloudflare.com URL
Then configure the agent's MCP connection with <tunnel-url>/mcp plus the
token from .token as a header (X-Auth-Token).
For clients that cannot send custom headers (e.g. web connectors that only take an endpoint URL), create a scoped API key — a persistent, optionally time-limited credential — and embed it in the URL path:
# Create a 24-hour scoped key (requires the server to be running)
curl -X POST http://127.0.0.1:8745/api/keys -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"action":"create","name":"spark","expires_in_hours":24}'
# Connector endpoint becomes:
# https://<tunnel-url>/mcp/<scoped-key>
Design guarantees (SC-06):
POST /api/keys
({"action":"revoke","name":"spark"}) — revocation takes effect
immediately on every endpoint.apikeys file stores only SHA-256 hashes, never raw keys⚠️ A tunnel exposes PC control to the internet. Keep the token secret, prefer short-lived tunnels and scoped keys for headerless connectors, and stop the server when not in use.
start-server.bat automates the whole cloud setup and prints everything
your cloud agent needs, ready to paste:
cloudflared.exe if missing (portable, no admin required)/token and /health probes)tunnel.log, and prints the summary: ============================================================
ALL SYSTEMS RUNNING
============================================================
Local REST API : http://127.0.0.1:8745
Local MCP : http://127.0.0.1:8751/mcp
Public MCP URL : https://<random>.trycloudflare.com/mcp
------------------------------------------------------------
PASTE INTO YOUR CLOUD AGENT (MCP connector settings)
------------------------------------------------------------
Endpoint : https://<random>.trycloudflare.com/mcp
Header : X-Auth-Token: <token>
URL form : https://<random>.trycloudflare.com/mcp?token=<token>
(only if the connector cannot send headers)
------------------------------------------------------------
stop-server.bat stops all three (REST, MCP, tunnel) in one go.
Note: This project targets MCP Python SDK 2.x (
MCPServerAPI). With SDK 1.x, replace the import withfrom mcp.server.fastmcp import FastMCP, ImageandMCPServerwithFastMCP.
Even bound to 127.0.0.1, a malicious page in the browser can trigger
non-preflighted requests (text/plain fetch, HTML form POST) to localhost.
The browser blocks the response but not the request — the server
would still execute the command.
Mitigation: Every request requires X-Auth-Token. A foreign page
cannot read this token (Same-Origin Policy), so it cannot authenticate.
Additional layers:
Content-Type: application/json (415 otherwise)Host header are refused with 421 — a
rebinding page that resolves its domain to 127.0.0.1 cannot read
/token or call the API/token and / responses carry Cache-Control: no-store so the
credential is never persisted by browsers or proxiesIn game mode, ClipCursor pins the cursor to a 2×2 box — the classic
pyautogui failsafe (cursor to top-left) does not work.
Mitigations:
Esc / Alt+Tab — real hardware input; this API cannot
block it, and it always worksPOST /api/release_all — instant release of everythingMitigations:
expect_hwnd guard on /api/key — if the foreground window doesn't
match, typing is refused with 409/api/window/post verifies the focus after the
focus switch and before any synthetic input (409 on mismatch) —
input is never replayed into whatever window happens to be foregroundfocus_window() raises on failure instead of silently returningMitigation: Blocked at the API level (403) on every delivery path —
the direct /api/key route, the background /api/window/post route
(PostMessage), and the focused fallback route share one safety policy
(control._assert_allowed):
Alt+F4 — the only banned Alt combo (Alt+Tab, Alt+menu are legitimate)Ctrl+Alt+Del — system security screenShift+Delete style — prevents permanent deletionMitigations:
winlogon.exe, csrss.exe, smss.exe, services.exe, lsass.exe,
svchost.exe, system, registry, dwm.exeexpect_process confirmation: a mismatch aborts the kill with
409 — protects against killing a newly-reused PIDA token-holding but misbehaving client should not be able to exhaust memory or starve the input lock.
Mitigations:
MAX_CONTENT_LENGTH = 1 MB — oversized request bodies are rejected (413)region width/height/area and scale are bounded (400 otherwise)text payloads are capped at 10,000 characters per input callThe server binds to 127.0.0.1 by default. To expose it to the network:
python server.py --host 0.0.0.0 # ⚠️ anyone on the network can control this machine
# 1. Focus the game window
curl -X POST http://127.0.0.1:8745/api/window -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"hwnd":GAME_HWND,"action":"focus"}'
# 2. Start game mode
curl -X POST http://127.0.0.1:8745/api/game -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"start","sensitivity":12}'
# Look right
curl -X POST http://127.0.0.1:8745/api/game -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"move","dx":50,"dy":0}'
# Look down
curl -X POST http://127.0.0.1:8745/api/game -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"move","dx":0,"dy":30}'
# Walk forward (hold W)
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"down","key":"w"}'
# ... walk for a while ...
# Release W
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"up","key":"w"}'
# Place block (right-click)
curl -X POST http://127.0.0.1:8745/api/mouse -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"action":"click","button":"right","x":960,"y":540}'
# Break block (hold left-click)
curl -X POST http://127.0.0.1:8745/api/mouse -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"down","button":"left"}'
# ... after breaking ...
curl -X POST http://127.0.0.1:8745/api/mouse -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"up","button":"left"}'
# Select hotbar slot
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"press","key":"1"}'
# Open inventory
curl -X POST http://127.0.0.1:8745/api/key -H "X-Auth-Token: $TOKEN" \
-H "Content-Type: application/json" -d '{"action":"press","key":"e"}'
If the consuming model can process images, use the vision endpoints directly:
GET /api/vision/frame?scale=0.5&gray=1&quality=70
This returns a single JPEG that the model can analyze for:
Use the diff endpoint for motion detection without vision:
POST /api/vision/diff {"grab":"gray"} → first call: stores frame
POST /api/vision/diff → subsequent calls: returns diff
The response tells you where things changed (tile coordinates) and how much (percentage), which is sufficient for:
pip install rapidocr-onnxruntime
# Find the process using port 8745
netstat -ano | findstr ":8745"
# Kill it
taskkill /PID <pid> /F
The foreground window changed between the focus call and the type call.
Solution: always pass expect_hwnd and verify focus before typing.
The window may have been closed or may be a system window that
EnumWindows doesn't expose. Try:
curl http://127.0.0.1:8745/api/windows -H "X-Auth-Token: $TOKEN"
Use POST /api/release_all or press Esc / Alt+Tab physically.
OCR on a full 1920×1080 screen can take from a few seconds up to ~30 s depending on your CPU and on-screen complexity. Use a region — small crops are typically 10× faster:
{"region": [0, 0, 800, 100]}
The system uses SendInput + KEYEVENTF_UNICODE which is layout-independent.
If characters still don't appear, the target app may not support Unicode
input — try POST /api/window/post with action: "type" instead.
Expected on a Wayland session — the backend fails fast with
UNSUPPORTED_DISPLAY_SERVER (501) instead of half-working. Log out, choose
the X11 session ("Cinnamon on Xorg") at the login screen, then start the
server again. See Linux (X11).
The X11 helper tools are missing. Install them (or re-run the installer):
sudo apt install xdotool wmctrl
screen-control/
├── server.py # Flask HTTP server + all API endpoints
├── core/ # PlatformBackend interface + standardized errors
│ ├── backends.py # Abstract backend + lazy discovery
│ └── errors.py # ApiError envelope + error codes
├── backends/ # OS implementations behind PlatformBackend
│ ├── windows.py # Reference backend (moved from control.py)
│ ├── linux.py # X11 backend: xdotool/wmctrl + mss + uinput (Phase 5)
│ ├── imageops.py # Shared PIL helpers (frame diff, JPEG, scaling)
│ ├── uinput.py # Virtual input device for Linux game mode
│ ├── macos.py # Fail-closed stub (ROADMAP Phase 7)
│ ├── fake.py # In-memory backend for tests
│ └── forbidden.py # Shared blocked-key policy
├── control.py # Compatibility shim re-exporting backends.windows
├── tests/unit/ # Offline unit + integration tests (no real input)
├── mcp_server.py # MCP server (stdio + streamable-HTTP) — thin wrapper over the API
├── sdk/
│ └── screen_control.py # Python SDK client (pip-installable style)
├── index.html # Bundled web UI (live view + control panels)
├── docs/images/ # README assets (demo GIF captured by the API itself)
├── requirements.txt # Python dependencies
├── start-server.bat # One command: REST + MCP + cloud tunnel (Windows)
├── stop-server.bat # Stop all three processes
├── install-linux.sh # One-command Linux setup (apt, venv, uinput check)
├── start-server.sh # Linux launcher (.venv/bin/python server.py)
├── test-security.py # Security + game-mode test suite (34 checks)
├── test-linux.py # Live Linux X11 acceptance suite (7 items)
├── test-game.py # Live game-mechanics test (app launch → draw → safe close)
├── test-endtoend.py # End-to-end test: open Notepad → type → save → verify
├── .github/workflows/ # CI: runs the security suite on every push
├── AGENT_GUIDE.md # AI agent integration guide (separate from this file)
├── README.md # This file
└── .token # Auto-generated auth token (gitignored)
The server must be running:
cd screen-control
python server.py
Tests authentication, blocked key combos, window management, safe close, critical process protection, and game mode — all non-destructive.
cd screen-control
python test-security.py
Expected output:
== Token Authentication ==
✓ Missing token -> 401
✓ Wrong token -> 401
✓ Correct token -> 200
✓ Non-JSON POST -> 415
== Blocked Key Combos ==
✓ Alt+F4 blocked (403)
✓ Win key blocked (403)
✓ Win+D blocked (403)
✓ Delete blocked (403)
== Window List ==
✓ Windows list requires GET
✓ Window list is non-empty — 8 windows
✓ Exactly one focused window
== Safe Close Verification ==
✓ Wrong title aborts close
== Critical Process Protection ==
✓ System process (pid 4) rejected (403)
✓ pid 0 rejected (403)
== Game Mode ==
✓ Game mode started
✓ Relative camera look
✓ Game mode stopped
== Watchdog (dry run) ==
✓ Held state returns ok
✓ Watchdog count reported
========================================
RESULT: 19 passed, 0 failed
Launches a real application (mspaint or notepad), performs hold-to-draw game mechanics, verifies via pixel analysis, then safely closes with "Don't Save" dialog handling.
cd screen-control
python test-game.py
Note: This test launches a real application. It handles cleanup automatically (sends WM_CLOSE and clicks "Don't Save" if a dialog appears).
Seven-item live check of the Linux backend: mouse move/click, typing +
hotkey, screenshot, OCR, window list/focus, game mode (uinput) and the
native-game item (printed as SKIP — no Linux game is installed). Needs a
running server and a real X11 session; every helper window is pid-unique and
only those are ever focused or clicked.
cd screen-control
./start-server.sh & # server must be running
.venv/bin/python test-linux.py # → "RESULT: 6 passed, 0 failed"; item 7 prints SKIP
MIT License. See LICENSE for details.
Built with ❤️ for local automation and AI agent research.
Safety-gated computer use for AI agents: screen OCR and vision, mouse, keyboard, window control.
เครื่องมือ
ยังไม่มีรายการเครื่องมือ เซิร์ฟเวอร์แบบรีโมตจะได้รายการหลังการตรวจสอบสถานะครั้งแรก
FAQ
คัดลอกคอนฟิก stdio สำหรับ Cursor, Claude Desktop, Claude Code หรือ VS Code ได้ในหน้านี้ คอนฟิกนี้ใช้การติดตั้งของผู้เผยแพร่ ไม่ใช่พร็อกซีของ PlugMCP
ระเบียนนี้ไม่ได้ระบุตัวแปรสภาพแวดล้อมที่จำเป็น
Screen Control ทำงานแบบโลคัลผ่าน stdio
ตรวจสอบล่าสุดเมื่อ 2026-11-09 Safety-gated computer use for AI agents: screen OCR and vision, mouse, keyboard, window control.