Utilities
Quickstart
Add the tools to your AI assistant:
claude mcp add openreality -- npx -y openreality-mcp serve
codex mcp add openreality -- npx -y openreality-mcp serve
Add this to claude_desktop_config.json (Settings → Developer → Edit Config)
or ~/.cursor/mcp.json:
{
"mcpServers": {
"openreality": {
"command": "npx",
"args": ["-y", "openreality-mcp", "serve"]
}
}
}
Claude Code plugin (the MCP server plus the workflow skill in one step):
/plugin marketplace add reality-opened/openreality
/plugin install openreality@openreality
Claude desktop extension: download openreality-mcp-<version>.mcpb from the
latest release
and double-click it, or drag it into the Claude desktop window. No terminal needed.
Then sign in once (opens your browser, stores a revocable API key on your machine):
npx -y openreality-mcp login
Scan a room with your phone at open-reality.io, or just ask your assistant to upload a video file. Full per-client setup: open-reality.io/mcp.
Examples
Once connected, talk to your assistant like this:
"Upload
~/Videos/kitchen.mp4and reconstruct it."
"What objects are in my latest scan, and how big is the room?"
"The counter edge to the window is 2.4 m. Calibrate the scene, then measure the couch."
"Plan a path from the door to the desk and describe it."
"Export this scan as robot-training data and save the zip locally."
Self Hosting
The whole workflow is self-hostable. Read
server/docs/self-hosting.md; the short version:
git clone https://github.com/reality-opened/openreality
cd openreality/server
# Path A: your own GPU box (one process, local disk)
python -m server.selfhost --data-dir ~/openreality-data
# Path B: your own Modal account (CPU web server + GPU worker)
modal deploy modal_selfhost.py
Self-hosted servers need no account: a single token printed at first start is
your login, and the MCP client connects with OPENREALITY_URL plus that token.
[!IMPORTANT] Licensing. This repo is BSD-2-Clause, but the 3D reconstruction model a self-hosted server downloads (VGGT-1B and the VGGT code it runs on) is licensed by its owners as CC BY-NC 4.0, non-commercial use only. Nothing here redistributes it; your server fetches it from the source under their terms. For commercial use, use the hosted service (which runs a commercially licensed model) or get your own license from the model owners.
Repo index
server/ and core/ are curated mirrors of our private working repos, synced
by hand; each carries a MIRROR.md that says exactly what is included and how
it is synced. mcp/ is developed in this repo directly.
Technical details
flowchart LR
A["📱 phone video"] --> B["server: 3D reconstruction<br/>(GPU job)"]
B --> C[("persistent scene:<br/>geometry + objects + report")]
C --> D["41 MCP tools"]
D --> E["🤖 Claude / Codex / Cursor"]
E -->|"measure · navigate · export"| D
The MCP process always runs on your machine and holds your credentials; every tool call is a typed REST call to a server (ours or yours). Big files are written to your disk, never pasted into the AI's context. Server refusals and uncertainty labels reach the AI unedited, so it cannot pretend a relative number is metres.
The measurement rule is explained in Why our 3D tools refuse to say "metres" until you prove it.
🙏 Acknowledgements
Open Reality stands on other people's research and code. Thank you to:
- VGGT-SLAM by Dominic Maggio, Hyungtae Lim and Luca Carlone (MIT SPARK Lab). The reconstruction core in
core/is the VGGT-SLAM 2.0 line, and the scan demo above is theirs. - VGGT (Meta AI), the feed-forward 3D backbone a self-hosted server fetches under its own licence (see the note above).
- Depth Anything V2, the metric-depth model behind our scale-drift correction between map segments.
- gsplat from the Nerfstudio project, the optional GPU refinement in our splat export.
- SAM 3 (Meta AI) for segmentation, and SAM 3D Objects and TRELLIS (Microsoft) for object completion and variants on the hosted service.
- LeRobot (Hugging Face), NVIDIA GR00T and Isaac Sim, whose dataset and scene formats our robot-training exports target.
- GTSAM for factor-graph optimization, cordis for the plugin runtime, and the MCP TypeScript SDK.
🤝 Contributing
Issues and pull requests are welcome on any component. Changes to mcp/ land
here directly; fixes to server/ and core/ are folded back into the private
working repos and re-synced out. If you self-host and something breaks, an
issue with your logs is a gift: the self-host paths are young.
License
BSD-2-Clause for everything in this repository. Third-party models
are fetched from their owners under their own licenses (see the licensing note
above and server/docs/self-hosting.md).
