DeskMate is a local-first desktop activity recorder for Windows. It captures
your screenshots, on-screen text (OCR / accessibility tree), keyboard/mouse/clipboard
events, and optional audio transcription into a SQLite database ,
then lets you use that data through a browser UI, a REST API, and an
that plugs straight into AI coding agents like and —
so your assistant can search your activity, ask natural-language questions, and
run auto-generated reports (see ).
DeskMate can also read what is on your screen and — start a
learning session, start a meeting recording, reply to an email, book a meeting room —
which you confirm with one click (see
).
Local-first PC activity recorder for Windows. Captures screenshots, accessibility text, UI events, clipboard activity, and optional audio transcription into a local SQLite database — then exposes everything through a REST API, browser UI, and MCP server.
Verify: open PowerShell and run python --version — you should see Python 3.10.x
or higher.
Step 2 — Get the code and create an isolated environment
git clone <repo-url>
cd deskmate
# Create an isolated Python environment (keeps your system clean)
python -m venv .venv
.\.venv\Scripts\Activate.ps1
Every time you open a new terminal to use DeskMate, cd into the folder and run
.\.venv\Scripts\Activate.ps1 again to activate the environment. You'll know it
worked when you see (.venv) at the start of your prompt.
Step 3 — Install (recommended default combo)
pip install -e ".[ocr-winrt,audio,vad,mcp]"
This installs: core features + built-in Windows OCR + audio transcription + voice
activity detection + MCP. That's enough for most people.
(What each [...] means and what else you can add → see Install features as needed.)
On first run it auto-creates a config and data folder under
C:\Users\<your-username>\.deskmate\. You're up and running.
To stop: press Ctrl + C in the terminal where deskmate ui is running.
🧩 Install features as needed (submodule dependencies)
DeskMate splits the "heavy" features into optional modules (extras) — install only
what you need, not everything at once. The install format is always:
pip install -e ".[module-name]" # install one
pip install -e ".[module1,module2]" # install several (comma-separated, NO spaces)
Needs a mic or system audio (WASAPI). Default transcription backend
audio-openvino
Whisper on Intel NPU/GPU/CPU
Accelerated on Intel Core Ultra (NPU) or Arc GPU, see below
vad
Voice activity detection (Silero VAD)
Skips silent segments to save compute; install alongside audio
speaker
Speaker diarization
Tells "who said what"
redact-onnx
PII redaction
Optional privacy-protection model
semantic
Semantic (vector) search
Search by meaning, not just keywords
pipes
Scheduled tasks (YAML)
Run reports automatically on a schedule
mcp
MCP server
Integrate with AI agents / Claude, etc.
route
Decision routing (local kev-0.8b model)
Suggests starting a learning session / meeting, replying to an email, or booking a room; see Decision routing
training
Local LoRA fine-tuning
Fine-tune a small model on your own data, see training docs
full
Most common extras at once
= tesseract + audio + vad + redact + semantic + pipes. Excludes OpenVINO & training
dev
Dev tools (pytest, ruff)
Only needed to modify code / run tests
Beginner tip: start with the default combo from Step 3. Add a module later when you
decide you want a feature (e.g. Chinese OCR, semantic search) — re-running
pip install -e ".[...]" is safe.
Battery Saver / Power Manager is built in and uses Windows APIs through the
standard library; it does not need an extra package.
Install from a built wheel (instead of the source checkout)
If you have the built .whl (see Building a wheel below)
instead of the source tree, use the same [extras] syntax on the wheel file:
:: core only
pip install deskmate-0.1.0-py3-none-any.whl
:: recommended combo (same as Step 3)
pip install "deskmate-0.1.0-py3-none-any.whl[ocr-winrt,audio,vad,mcp]"
Install (almost) everything in one go — every optional feature at once. Note
there is no single "all" extra, so list them explicitly. This pulls in the heavy
deps (PyTorch, OpenVINO, ONNX Runtime, Whisper, RapidOCR…), so expect a large download:
This deliberately omits training (LoRA fine-tuning — only needed if you train
models; see training docs), redact-onnx-dml (the
DirectML build of redact-onnx — pick one, not both), and dev (test tooling).
The same works on the source checkout — just swap the wheel filename for -e ".",
e.g. pip install -e ".[ocr-winrt,audio,vad,speaker,semantic,mcp,pipes]".
⚠️ Most extras pull Windows-only / GPU-specific wheels and large model runtimes.
Don't install everything unless you actually need it — the recommended combo
is enough for most users.
📦 Building a wheel (for distribution)
To produce a distributable wheel (e.g. to install on another machine):
The wheel bundles all of DeskMate's own code and the built-in apps
(deskmate/apps/), but not third-party dependencies — those are recorded as
requirements and fetched by pip install from PyPI. For an offline install,
pre-download everything into a folder first:
pip download "deskmate-0.1.0-py3-none-any.whl[ocr-winrt,audio,vad,mcp]" -d wheelhouse
:: then on the offline machine:
pip install --no-index --find-links wheelhouse "deskmate[ocr-winrt,audio,vad,mcp]"
🧠 Installing the training extra (LoRA fine-tuning)
The training extra (torch + unsloth + transformers + peft + accelerate)
is deliberately left out of the "everything" command above, because the right
torch build depends on your accelerator. A pip [extra] can't pin a
hardware-specific wheel (the Intel XPU torch lives on PyTorch's own index, not
PyPI). Below, WHL = deskmate-0.1.0-py3-none-any.whl; on a source checkout use
-e "." instead.
Intel Arc / Core-Ultra iGPU (XPU): the XPU torch wheel and an
oneAPI/Level-Zero/MSVC toolchain are required — there are several non-obvious
gotchas, so this path is scripted. Install the extra, then run the setup script:
Full walkthrough (oneAPI, MSVC, the toolchain gotchas) → training docs.
CPU only (no GPU — slow, for validating the pipeline on a tiny base like
Qwen/Qwen3-0.6B, not real ≥1B runs):
pip install "WHL[training]" :: the default torch is already CPU-only
Verify XPU is live after setup:
python -c "import torch; print(torch.__version__, getattr(torch, 'xpu', None) is not None and torch.xpu.is_available())"
:: expect: 2.10.0+xpu True
🤖 Want Q&A / auto-reports? Install Ollama first (optional but recommended)
DeskMate's natural-language Q&A (Ask) and the LLM apps under My Apps (daily
recap, todo extraction, meeting summaries…) need a local LLM server, Ollama. If you
don't use these, you can skip this section — recording and search work fine without it.
💡 Recommended: use the Ollama OpenVINO build on Intel hardware
If you're on an Intel CPU / Arc GPU / Core Ultra (NPU), we strongly recommend
Ollama-OV (OpenVINO backend) — it runs models on Intel CPU/GPU/NPU, faster and
more power-efficient:
👉 https://github.com/zhaohb/ollama_openvino
. On regular GPUs / other platforms, the official
Ollama works fine.
Option A — Ollama OpenVINO build (recommended on Intel)
Import a model in OpenVINO IR format as described there (get it from HuggingFace /
ModelScope → write a Modelfile with ModelBackend "OpenVINO" → ollama create <name> -f Modelfile).
Put the model name into config.toml (below).
Option B — Official Ollama (other platforms / simplest)
Install Ollama: https://ollama.com/download (it runs a local service in the
background at http://127.0.0.1:11434).
Pull a model: ollama pull qwen3.
Final step for both options — in C:\Users\<your-username>\.deskmate\config.toml,
set the model name to the one you actually have:
[ollama]
base = "http://127.0.0.1:11434"
model = "qwen3_8b_ov:v1" # OpenVINO build: the name you imported it as.
# Official build: the name you pulled (e.g. qwen3).
Then restart deskmate ui — you can now use Ask on the Home page and run reports under My Apps.
⚡ Advanced features (read when you need them)
These are more specialized capabilities - beginners can ignore them for now and
come back when needed:
DeskMate can watch what is happening — the foreground window, on-screen text, audio
transcripts, key text, and whether you are in a meeting or a study session — and
suggest what to do next. A small local model (kev-0.8b, run in-process; no
cloud, no extra service) judges four things:
Suggestion
When it shows up
What Confirm does
Start learning
A course / lecture / tutorial is in the foreground
Opens a learning session. You end it yourself
Start meeting
A live call UI is in the foreground (Meet, Teams, Zoom, Tencent Meeting, Feishu, DingTalk, …)
Opens a meeting recording. You end it yourself from the Meetings page
Reply to email
The email you are reading asks you a question or a favour
Records your confirmation only (nothing is sent)
Book a meeting room
A mail, chat, or call asks you to book a room
Records your confirmation only (nothing is booked); keeps a time / size hint
Nothing runs without your click. Booking a room and replying to an email are not
automated yet — the confirmation is stored so a future executor can pick it up
(GET /route/executions?status=confirmed).
How it works
You open a course page → a Windows notification appears ("Looks like you are
studying — start a learning session?"). Email / room notifications show the mail
subject (basic redaction applied; see toast_show_subject).
Click Open → the Suggestions page highlights the card.
Click Confirm (or Not this). Your answer is saved as feedback for later
threshold tuning. Sessions you start this way are marked From Suggestions.
It stays quiet on purpose: quiet hours ([habits] quiet_hours) are honoured, the same
page or mail is only suggested once (2 h; 8 h after you said Not this), nothing is
suggested while a learning session or meeting is already open, and terminals, editors,
and the desktop / taskbar are never scored. Outside a meeting, audio is not used
to decide on room booking, so a video playing in the background cannot trigger it.
Turn it on
Off by default. Install the extra, then enable it and restart DeskMate:
pip install -e ".[route]"
If you already run a tuned PyTorch (for example an Intel XPU build), install kev
without dependencies so it does not replace it — see the note above the route
extra in pyproject.toml.
[route]
enabled = true # default: false
# kev_device = "xpu" # Intel GPU; falls back to "cpu" automatically
# executor_actions = ["start_learning", "start_meeting", "reply_email", "book_room"]
# toast_show_subject = true # false: notifications stay generic, subjects only on the page
# screen_excluded_apps = [...] # windows that are never scored (terminals, desktop, ...)
[meeting]
nudge_minutes = 15 # remind you that a meeting you started is still open (0 = off)
The model loads on the first judgment (about 12 s, ~6 GB RAM on XPU; roughly 0.5 s per
judgment afterwards; on CPU it is several times slower). Model files are read from
your Hugging Face cache — set HF_HUB_OFFLINE=1 to forbid downloads.
Privacy
Everything stays on your machine. Routing reads the same text DeskMate already stores
(the redacted version when [redact] is enabled), and mail subjects are additionally
masked for e-mail addresses and phone numbers before they are shown — also only when
[redact] is enabled; names and project names are never masked. Pausing capture stops
routing, and /capture/forget also removes routing data (and any session or meeting
that was started from a suggestion in the forgotten span).
GET /route/live POST /route/suggestions/{id}/execute
GET /route/suggestions POST /route/suggestions/{id}/dismiss
GET /route/executions?status=pending POST /route/suggestions/{id}/ack
GET /route/intents GET /route/scores
POST /route/feedback GET /route/segments
Known limits
A live webinar or livestream can look exactly like a meeting on screen; you get
one extra notification, never an automatic meeting.
The reply to email threshold is not calibrated yet and can miss a plain question.
Tested end to end on Chrome / Gmail and Google Meet; other call apps are covered by
offline samples only.
Pages whose title keeps changing (live counters, progress in the title) may be
suggested more than once.
Windows toasts appear under the PowerShell app name.
Run the tests with pytest tests/test_route_*.py; the real-model contract tests are
opt-in (RUN_KEV_TESTS=1).
Requirements
Windows 10 / 11
Python 3.10+
Optional, by feature:
OCR: ocr-winrt needs nothing extra; ocr-tesseract needs Tesseract on PATH;
ocr-rapidocr just needs the extra (best for Chinese)
Audio: a microphone, or a system-audio loopback (WASAPI) device
OpenVINO acceleration: an Intel CPU; an Intel Core Ultra (NPU) or Arc/iGPU
unlocks NPU/GPU inference
Decision routing: live status, items waiting for your confirmation, intents, scores, correct / not-this feedback
Learning
Start/end a study session (or confirm one from Suggestions); recap, review queue, knowledge graph
Todos
Action items extracted from email + meetings
Meetings
Start/end a session (or confirm one from Suggestions); audio + screen text, recap video, summary
Capture
Capture-control view for recording sources and runtime toggles
Training
Local LoRA fine-tuning
Model Service
Start/stop local model backends and inspect service logs
Diagnostics
Environment self-checks for common setup issues
Reminders
Proactive habit reminders, acknowledgement, and history
Battery Saver
EcoQoS battery saver for background AI workers and selected apps; UI text is localized in Chinese/English
Settings
Config and monitors
CLI
deskmate serve # HTTP API only (records by default)
deskmate record # recorder only
deskmate capture-once # capture one frame manually
deskmate route tail # latest decision-routing suggestions (add --follow / --intents)
deskmate search "query" # keyword search via the API
deskmate mcp # MCP server for AI agents (see "MCP server" below; API must be running)
Split API and recorder into two processes:
deskmate record
deskmate serve --no-run-daemon
LLM apps (My Apps)
Built-in apps live in deskmate/apps/ (and ship inside the installed package);
your own apps go in ~/.deskmate/apps/plugins/. Run them from the My Apps
page or the CLI. Full list, examples, and how to write your own in
deskmate/apps/README.md.
App
Purpose
todo-list
Unified checkbox todos from email + meetings
meeting-summary
Summarize a meeting you started/stopped
user-learning
Recap a study session (concepts + notes)
email-digest
Inbox overview
email-compose
Draft / reply via Gmail or Outlook
day-recap / time-breakdown / ai-habits
Daily recap / time split / AI-tool usage habits
user-profile / habit-report
Multi-day user profile / routine (default: last 7 days)
These apps need both Ollama and the DeskMate API running.
Configuration
Config file: C:\Users\<your-username>\.deskmate\config.toml (auto-created on first run).
C:\Users\<your-username>\.deskmate\
├── config.toml # configuration
├── data.db # SQLite database (full-text search; also routing suggestions and feedback)
├── frames\ # screenshots
├── audio\ # audio chunks (when enabled)
├── videos\ # video chunks
├── checkpoints\ # LoRA training artifacts (when training)
├── apps\ # LLM app output reports; apps\plugins\ for your own apps
└── logs\ # logs
To wipe and start over: stop DeskMate, then delete data.db and the folders above.
API
Default base URL: http://127.0.0.1:3030. Full endpoint list is in the running
server's /docs (OpenAPI). Common ones:
GET /health GET /search?q=... POST /ask
GET /frames POST /capture GET /meetings
GET /todos POST /todos PATCH /todos/{id}
GET /route/live GET /route/suggestions POST /route/suggestions/{id}/execute
DeskMate ships an MCP stdio server so other AI
agents (Claude Desktop, IDE assistants, custom clients) can query your local
activity and drive DeskMate's Q&A and report apps. It's a thin proxy over the
local HTTP API, so the DeskMate API must already be running (deskmate serve
or deskmate ui).
Install & launch
pip install -e ".[mcp]" # or add `mcp` to your extras combo
deskmate serve # in one terminal — the API on :3030
deskmate mcp # in another — the MCP stdio server
Tools exposed (all prefixed deskmate_ so they don't collide with other MCP servers):
List latest UI events (clicks, focus, key text, clipboard)
deskmate_health
read
Daemon liveness + counters
deskmate_capture_once
action
Trigger a paired capture now
deskmate_ask
action
Natural-language Q&A: an LLM searches your local context and answers (slow)
deskmate_list_apps
read
List the report apps (day-recap, standup-update, todo-list, meeting-summary, email-compose, …)
deskmate_run_app
action
Run a report app and return its result (slow; params vary per app)
deskmate_learning_sessions
read
List recent study sessions
deskmate_learning_reviews
read
Due review cards (spaced repetition)
deskmate_list_app_outputs
read
List an app's past run outputs
deskmate_get_app_output
read
Fetch one output file (markdown/json) of a past run
deskmate_ask and deskmate_run_app invoke a local LLM, so they need Ollama +
the API running and can take tens of seconds to minutes.
💡 Why connect it to Claude Code / Cursor?
Once DeskMate is registered, your coding assistant knows what you've been doing
locally — no copy-pasting context. While you work in Claude Code or Cursor you can ask:
deskmate_ask — natural-language Q&A over your activity
"Use deskmate to recap the last hour: which apps I used, how long on each, and what I was doing — as a timeline."
"Ask deskmate: between 2pm and 4pm today, what task was I mainly working on, and which files/windows were involved?"
"From my local activity record (deskmate), what was I last reading about in the browser?"
deskmate_search — full-text search over OCR / UI / audio
"Use deskmate to search my screen OCR, UI text and audio transcripts for 'recursive doubling', list the hits by time with the app/window they appeared in."
"Use deskmate to find frames in the last 3 hours whose OCR text contains 'error' or 'exception', and tell me which app each was in."
"Use deskmate to find which .py files I opened in VS Code today."
deskmate_list_apps / run_app — report apps
"Use deskmate to list all available report apps and what each one does."
"Use deskmate to run the time-breakdown report for the last hour."
"Use deskmate to run day-recap and summarize what I did today."
"Use deskmate to run standup-update: what I did, what's next, and any blockers."
"Use deskmate to run user-learning for my latest study session."
deskmate_learning_sessions / reviews — study recap
"Use deskmate to list my recent study sessions and what's due for review."
deskmate_recent_events / recent_frames — raw activity stream
"Use deskmate to show my last 50 UI events (clicks/focus/typing/clipboard) and summarize what I was just doing."
"Use deskmate to list the last 20 captured frames and tell me which apps/windows I've been switching between."
deskmate_health / capture_once — status & instant capture
"Use deskmate to check whether the recorder daemon is healthy and how many frames/transcripts it has captured."
"Use deskmate to capture the screen right now and tell me what's on it."
deskmate_list_app_outputs / get_app_output — past reports
"Use deskmate to list day-recap's past report runs and show me the contents of the latest one."
The assistant reads your local activity record on demand and answers in-context,
entirely on your machine.
Register the server
Point command at the Python env that has the mcp extra installed.
Claude Code (CLI):
claude mcp add deskmate -e DESKMATE_API=http://127.0.0.1:3030 \
-- C:/path/to/env/python.exe -m deskmate mcp
# verify: claude mcp list (should show deskmate ✓ connected)
Cursor — add to ~/.cursor/mcp.json (global) or <project>/.cursor/mcp.json:
Claude Desktop — same block in claude_desktop_config.json.
After editing config, fully restart the client — MCP servers are only loaded
at startup.
Notes
DESKMATE_API (optional) overrides the API base URL; defaults to
http://127.0.0.1:3030.
Slow tools / timeouts:deskmate_ask and deskmate_run_app invoke a local
LLM and can take minutes. The server-side budget defaults to 30 min and is
configurable via DESKMATE_MCP_ASK_TIMEOUT / DESKMATE_MCP_RUN_APP_TIMEOUT
(seconds, in the server's env). These tools also stream MCP progress every
~15s, which keeps most clients from cutting the call off; if your client still
times out, raise its tool timeout with MCP_TOOL_TIMEOUT (milliseconds; e.g.
1800000 for 30 min).
Behind a corporate proxy? No action needed — the MCP server talks only to
the local API and deliberately ignores HTTP(S)_PROXY, so 127.0.0.1 traffic
is never routed through a proxy.
The MCP server and the daemon are independent processes coupled only by the
API, so the MCP server can run from a different environment than the daemon —
it just needs the mcp extra.
FAQ
Install error / a feature is missing? Most likely the matching extra isn't
installed. Go back to Install features as needed
and add that one.
Ask / My Apps does nothing or errors? Check that Ollama is running
(ollama list should list models) and that [ollama] model in config.toml matches
a model you actually have. On Intel hardware, use the
Ollama OpenVINO build (the default model
name qwen3_8b_ov:v1 is meant for it).
deskmate not found in a new terminal? You didn't activate the virtual
environment. cd into the project folder, then run .\.venv\Scripts\Activate.ps1.
Decision routing never suggests anything? Check that the route extra is
installed, that [route] enabled = true is set in config.toml, and that you
restarted DeskMate. GET /route/live shows whether the model loaded, and
deskmate route tail --follow shows suggestions as they arrive. It stays silent in
quiet hours, in terminals / editors / the desktop, and while a session or meeting is
already open.
Want the internals / finer config of a feature? It's all in docs/,
numbered by module.
DeskMate is free to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, subject to the MIT license terms.
D
Deskmate
zhaohb·113 étoiles·
Local-first PC activity recorder for Windows. Captures screenshots, accessibility text, UI events, clipboard activity, and optional audio transcription into a local SQLite database — then exposes ever
Local-first PC activity recorder for Windows. Captures screenshots, accessibility text, UI events, clipboard activity, and optional audio transcription into a local SQLite database — then exposes everything through a REST API, browser UI, and MCP server.
Outils
Outils
Pas encore de liste d'outils. Les serveurs distants en obtiennent une après le premier contrôle d'état.
FAQ
Questions
Comment installer Deskmate ?
Copiez sur cette page la configuration stdio pour Cursor, Claude Desktop, Claude Code ou VS Code. Elle utilise l'installation de l'éditeur, pas un proxy PlugMCP.
Aucune variable d'environnement requise n'est indiquée sur cette fiche.
Quel transport Deskmate utilise-t-il ?
Deskmate s'exécute en local via stdio.
Quand Deskmate a-t-il été vérifié pour la dernière fois ?
Dernière vérification : 2026-10-28. Local-first PC activity recorder for Windows. Captures screenshots, accessibility text, UI events, clipboard activity, and optional audio transcription into a local SQLite database — then exposes everything through a REST API, browser UI, and MCP server.