witness — a distillation engine that keeps the history of how things changed.
witness turns a stream of text into a dated record of how its subject changed. Feed it your AI coding sessions and it tracks how you think and work. Feed it market commentary, research notes, or any document stream and it tracks how that changed instead. One pure-Go binary, a local SQLite archive plus plain markdown, served to your agent over MCP.
The thing it is built around is change over time, with provenance. When an attribute's value changes, witness does not overwrite it — it closes the old value with a date and opens a new one, and records which source records drove the shift. So the archive answers "how did this get here", not just "what is true now".
"Aah, you were at my side, all along. My true mentor... My guiding moonlight..." — Ludwig, the Holy Blade
Two ways people use it
1. Watch your own growth as you work with AI. Install it into Claude Code or OpenCode and it captures your sessions in the background — no prompting, no ritual. It distills how your thinking, workstyle, habits, and blind spots shift over months, with evidence for every claim. This is the setup witness ships configured out of the box.
It is reflection-oriented, not clone-oriented: the point is to let your agent understand you, and to leave you a re-readable record of how you thought and grew. It is a pure tool, not a coach — it captures, structures, and serves. Building a coach on top ("you've done this three times…") is left to other projects that read its output.
2. Build a change-history over any text corpus. Write a lens — a prompt describing what to notice
and which dimensions to track — point witness ingest at NDJSON records, and the same four-stage
engine runs. Nothing about the machinery is person-specific: raw records → observations → dated
facets → narrative.
A worked example, run end to end while writing this section. A regime lens over market commentary,
in an archive with no person lens enabled at all: 7 news records became 22 observations and 13
facets. Then the regime flipped hawkish→dovish, and the archive recorded it as history rather than
replacing it:
inflation/core_trend
"Disinflation has broken rather than paused: core CPI reaccelerating…" valid_to 2026-06-…
"Core is disinflating persistently rather than reaccelerating…" (current)
Five facets closed and reopened that way in one review. The generated brief even flagged that its own earlier sequencing thesis had been falsified by the new data — which is the whole point of keeping the history instead of the latest snapshot.
Honest caveat: the engine is general, but everything witness ships is person-shaped — the
built-in default lens, the example lens, and the summary prompts all say "notice things about the
person". For a non-person corpus you write your own lens (three files) and, if you want, override the
summary prompt with one file. There is no market-lens or research-lens preset yet; the machinery is
domain-agnostic, the batteries included are not.
Is this you?
witness is the answer if you've ever wanted to:
- Remember what you learned across your AI coding sessions — Claude Code, OpenCode — instead of losing it when the context window resets.
- Have a second brain / AI memory for your thinking, workstyle, and habits, queryable by your agent.
- Track how you grow as a developer over time — the traps you fall into and climb out of — with provenance for every recorded change.
- Keep a journal / retrospective of how you thought and changed, re-readable months later.
- Distill a corpus you care about — market news, a knowledge base, research notes, meeting logs — into a dated history of how its state changed, not a pile of summaries. Feed it NDJSON with
witness ingest. - Query any of the above from your agent, over MCP, instead of re-explaining context every session.
Contents: How it works · Lenses · Example · Reading the archive · Commands · Install · Configuration · Your data
How it works
Four layers — one ground-truth, three derived and regenerable from it:
The archive is collect-only / pull-only: witness captures and distills everywhere, but never
injects anything into a session. Nothing is pushed — you (or an agent) read the profile on demand.
raw/observations/facets live in a single embedded SQLite database (witness.db); the profile is
plain markdown under profile/.
Lenses
Every observation/facet carries a lens tag:
default— global, runs on every session, cross-domain. This is the part no single-domain tracker can be: it sees that "diagnoses gaps precisely" fires in math and coding and career.- registered lenses (e.g.
math) — domain-specific lenses you register once and enable globally.witness lens register math ./math/adds the definition (a directory) to a central registry;witness lens enable mathmakes it run on every session (alongsidedefault). Lenses are shared, not tied to any repo, so the samemathlens covers all your math work.
Writing a lens
A lens is a directory of three files:
math/
lens.json settings: name, dimensions, optional per-lens models
extract.md per-session — mines observations (the whole file is the prompt)
review.md periodic — synthesizes observations into facets (the whole file is the prompt)
// math/lens.json
{ "name": "math", "dimensions": ["speed", "independence", "proof_rigor", "abstraction", "confusion_tolerance"] }
You are observing one session through a MATH-LEARNING lens. Notice things about the
person as a mathematician — how they reason, get stuck, and climb out…
Return ONLY a JSON array. Each element:
[{ "dimension": "proof_rigor", "observation": "…", "evidence": "…", "poignancy": 6 }]
The one rule to remember: each prompt file is used verbatim as the system prompt and replaces
the built-in default prompts — it doesn't extend them — so each must be self-contained,
including its output JSON schema (the tool appends the transcript / observations as the user
message, but injects no schema for you).
Two complete, copy-paste-ready lenses ship as starting points. Copy the directory and rewrite the dimensions and prose for your domain:
prompts/lens/example/— a person lens (math learning), for tracking someone as they work.prompts/lens/corpus-example/— a non-person lens (market regime), for tracking a subject that is not you. Start here for research notes, a knowledge base, incident reports, or any document stream; its README explains what has to change when the subject stops being a person.
cp -R "$CLAUDE_PLUGIN_ROOT/prompts/lens/example" ./math # edit the files, then:
witness lens register math ./math # copies the definition into your store (a snapshot)
witness lens enable math # start running it on every session
register stores a copy — editing the original afterward has no effect until you re-register.
enable is the separate switch that makes it actually run.
Per-lens models (optional). By default every lens rides the default models (witness config set triage_model / distill_model). A rare heavy lens can pin a stronger model just for itself —
without paying for it on every session — by adding extract_model / review_model to its
lens.json:
{ "name": "math", "dimensions": ["proof_rigor"], "extract_model": "claude-sonnet-5" }
Omit a field (or leave it empty) to ride the default. Since register stores a snapshot, edit the
registered copy under <witness-data-dir>/lenses/<name>/lens.json — or edit the source and
re-register. Verify what a lens resolved to with witness lens show <name>.
The source directory may live anywhere. As a recommended canonical location, witness keeps the
registered copy beside config.toml under <witness-data-dir>/lenses/<name>/ (normally
~/.local/share/witness/lenses/<name>/, or $WITNESS_HOME/lenses/<name>/). You can edit that
registered copy directly, but this location is a convention rather than a restriction on the
directory passed to lens register.
Example: one moment, end to end
Say a session contains this exchange (fictional):
you: the migration keeps failing on prod but passes locally — I'll just run it by hand and move on
you: …wait, what's actually different about prod? let me diff the two schemas before I touch anything
Here's what each layer makes of it.
raw (L0) — captured verbatim, nothing interpreted:
user the migration keeps failing on prod but passes locally — I'll just run it by hand and move on
user wait, what's actually different about prod? let me diff the two schemas before I touch anything
observations (L1) — the worker mines one atomic, evidence-anchored noticing:
[thinking] Caught the urge to hand-patch around a failure and redirected to isolating the
prod/local difference before acting.
evidence: "run it by hand and move on" → "what's different about prod? diff before I touch anything"
poignancy: 6 lens: default
facets (L2) — after several such moments the reviewer synthesizes an evolving attribute, and keeps the history (the whole point — it shows change, not just current state):
default · thinking · diagnoses_before_acting confidence 0.82
2026-05 → now Catches the reflex to work around a failure and isolates the
mechanism first; gates action on understanding the cause.
2026-02 → 2026-05 Tended to apply the first workaround that unblocked the task. (superseded)
profile (L4) — the narrative you actually read (witness profile):
default
You've been converging on a diagnose-first way of working. A few months ago the pattern was to reach for whatever unblocked the task; now you routinely catch that urge and turn to isolating the mechanism before you touch anything…
Nothing here is pushed into your sessions — you read it when you want it (witness profile), or an
agent pulls the relevant facet on demand.
Writing your own summary prompt
The shipped unified prompt writes a personal growth portrait. If your archive is something else
— market records, research notes, a project log — drop your own prompt in and it wins:
<data-root>/summarize/unified.md your prompt (overrides the built-in one)
<data-root>/summarize/lens.md same, for the per-lens summaries
witness doctor prints the data root. The file is the whole interface: no command to run, no
registration step. Delete it to go back to the built-in prompt.
The same facets can then produce a completely different document — a terse risk memo instead of a character portrait, say. Because the built-in prompt is still used when you have no override, a witness upgrade that ships an improved default still reaches you; once you override, your file is never touched by an upgrade.
Reading the archive
Humans read the narrative; agents read the structured data. Over MCP:
get_profile(lens)— the narrative profile (prose); omitlensfor the unified portrait.get_facets(lens)— the current structured facets.search_observations(query, lens)— local vector search over observations.record_observation(...)— an in-session agent writes a decision-aware observation directly (passed through verbatim), capturing context a later reviewer would miss.delete_observation(obs_id)— prune a wrong observation.
Commands
The visible front door, exactly as witness --help groups it:
Capture, the worker, and the MCP server are internal entry points invoked by
Claude Code/OpenCode, not typed by hand. witness import and witness worker … are hidden:
they exist for recovery and debugging, and everything they do also happens automatically.
witness profile [lens]— print the narrative profile (default: the unified portrait).witness facets [lens]— print current structured facets (CLI equivalent of MCPget_facets).witness observations search <query> [--lens <lens>] [-k N]— semantic search over observations.witness observations record --session <id> --dimension <name> --observation <text>— stage an active observation and kick the worker.witness observations delete <obs_id>— prune a wrong observation.witness worker review [--full]— force an L2 review and regenerate L4 profiles from existing observations. (Hidden, like the rest ofworker: review normally runs on its own schedule —review_every/review_poignancy— so reaching for this by hand is the exception.--fullalso runs the emergent long-arc pass.)witness lens register|enable|disable|list— manage lenses.witness lens backfill <name> [--fresh]— re-mine one lens over the whole history and refresh its facets;--freshfirst drops the lens's observations + facets (for a changed prompt).witness lens load-default— re-seed / restore the built-in "default" person-growth lens (it is auto-seeded once on first use and fully deletable, so this is how you bring it back).witness import --agent opencode— incrementally reconcile OpenCode's local session DB into L0 and kick background distillation without waiting.witness import --agent claude— kick distillation for already-captured Claude Code hook data.witness status— what has been captured, whether the worker is running, and how fresh the distilled data is (--jsonfor scripts).witness worker run/witness worker stop— the operator escape hatch for the background distillation worker (hidden from--help; normally it runs off editor hooks).rundrains in the foreground,--detachbackgrounds it. It accepts--since/--untilto select pending sessions by their latest raw timestamp — for examplewitness worker run --since 7ddistills sessions updated in the last seven days. Bounds also accept RFC3339 timestamps or UTC dates (YYYY-MM-DD) and do not discard sessions outside the selected range.witness cleanup— interactively reclaim old raw transcripts (keeps observations + profile).
Single binary, no runtime
The whole thing is one self-contained Go binary — no Python, no external services, no vector
DB, no cloud key. Local multilingual (English and Chinese) embeddings run pure-Go via GoMLX
(CGO_ENABLED=0, verified: matches ONNX Runtime exactly). Distillation defaults to your existing
Claude Code auth via claude -p; set runner = opencode to use a private opencode serve runner instead.
Install
./install.sh claude # Claude Code: build, fetch model (~448MB once), wire hooks + MCP
./install.sh opencode # OpenCode: build, fetch model, wire local plugin + MCP
That's the whole thing — idempotent, safe to re-run after a git pull. The target
is required: the script builds the binary, provisions the archive, and binds the matching
distillation runtime into config.toml (runner = claude or runner = opencode). It also
offers to add a witness command to your PATH (for witness profile, doctor, lens,
import, status, cleanup, ingest). Equivalent make targets exist (make install,
make install-opencode, make build, make doctor, make uninstall,
make uninstall-opencode, make clean). To remove the editor integration: make uninstall
or make uninstall-opencode (strips wiring; your data is untouched).
Windows
Windows uses a self-contained zip instead of the shell installer (there is no
guaranteed shell to run the hook shim). Download witness-windows-amd64.zip
(Intel/AMD) or witness-windows-arm64.zip from the releases page — each unpacks
to a witness\ folder holding witness.exe and the embedding model. Then, from
inside that folder in PowerShell:
.\witness.exe wire claude # Claude Code
.\witness.exe wire opencode # OpenCode
Either command copies the bundle into %LOCALAPPDATA%\witness, adds it to your user
PATH, and provisions the archive. The zip carries the prompt templates and the ~448MB
model alongside the exe; the binary resolves both relative to itself. Running both is
fine — the copy is idempotent, and they wire different editors.
What each wires, and why they differ: Claude Code spawns the hook itself, so wire claude writes exec-form hooks ({command: witness.exe, args: [...]}) into
settings.json. OpenCode is the other way round — its plugin spawns witness, so
wire opencode bakes the installed witness.exe path into the plugin and the MCP
entry. Neither needs a shell or Git Bash.
To remove an editor integration: witness.exe unwire claude / unwire opencode
(strips the hooks/plugin + MCP entry). The copied files and PATH entry are left in
place, since the other integration may still be using them.
OpenCode support
OpenCode support has two pieces:
- A plugin reconciles OpenCode's SQLite DB on startup and when a session goes idle, then asks the
laptop-friendly auto-start gate to distill when allowed. From-source installs write a
local plugin to
~/.config/opencode/plugins/witness.js; published installs can use the npm plugin@witness-ai/opencode. (That path is the same on Windows — OpenCode resolves its config asXDG_CONFIG_HOMEelse~/.config, with no Windows-specific branch, so the plugin lands in%USERPROFILE%\.config\opencode\plugins\.) - An OpenCode MCP entry named
witnesslaunches the same MCP server as Claude Code, exposingget_profile,get_facets,search_observations,record_observation, anddelete_observation.
The npm package ships the OpenCode plugin, a witness CLI shim, prebuilt witness binaries, and prompts.
The config-only path is the default: add the plugin to ~/.config/opencode/opencode.json, and OpenCode
installs it automatically with Bun on startup. If mcp.witness is absent, the plugin auto-registers it
for you.
The npm package supports OpenCode integration and general CLI usage (witness profile, ingest, etc.).
Editor wiring (witness wire / unwire) is a source-checkout command — npm users configure the plugin
directly in opencode.json and the plugin auto-registers MCP. Archive provisioning and record ingestion
(witness install, witness ingest) work normally with the npm package.
The npm distribution supports exactly these platforms:
macOS Intel, Linux ARM, and Windows are not supported by the npm distribution. Each binary is published as an optional platform package, so npm installs only the binary for the current machine.
{
"$schema": "https://opencode.ai/config.json",
"plugin": ["@witness-ai/opencode"]
}
To test the current prerelease without replacing latest, pin the plugin entry:
{
"$schema": "https://opencode.ai/config.json",
"plugin": ["@witness-ai/opencode@beta"]
}
Optional: install it globally if you also want a witness command on your shell PATH:
npm install -g @witness-ai/opencode
Optional: run it ad hoc without a global install:
npm exec --yes --package=@witness-ai/opencode -- witness doctor
The main npm package contains the plugin, CLI wrapper, and prompts; the matching optional platform package
contains one binary. The first embedding-model download is about 470MB. Installing the packages does not
start that download: the plugin starts it when OpenCode next runs. Keep OpenCode running until the first
download finishes. The plugin owns the downloader, stops it on shutdown, and retries later with bounded
backoff.
If you already have your own mcp.witness config, the plugin leaves it untouched. The npm wrapper does
not support witness wire / witness unwire; those commands are for source-checkout editor wiring.
Custom model mirrors must provide WITNESS_MODEL_SHA256 and WITNESS_TOKENIZER_SHA256 alongside
WITNESS_MODEL_BASE_URL.
After the first download, verify the model, OpenCode runner, archive, and queue:
npm exec --yes --package=@witness-ai/opencode@beta -- witness doctor
npm exec --yes --package=@witness-ai/opencode@beta -- witness status
Upgrade note: Older witness releases could leave an OpenCode session whose agent or title is
witness-distillwhen distillation was interrupted. Current releases no longer create or filter those sessions in OpenCode's user database. Before the first import after upgrading, remove any leftoverwitness-distillsessions with OpenCode's supported session-management tools so they are not captured as normal archive data.
The npm package lives in npm/opencode. Stage prebuilt binaries and prompts before publishing:
make npm-opencode-package
npm publish ./npm/platform/darwin-arm64 --access public --tag beta
npm publish ./npm/platform/linux-x64 --access public --tag beta
(cd npm/opencode && npm publish --access public --ignore-scripts --tag beta)
Configure npm Trusted Publishing separately for the main package and both platform packages before using
the release workflow. The npm package page renders npm/opencode/README.md;
the workflow verifies after publishing that npm identifies it as the package README and that the published
tarball contains it.
For the first platform-package release only, publish both platform packages manually before creating the GitHub Release, then configure their Trusted Publishers on npm. npm requires a package to exist before its package-level Trusted Publisher can be configured. The release workflow is idempotent: it skips an already published package version, publishes any missing platform versions first, then publishes the main package.
Manual verification path:
witness lens register math prompts/lens/example # optional: register an extra lens
witness lens enable math
witness import --agent opencode # reconciles ~/.local/share/opencode/opencode.db and returns
witness status # watch non-blocking distillation progress
witness worker review # forces L2 facets + L4 markdown profiles (hidden cmd)
witness profile opencode # per-lens L4 report
witness profile # unified L4 report
Configuration
~/.local/share/witness/config.toml (all optional; sensible defaults):
runner = "claude" # "claude" (default) or "opencode"
triage_model = "" # MINING (L0 -> L1), once per session; "" = the runner's default
distill_model = "" # REVIEW (L1 -> L2/L4), batched; "" falls back to triage_model
review_every = 5 # run the reviewer every N distilled sessions...
review_poignancy = 30 # ...or sooner once accumulated salience crosses this (0 = off)
auto_distill = true # hooks/plugins may start model work automatically
mine_concurrency = 4 # parallel per-session miners (<= 0 restores the default)
Model ids are passed through to whichever runner is bound, so use that runtime's spelling —
claude-haiku-4-5 or claude-sonnet-5 for runner = "claude", openai/gpt-5.5-style ids for
runner = "opencode". Leaving both empty is the safe default and what witness install writes;
note that an empty triage_model under the claude runner inherits your ambient interactive
model, which may be heavier than you want for per-session mining, so pinning a light one is
usually worth it.
Set auto_distill = false for capture-only mode on battery-constrained machines, then run
witness worker run --detach manually when plugged in. Automatic workers are short-lived: they load the
embed model only while draining queued sessions, then exit.
When runner = opencode, triage_model and distill_model should use OpenCode model names such
as openai/gpt-5.5; empty values use your OpenCode defaults. Non-empty OpenCode model names are
validated against opencode models <provider> before distillation, and witness doctor reports the
same check as opencode models: OK or an explicit invalid-model error.
Enabled lenses are managed for you (witness lens enable/disable <name>) and appear as simple
lines, each naming a registered lens that runs on every session:
lens = math
There is no automatic retention knob: raw transcripts are kept until you deliberately reclaim
them with witness cleanup (which never touches your observations or profile).
Your data is yours
Everything lives under ~/.local/share/witness/ (override with WITNESS_HOME; installs
predating the rename keep using ~/.local/share/claude-witness/, adopted automatically), is 0700
(the DB and profile files 0600), and never leaves your machine. The repo ships the framework,
schema, and prompts — never anyone's archive.
Troubleshooting. Distillation runs in a detached worker, so failures land in
$WITNESS_HOME/witness.log (JSON lines) rather than on your terminal. It records the worker's
startup steps, each mine's input size and duration, and each model call's outcome — enough to tell a
slow model from a stalled one. For more detail, set WITNESS_LOG_LEVEL=debug (also accepts warn /
error; anything unrecognized falls back to the info default, so a typo can never stop witness
capturing).
Backup / sync. Use witness export --all <dir> — it writes a complete archive you can point
a syncer at. Do not sync the live data directory directly: the database runs in WAL mode (.db +
-wal + -shm), and a syncer racing those files can corrupt it. Wire it up yourself, e.g. a
cron/launchd job:
witness export --all ~/Dropbox/witness-backup --force # complete archive, safe to sync
--all matters because only one of the four things in your data directory is the database:
Left out on purpose: witness.log (diagnostics), runtime/ (OpenCode's disposable private runtime),
and the lock files. Either form runs safely while the worker is writing — no need to stop it.
Restoring needs no separate command. An --all directory has the same layout as your data
directory, so either point witness at it in place (WITNESS_HOME=~/Dropbox/witness-backup) or, with
witness stopped, copy its contents into your data dir. A plain database-only snapshot restores the
same way as witness.db, and witness worker review rebuilds the narrative from it.
License
MIT — see LICENSE.
