A local, single-binary search-and-recall service for your own files and chat
history. It indexes notes, source code, and assistant session logs into a local
corpus, runs hybrid semantic + keyword retrieval over them, and exposes the
results to any MCP client (Claude Desktop, Cursor, Claude Code, the ostk kernel,
and others). Data and indexes stay on your machine; nothing is sent off-box.
Local-first cross-corpus retrieval MCP server — model2vec embeddings + LanceDB (Tantivy BM25) + SQLite. One binary; hybrid (dense + BM25) search across markdown, code, and Claude Code sessions.
星數
6
工具
—
可用率
—
Built on model2vec-rs for static
embeddings, LanceDB (Arrow + Tantivy BM25) for the vector
and full-text store, SQLite (via rusqlite) for the
ingest manifest, audit log, and concept/thread ledger, and
fastembed-rs for the optional
cross-encoder reranker.
What it does
Ingests many source kinds into one corpus — markdown trees, source code,
Claude Code, Gemini, and Codex CLI session logs, arbitrary globs, Claude.ai
.zip exports, and ostk .ostk/ directories.
Hybrid retrieval — model2vec dense vectors and Tantivy BM25, fused with
Reciprocal Rank Fusion, with an optional cross-encoder rerank pass.
A concept ledger — typed nodes and attributed, directed edges, durable in
SQLite. Grow them from markdown frontmatter at scan time, or write them
directly over MCP. Each edge records its origin (authored / observed /
promoted) and derives a conductance from confidence and recency rather than
storing a weight. Diffusion also walks the latent (vector-similarity) half of
the graph; an off-diagonal bridge walked during consolidation is promoted into
a weak reified edge that must then earn its conductance through use, or decay.
A live attention runtime — a turn observer, an auto-weaver that links new
chunks to thread anchors, and an idle curator that fades inactive threads. It
also maintains an ambient "memory lens" (an MCP resource) aligned to whatever
the current attention vector is focused on.
Two agent-facing MCP tools (recall and remember) plus a resources
surface, served over stdio or a shared local-socket daemon. Historical tool
names remain hidden compatibility aliases for one transition cycle.
Status
Pre-alpha but functional; the maintainer runs it as a daily driver.
Not yet built:
Per-file offset cursors for claude_code / gemini incremental scan
(append-only JSONL currently falls back to a full-source rescan when poked by
the watcher; content-addressed chunk ids keep that idempotent, just wasteful).
MCP transports beyond stdio (HTTP / SSE).
Install
Release binaries
Pre-built binaries for Linux x86_64, macOS arm64, and Windows x86_64 ship with
every tagged release: https://github.com/os-tack/ostk-recall/releases. Untar
(unzip on Windows), put the binary on your PATH, done.
From source
git clone https://github.com/os-tack/ostk-recall
cd ostk-recall
make install # → ~/.cargo/bin/ostk-recall
make install wraps cargo install --path crates/cli --locked --force.
make help lists every target; make version shows the installed version and
the workspace git sha.
Build-time native dependency (pulled in transitively, but the system tool must
be present):
[corpus]
root = "~/.local/share/ostk-recall"
[embedder]
model = "minishlab/potion-retrieval-32M" # repo id "org/name" — the prefix is required
[[sources]]
kind = "markdown"
project = "notes"
paths = ["~/notes"]
config.example.toml is a complete, commented
reference for every option — globals ([corpus], [embedder], [reranker],
[runtime], [lens], [weaver], [[record_rule]]), all source kinds, the
relational entity_type / edges scanner, and the [watch] block.
Quickstart
ostk-recall init # create the corpus root, download the model
ostk-recall scan # ingest the configured sources
ostk-recall serve # run the daemon: MCP over a local socket + ambient lens
ostk-recall serve --watch # daemon + in-process file-watcher (keeps the corpus fresh)
ostk-recall serve --stdio # direct stdio MCP transport (this process is the server)
ostk-recall connect # bridge a stdio MCP client to a running daemon
ostk-recall watch # standalone file-watcher → pokes a running daemon
ostk-recall weave # weave bulk-scanned content into the thread graph
ostk-recall verify # reconcile counts across store + manifest
ostk-recall optimize # compact and fold indexes
ostk-recall lens show # print the current ambient memory lens
ostk-recall lens status # inspect lens freshness and resource behavior
ostk-recall --help documents the full CLI; ostk-recall <verb> --help covers
each verb.
Examples
The examples/ directory has four self-contained, runnable
setups — each with sample content, a config.toml, a run.sh, and a README
walking the scan → query loop:
.ostk/threads/*.md files; tension state as metadata
one chunk per thread file
A markdown or code source can also set entity_type and edges to seed typed
concept nodes and authored edges from each file's frontmatter at scan time — see
examples/02-personal-kb and
config.example.toml.
MCP tools
Two agent-facing tools, callable from any MCP client — either by spawning
ostk-recall serve --stdio directly, or, when a shared daemon is running,
through the ostk-recall connect bridge. serve also exposes an MCP
resources surface serving the ambient ostk://memory-lens.
tools/list advertises exactly recall and remember. A canonical request
selects one operation with action, for example:
{"name":"recall","arguments":{"action":"search","scope":{"project":"my-project"},"query":"shared storage backend","limit":5}}
{"name":"remember","arguments":{"action":"record","scope":{"project":"my-project"},"kind":"fact","text":"Shared deployments use PostgreSQL.","subject":"storage.shared","predicate":"backend","value":"postgres","idempotency_key":"storage-shared-v1"}}
The boundary is intentional: recall reads memory; remember deliberately
changes it. Legacy recall_*, memory_*, attention_*, and thread_* names
remain dispatchable during the compatibility window but are hidden from
tools/list. Deprecated flat scope fields and nested params are likewise
accepted but not advertised; canonical calls use action, scope, and
action-specific fields.
recall — read and inspect
action
purpose
search
Hybrid lexical+dense retrieval over both corpus and durable assertions, with relevant conflicts. Claims use bounded passage embeddings plus the configured reranker; a long match returns only its relevant authored passage, with the complete record available through get. Embeddings are persisted, backfilled, and indexed before remember acknowledges a write. Diagnostics expose retrieval mode and lexical-only degradation. Actionless calls retain the deprecated legacy search shape.
get
Dereference a chunk, assertion, concept, or conflict.
surface
Orient from current focus and compact active assertions. Detailed concepts, open_loops, attention, focus, and threads views remain explicitly selectable.
discover
Multi-signal workstream discovery.
conflicts
Inspect open or historical structured conflicts.
synthesize
Collapse retrieval into named virtual-memory pages.
status
Corpus, ingest, memory, and diagnostic status.
audit
Capability-gated read-only audit query.
Recall may log access telemetry, but it never creates or revises semantic
memory. Every search response includes a conflicts array and explicit
conflict_coverage; unstructured code, prose, and conversation results are not
silently presented as contradiction-checked. Healthy per-call coverage stays
compact and points to the standing contract in recall(action="status");
retrieval mechanics live under diagnostics.claim_retrieval instead of being
repeated as conflict boilerplate.
Canonical calls return the complete envelope once in MCP structuredContent.
The required text content is only a short fallback summary, avoiding duplicate
context in clients that expose both fields. surface(view="now") includes
focus plus compact active assertions; it does not inject concept-activation or
thread-resonance rankings. Those remain available through their explicit views.
remember — intentional memory changes
action
purpose
record
Store a note, fact, decision, constraint, preference, procedure, observation, or open question.
split
Turn a broad synthesis into ordered atomic children linked part_of the parent and continues from one sibling to the next.
supersede
Replace a formerly valid assertion while preserving history.
retract
Mark an assertion as wrong or untrusted.
forget
Suppress an assertion from ordinary recall; retains an audited anti-resurrection tombstone.
restore
Restore an allowed suppressed or retracted assertion.
resolve
Record resolution metadata after member claims were retracted, forgotten, superseded, or corrected by reingest.
relate
Author typed claim relationships or legacy concept relationships.
focus
Set, restore, or clear working focus.
track
Create/update a workstream and its evidence.
consolidate
Run concept/evidence consolidation.
Facts and decisions may carry a structured subject, predicate, value, validity
interval, and source-evidence snapshot. Conflicts are deterministic when
structured claims share a scoped key but assert incompatible overlapping
values. Semantic claim retrieval is enabled, but semantic/NLI conflict
candidates over arbitrary prose remain intentionally deferred: similarity can
find an assertion, but it does not prove that two assertions contradict. See
the architecture decision.
recall(action="status") therefore reports semantic potential-conflict
nomination as unavailable rather than returning a misleading empty set.
Claims should be atomic enough to retrieve, correct, and supersede
independently. A longer synthesis is still an ordinary claim with its honest
epistemic kind—for example, a narrative grouping may be a note, while a
runbook synthesis remains a procedure. split creates atomic children and
durable child --part_of--> parent links; it does not invent a special
synthesis kind. Ordered children also carry previous/next continuity, so
sentence-safe projections do not lose the rest of a passage when one child is
retrieved. With the default keep_parent:true, grouping does not
supersede either side; suppressing the broad parent is an explicit lifecycle
choice (keep_parent:false). Search returns matching children with compact
relationship and continuity IDs, never the parent body hidden in result
metadata. Use recall(action="get", kind="claim") to expand the hierarchy and
traverse its complete ordered sequence.
The ostk-project decision adapter applies the same rule during ingest. A
compound .ostk/decisions.jsonl record becomes a structural parent plus
bounded outcome/rationale children; ordinary search returns the children and
keeps the parent available through hierarchy traversal. The next ordinary
incremental scan migrates the v0.9.3 monolithic projection only after those
children are durable—no destructive --reingest is required. Exact
same-key source revisions invalidate their predecessor before recording the
replacement. Older source-derived claims with trustworthy source timestamps
receive bounded ranking pressure, reported on the result, but age never changes
their truth state or silently chooses a winner.
Long assistant transcript messages use the same retrieval principle without
rewriting history. The original authored block remains byte-for-byte as an
addressable archival parent. Deterministic section children follow markdown
headings, paragraph topic shifts, and sentence boundaries; their dense input
also receives a bounded preceding user question, while stored and returned
text remains authored-only. Ordinary recall ranks the children and suppresses
the duplicate archival parent, but direct relationship lookup can still recover
the complete original block. At startup, the daemon additively backfills these
projections from retained corpus rows, including transcripts whose source JSONL
has rotated away. It never deletes or stales a parent and is idempotent on
replay.
{"action":"record","scope":{"project":"my-project"},"kind":"note","text":"Storage synthesis: SQLite is the local default; PostgreSQL is required for shared deployments.","idempotency_key":"storage-synthesis-v1"}
{"action":"split","scope":{"project":"my-project"},"id":42,"children":[{"kind":"fact","text":"SQLite is the local storage default.","subject":"storage.local","predicate":"backend","value":"sqlite"},{"kind":"constraint","text":"Shared deployments require PostgreSQL.","subject":"storage.shared","predicate":"backend","value":"postgres"}],"keep_parent":true,"reason":"separate independently retrievable assertions","idempotency_key":"storage-synthesis-v1-split"}
Ordinary record calls do not accept a free-form string evidence: use
structured support for an exact source coordinate, or remember.relate when
recording durable derivation/grouping provenance. This prevents ungrounded text
from being silently accepted and discarded.
Memory lens
ostk://memory-lens is a small, changing markdown portfolio aligned with
recently attended conversation. It combines bounded corpus evidence with a
bounded lane of active operator assertions and eligible source-derived claims.
It is the collective ambient view: read-only, shared by daemon clients, and
deliberately unaffected by any one caller's focus pin. Explicit focus still
shapes that caller's recall and memory surface.
The server publishes the resource; the MCP client controls context placement.
A conforming client discovers and reads it, subscribes, then re-reads after an
update notification:
resources/list
resources/read {"uri":"ostk://memory-lens"}
resources/subscribe {"uri":"ostk://memory-lens"}
# server later sends notifications/resources/updated
resources/read {"uri":"ostk://memory-lens"}
resources/unsubscribe {"uri":"ostk://memory-lens"}
The lens excludes harness/audit apparatus from ambient surfacing, collapses
duplicate derived excerpts, rotates recently surfaced chunks, and stays within
its configured token budget. Claim-ledger mutations independently invalidate
the resource generation and notify subscribers even when attention has not
drifted. Claim IDs and revisions are stable follow-up handles; rendering a claim
is observed, but use of any individual entry is not inferred. Excluded content
remains available to explicit recall. ostk-recall lens show reads the
persisted snapshot without becoming an MCP client; lens status reports its
freshness and lifecycle contract.
Salience health treats entropy collapse and measured active-vs-decided drift as
failures. “Surfaced without observed use” remains visible as an advisory: the
daemon audits successful Lens reads against the stable chunk and claim handles
in that rendered snapshot, but does not infer that the client used or benefited
from any individual entry. It therefore does not declare legitimate
transcript-derived concepts unhealthy from missing use feedback.
Connecting clients
connect never starts the server. The normal shared setup is one long-lived
ostk-recall serve --watch process plus any number of MCP clients configured to
spawn ostk-recall connect. The bridge injects the caller project, heals across
daemon restarts, replays the MCP handshake, and materializes paged results.
Alternatively, serve --stdio is a self-contained server owned by one client.
A .serve.lock admits only one server per corpus, so use either the shared
daemon (connect clients) or per-client serve --stdio, not both.
To join a shared daemon instead, replace the args with ["connect"].
ostk
ostk v6.0.0+ ships fcp-recall in its driver defaults. Its kernel-side legacy
mem_fault_recall adapter remains compatibility plumbing; agent-facing MCP
clients see the canonical recall and remember tools. Keep ostk-recall on
PATH. For earlier ostk, register via HUMANFILE:
DRIVER recall ostk-recall serve --stdio
Live updates
The daemon binds a scan-trigger socket at corpus.root/recall.sock (named pipe
\\.\pipe\ostk-recall-recall on Windows). ostk-recall serve --watch runs the
watcher in-process; alternatively run a separate ostk-recall watch. Either way,
edits under a configured source path are debounced and re-ingested without a
manual rescan:
At startup the watcher registers filesystem watches first, then catches up the
watched ostk-project audit journals before settling into live updates. Catch-up
waits for the shared scan lock and retries transient failures with bounded
backoff, closing the gap created by journal appends while the daemon was down.
The watcher status file reports startup_catchup_attempts,
startup_catchup_errors, startup_catchup_paths, and
last_startup_catchup.
As of the v0.6 TurnEnd gate, only watched conversation transcripts
(claude_code / gemini) drive the live attention runtime; other sources are
ingested and searchable but do not wake the observer or weaver.
Consolidation
serve handles live cognition (a watched transcript turn wakes the observer and
weaver). Offline consolidation is a separate, operator-scheduled step — run it
from cron / launchd / systemd, not inside serve:
weave --consolidate runs the full cycle over its --since window; the window
is the consolidation horizon, so pick it per schedule tier. The CLI is the only
contract — the scheduler is yours. Set HF_HUB_OFFLINE=1 for offline runs once
the model is cached.
Read path. The daemon serves MCP to many clients over a local socket; stdio
clients reach it through connect. serve --stdio is the alternative direct
transport for a client that prefers to spawn its own server.
Write path. The daemon binds recall.sock; a watcher debounces filesystem
events and delivers changed paths; a scan mutex serialises every trigger so
per-path ingest never overlaps the single writer.
Attention loop. On a watched transcript TurnEnd, TurnObserver emits membrane
chunks via Pipeline::ingest_synthetic and AutoWeaver writes derived evidence
links — both gate on IngestEvent::is_turn_end(), so bulk ingest is skipped.
ostk-recall weave runs the same weaver over bulk content on demand. IdleCurator
fades inactive threads on a timer. The InMemoryAttention score tier is
per-process and rebuilds from the threads.sqlite chain ledger on boot.
make check # fmt-check + clippy + tests (CI parity)
make test # tests only
make lint-strict # clippy with -D warnings
OSTK_RECALL_E2E=1 cargo test --workspace # network-enabled tests (model download)