-brightgreen?style=flat-square)
English | 简体中文 | 日本語 | العربية | Español | 한국어 | فارسی
Most AI search finds text in milliseconds and then asks a language model to write the answer. The model is what makes a RAG answer good, and it is also where the seconds and the per-question cost come from.
XERJ Query-1 works the other way. It reads your question and writes a short program (look this up, compare these,
count those), and the engine runs that program over facts you stored. Lookup and comparison steps point at a stored
fact and its quoted span; counts return a program hash, not the records. If the data cannot support an answer you get
UNKNOWN; if two facts disagree you get CONFLICT. It runs on four CPU cores, offline, with no embeddings to build
and no GPU.
Retrieval quality (R@k, nDCG@k) on BEIR, on SearchTome and on our 160-question test, the latency split and every data set: benchmarks/query-1-2026-11.
The model is off by default; without the flag there is no /_facts route and the MCP tool list is unchanged. rc.101
upgrades the data directory to format 4 on first write, and rc.100 cannot open it again. What it does not do yet,
measured: sums and totals (a sum question can come back as a member count), and large fact sets (on a 95,000-fact
namespace asks took 0.82 s at the median and all 20 test questions were declined). It declines more often than an LLM
would guess. The numbers and their caveats: blog post,
capabilities.
Try it with the model
curl -fsSL https://xerj.org/get | sh -s -- --with-model
Verify: the installer ends by printing the start line with the model directory it unpacked.
xerj --facts-model-dir ~/.local/share/xerj/model --insecure --data-dir ~/xerj-data &
curl -s localhost:9200/_facts | grep -o '"state":"loaded"'
Verify: prints "state":"loaded". In the first few seconds after start it prints nothing, because the model is still loading ("not_loaded"); run it again.
Write a fact and ask about it: docs/MODEL_QUICKSTART.md.
From an agent with no shell, the same calls work over HTTP and the MCP tool xerj_facts_ask;
see llms.txt.
XERJ is research in the open. The model was trained on donated GPU time; if it is useful to you, xerj.org/donate funds the next models. The engine is open source; the model ships as a binary bundle, and its training and evaluation data are not public yet.
The engine underneath
Local-first search engine and MCP server for AI agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search and agent memory, behind an Elasticsearch-compatible API. Works with Claude Code, Cursor and any MCP client.
XERJ is a local AI search engine that indexes any folder automatically, so your coding agent stops burning tokens reading files one by one and pulls the exact code it needs instead. Reference coding is its main use case and the clearest win: point an agent at a task and it downloads the open-source repos closest to it, indexes them, and reuses how they solved the problem before writing its own code. In a controlled study that cut a coding agent's output tokens by 2.7x at the same 16/16 solve rate (case study), and people report roughly 5x in everyday work (field reports). It is enough for a smaller, cheaper model to out-code a pricier one working from memory, while you spend less time in fix loops. Try it now with a one-prompt install:
Install XERJ (docs: https://xerj.org/llms.txt), index this project's sources, and set up
reference coding: clone and index the open-source repos closest to what we're building,
and search how they solved a problem before writing code.
Or run it yourself:
curl -fsSL https://xerj.org/get | sh
xerj --insecure --data-dir ~/xerj-data &
xerj init # wire XERJ into Claude Code/Cursor (MCP + skill), one command
xerj autoindex ~/my-project
xerj search "how do we rotate the WAL segment" # ranked passages, file:line
xerj def "euler_to_rotationmatrix" # go-to-definition: file:line + signature
xerj gain # what it did for you — counted, not estimated
For several IDs or questions at once, use xerj batch-search --input queries.json
or MCP xerj_batch_search: ordered results, citations and per-item errors.
Keep that data directory outside any folder you will autoindex. data/ is not in the
built-in ignore list, so --data-dir ./data inside a tree you then index makes the node
index its own storage.
At startup, Data dir shows the resolved absolute path, Data source identifies
the flag, selected config file or default, and Data status says whether this
launch created the directory. A new directory can make an older node's indices
or owner account appear missing: restart with the previous --data-dir or
--config. XERJ does not discover or migrate other data directories automatically.
Windows and a hand-verified install are under Install by hand.
That --insecure flag is a trust boundary, not a formality. It turns off TLS and API-key
auth, so every caller that can reach the port is a superuser who can read and delete every
index, including every /_memory namespace. It is meant for one user on a dev machine: a
fresh node binds 127.0.0.1 only, and a cleartext bind on any other interface refuses to
start unless you explicitly allow it. The full model is in
docs/SECURITY_MODEL.md.
Join the community
You join XERJ by running it and sending back one short field report on what it was like: a single commit or PR. That report is the membership card, and it is the only thing the project asks in return for a free binary (the engine source is Apache-2.0; the XERJ Query-1 crates in release binaries since rc.101 are proprietary). It is what keeps the engine improving for whoever runs it next.
Humans and AI agents are both welcome; this is deliberately not an agents-only project. Whether you indexed a repo over lunch or an agent installed XERJ mid-session in someone else's codebase, the ask is the same: one short report saying what you pointed it at, what you used it for, and your honest verdict. A negative verdict is a good report. If something broke, that is a separate issue or fix on top.
Field-report PRs are CLA-exempt, so they merge without a signature. The path that works on
any build is a plain gh pr create adding exactly one markdown file under
user-feedback/16-agent-field-reports/, with
nothing else in the diff. On a build that has it, xerj feedback drafts the report and
fills in version, OS and what was indexed, and xerj feedback --open-pr opens the PR for
you. Check xerj --help for whether your binary has the command.
The measured reason: reference coding
An agent that greps pulls whole files into its context, up to 1.06M input tokens on one
corpus in our measurements, and still has to read them. An agent that queries XERJ gets the
passage. Measured end to end on code the model had not memorised
(case study: 8 tasks, 4 languages,
16 runs per arm, real claude -p token counts):
That is 2.7x fewer output tokens than grep and 26x fewer than memory alone at the same solve rate, and up to 278x fewer on a single Java task. In real development, users report roughly 5x fewer tokens end to end (field reports). The value is gated by memorisation: it wins on private, internal, niche or post-cutoff code, and is neutral to harmful on popular public libraries the model already knows. The honest limits are in the case study.
Feed it any folder
Reference coding is one use of the same primitive: xerj autoindex <folder> makes an agent
know a corpus instead of grepping it. One command indexes code, docs, logs, PDFs, SQLite and
awkward CSVs into typed, queryable indices for search, RAG, security audits and agent
memory, with no schema to write and no pipeline to configure.
xerj --insecure --data-dir ~/xerj-data & # start it
xerj autoindex ~/my-project # point it at anything
--insecure again means no TLS and no auth — every caller a superuser — so this is the
same one-user-on-a-dev-machine boundary as above (security model).
XERJ sniffs every file, works out what it is, and creates one index per dataset it finds. Code arrives with its symbols and line numbers through tree-sitter, not as flat text:
phase A: 593 datasets inferred, 1955 junk/skipped files
phase B: indexing 25329 files with 8 workers
done in 158.1s, 593 datasets, 83103 records live, 790 junk records
Share an offline data map with xerj autoindex map --prefix myproject --html xerj-map.html --redact-paths, or use --md xerj-map.md for Markdown. Reports
include dataset counts, symbol names, graph hubs when available, skipped-file
information and suggested searches. Review metadata names before sharing.
Examples, privacy scope and reproduction.
What people point it at
- Reference coding: index the OSS projects nearest your problem and retrieve how they solved it before writing code (the measured use case above).
- Codebase Q&A and RAG: index a repo, ask for the mechanism, get the passage with
file:lineinstead of a directory listing. - Security audits: index a target tree and query for sink patterns, secrets and dangerous calls across every file type at once.
- Log and incident analysis: mixed formats become typed indices with the aggregations you would expect from Elasticsearch.
- Agent long-term memory:
/_memory/{namespace}stores what an agent learns and recalls it by meaning next session.
How the prompt works
llms.txt gives your agent the ordered steps: install, start the
server, xerj autoindex ., query with any Elasticsearch client, and the reference-coding
loop (clone similar OSS, index it, retrieve the mechanism before writing, cite what you use,
respect licenses).
The reason it helps: an agent working from memory retry-loops on any API it has not memorised, and grep only tells it where to look; the recovery is still reading source into context, up to 1.06M input tokens on one corpus in our measurements. An agent that queries XERJ reads the exact passage instead. The 2.7x fewer output tokens and 2.1x cheaper than grep at the same 16/16 solve rate (26x fewer tokens than memory alone, which solved 11 of 16) is in The measured reason above, with a companion run scoring 9/9 with retrieval versus 0/9 from memory on a Rust library the model had never seen.
More prompts that work on a fresh install:
- "Read https://xerj.org/llms.txt, set XERJ up as your search backend, index
./docs, and show me one example query per index it created." - "Run
xerj autoindex mapand tell me what is in this data (types, counts and the gotchas it recorded), then answer my questions with search instead of reading files." - "Use XERJ's
/_memory/notesAPI as your long-term memory for this project: store what you learn as you work, and recall it by meaning next session."
Worked, validated examples for each capability: xerj.org/docs/recipes.
Boot, bulk ingest, search, vector kNN, live dashboards, no cuts. Watch it on xerj.org or try the live playground.
Install by hand
curl -fsSL https://xerj.org/get | sh
Windows PowerShell:
irm https://xerj.org/get.ps1 | iex
Nix (builds from source; nix develop opens a shell with the Rust toolchain):
nix run github:xerj-org/xerj -- --version
One static binary, no JVM, no dependencies. Prebuilt for Linux, macOS and Windows on x86-64 and arm64. The x86-64 builds target x86-64-v2 (SSE4.2: Intel Nehalem 2008 and later, AMD Bulldozer and later) and use AVX2 and FMA at runtime when the CPU has them, so a pre-Haswell machine runs the same binary. It speaks the Elasticsearch API, so existing clients, dashboards and tooling work against it unchanged.
First commands after install (the installer prints where xerj landed; add it to your PATH
if needed): xerj --insecure --data-dir ~/xerj-data &, wait until http://localhost:9200
responds, then xerj autoindex ~/my-project. See Index a folder.
For a host with no runtime internet access, follow the air-gapped deployment recipe. The default lexical embedder is offline; neural mode needs the three model files staged locally before the first semantic operation.
Index a folder
Start the server, then point autoindex at anything:
xerj --insecure --data-dir ~/xerj-data & # local dev: no TLS, no auth
xerj autoindex ~/my-project
That is the dev-machine trust boundary from the top of this README — no TLS, no auth, every
caller a superuser. It is fine on a laptop because a fresh node binds 127.0.0.1 and
nothing else; a shared host wants the auth-on start just below.
If your server has auth on, which is the default for every start without --insecure
(including any start from a config file), hand autoindex the same key. It never picks the
key up from xerj.toml; pass --api-key or set XERJ_API_KEY, or every request comes back
401 Unauthorized:
xerj --data-dir ~/xerj-data & # auth on: key minted on first boot
export XERJ_API_KEY="$(cat ~/xerj-data/admin.key)" # <data_dir>/admin.key
xerj autoindex ~/my-project
That is the whole setup. There is no schema to write and no pipeline to configure. XERJ sniffs each file, works out what it is, and creates one index per dataset it finds:
phase A: 593 datasets inferred, 1955 junk/skipped files
phase B: indexing 25329 files with 8 workers
done in 158.1s, 593 datasets, 83103 records live, 790 junk records
Source files go through tree-sitter, so code arrives with its symbols and line numbers
instead of as flat text. CSV, JSON, JSONL, XML, YAML, SQLite, PDF, DOCX, PowerPoint
(.pptx, one record per slide), Excel (.xlsx, one dataset per sheet), EPUB books (one record per chapter), Jupyter notebooks
(one record per heading section, any kernel), man pages (roff, one record per section), HTML, mail (.eml and mbox mailboxes,
including a Google Takeout export) and common log formats are all handled. Unity projects get first-class treatment: text-serialized scenes,
prefabs and assets become one record per GameObject/Component, .meta files become a
GUID-to-path table, and MonoBehaviour records carry script_class/script_path so "which
scenes use this script?" is a single query (binary-serialized assets need Force Text to be
readable; generated dirs like Library/ are auto-skipped and recorded).
Search it
This is the Elasticsearch API, so you already know this part:
# what did it find?
curl localhost:9200/_cat/indices
# full-text
curl "localhost:9200/ax-*/_search?q=checkout+error"
# structured
curl localhost:9200/ax-orders/_search -H 'content-type: application/json' -d '{
"query": { "range": { "total": { "gte": 100 } } },
"aggs": { "by_status": { "terms": { "field": "status" } } }
}'
Vector and hybrid search use the same knn and semantic syntax you would send to
Elasticsearch. Any Elasticsearch client library works if you point it at localhost:9200.
Score results with Jev
A search score tells you which result is above which, but not whether a result really answers your question. Jev, a service from TypeSafe AI, does that: give it a question and a document and it returns a number from 0 to 1, how well the document answers. XERJ works with it in two ways.
1. Reorder your results with Jev (optional). Add "rerank": {} to a _search and XERJ sends
the top results to Jev, then sorts them by the numbers it gets back. On the public FiQA test,
result quality goes from 0.24 to 0.36 (nDCG@10), for about $0.64 in total for 648 searches.
Treat the numbers as a way to sort, not as exact percentages: they are not well calibrated.
This sends the text of those results to TypeSafe's servers. It stays off until you set a
key, and [rerank] enabled = false forbids it completely. Read
docs/RERANK.md before turning it on.
2. Get the same kind of answer locally. POST /v1/systemone and POST /_decide accept the
same questions in the same format, but XERJ answers them itself, by looking up the most similar
documents you have already labelled (for example messages already marked spam or not spam).
Nothing leaves your machine and no key is needed. It only works if you have labelled examples;
on AG News (120k labelled examples) it picks the right label 92% of the time. The official
jev-reranker Python client works against a XERJ node without changes
(test). XERJ answers under its own name, xerj-history-vote-1;
it is not Jev.
Measurements and how to reproduce them:
benchmarks/beir-hybrid and
benchmarks/decisions-as-retrieval.
Ask over your own facts
XERJ ships XERJ Query-1, a model that answers questions over facts you give it.
It is off by default; start a node with --facts-model-dir <DIR> to turn it on.
Write documents and facts to a namespace, then ask in plain language. The engine
returns a checked answer: lookups carry the facts they used and the program that
produced them; counts carry a program hash and a second-model agreement check, but
not the counted records.
# write a fact
curl -s localhost:9200/_facts/demo/_bulk -H 'content-type: application/json' -d '{
"documents":[{"doc_id":"d1","version":"v1","text":"Ana Bel is 170 cm tall."}],
"facts":[{"fact_id":"f1","subject":"Ana Bel","subject_id":"emp:1","relation":"height","kind":"number",
"value":"170","unit":"cm","doc_id":"d1","version":"v1","start":11,"end":14,"quote":"170"}]
}'
# ask
curl -s localhost:9200/_facts/demo/_ask -H 'content-type: application/json' \
-d '{"question":"How tall is Ana Bel?"}' | jq '{final_status,final_answer}'
# {"final_status":"ANSWER","final_answer":"170 cm"}
Which to use: _search for keyword and text lookups over indexed documents;
/_facts + _ask for a single-fact question over records you have written as
subject / relation / value facts. The model answers only over the facts[]
you wrote (kind is text, date, number or boolean; version is a
string; quote must equal text[start:end]) — documents alone answer UNKNOWN.
XERJ Query-1 runs on CPU with no GPU and nothing leaving the machine. Counting
questions ("how many people have office Berlin?") are served from an exact
executor, and only when every fact of the counted subjects carries a stable
subject_id ("emp:1"); without it the count is INCOMPLETE and the answer
is UNKNOWN. The executor is exact, but the model can pick the wrong set: in our
review an "or" question and a "total devices" question were both served as
verified counts of the wrong set. Set counts only on rc.101: numeric thresholds
decline, and a sum question about something the store has no facts for can be
served as a member count. It is a separate model from the search embedder. Get the bundle with
curl -fsSL https://xerj.org/get | sh -s -- --with-model, then start a node with
xerj --facts-model-dir <dir> --data-dir <dir> (XERJ_FACTS_MODEL_DIR=<dir> is
the environment equivalent); the model is a swappable xerj-model-bundle-v1
directory. Verify: curl -s localhost:9200/_facts | jq .model.state prints
"loaded" (the first seconds after start print "not_loaded" while the model
preloads; poll). On XERJ Score v2, our private development evaluation (1,600 questions
over MoMA, Tate, PyPI, OurAirports, USGS, NASA Exoplanet Archive and UCI Online Retail
records plus invented ones, the distribution the model was trained for), the shipped
model scores 90.29 where Opus 5.5 scores 100. It is not a test on unseen questions.
See docs/XERJ_QUERY_1.md for the seven
capabilities with a verified example each, docs/MODEL_QUICKSTART.md
to get it running and docs/FACTS_API.md for the full contract.
The model weights are a release asset, not part of this repository's source.
Connect an agent over MCP
Not every agent can run a shell command. Desktop assistants and function-calling hosts reach tools through the Model Context Protocol, and the binary you just installed is the MCP server. There is nothing else to download and nothing to compile:
xerj --insecure --data-dir ~/xerj-data & # 1. the node the tools query
xerj mcp # 2. MCP stdio server (your client runs this)
xerj mcp speaks MCP over stdio and proxies to the node named by XERJ_URL (default
http://localhost:9200). It does not start a node; step 1 is the prerequisite. The MCP
server adds no trust of its own — it is the same binary talking stdio to your client and
plain HTTP to the node, so the node's boundary is the one that counts, and step 1 above
boots the same --insecure dev node. Drop this into your MCP client's config:
{
"mcpServers": {
"xerj": {
"command": "/home/you/.local/bin/xerj",
"args": ["mcp"],
"env": { "XERJ_URL": "http://localhost:9200" }
}
}
}
Use an absolute path. The sh installer (Linux, macOS, and Windows Git Bash, MSYS and Cygwin) puts xerj in ~/.local/bin (xerj.exe on Windows; command -v xerj confirms) and does not edit PATH. The PowerShell installer (get.ps1) puts xerj.exe in %LOCALAPPDATA%\Programs\xerj and adds that directory to the user PATH ((Get-Command xerj).Source confirms; XERJ_NO_PATH=1 opts out). MCP hosts launched from a desktop icon do not inherit your
shell's PATH. If the node is running with auth (anything but --insecure), add
"XERJ_AUTH": "ApiKey <key>" alongside XERJ_URL; the key is in <data-dir>/admin.key.
Supported agents
Twenty-six clients can register xerj mcp as a local stdio server. The entry is always the same
two fields - the absolute path to the binary, and mcp as its only argument - but where that entry
lives and which key wraps it differ per client. tested means the server was registered in that
client and xerj_search appeared in its tool list; untested means the path and shape are
transcribed from the client's own documentation and nobody has run it yet. Six rows have been run.
OpenCode v2.1.2 on Windows against xerj v1.0.0-rc.97 answered initialize and tools/list
with its full tool list. Cline CLI 3.0.73 on Linux against xerj v1.0.0-rc.98 loaded this
section's mcpServers block at ~/.cline/data/settings/cline_mcp_settings.json. xerj_search
appeared in the tool list Cline handed the model (named xerj__xerj_search) among that binary's
20 tools, and Cline executed one xerj_search call. The model turn was a local OpenAI-compatible
stub, because no provider key was available; the MCP server was the real xerj mcp process. The
IDE's MCP Servers > Configure button was not part of that run. Kimi Code CLI 2.3.1 and the Kimi
desktop app on Linux against xerj v1.0.0-rc.99 (built from source) loaded this section's
mcpServers block at ~/.kimi-code/mcp.json: the client spawned xerj mcp as a child process
with the env block intact (verified through /proc/<pid>/environ), and xerj_search appeared
in the tool list. Claude Code 2.1.311 on Kali Linux against xerj v1.0.0-rc.100 registered
xerj mcp with claude mcp add in the local, project and user scopes; xerj_search appeared in
/mcp among that binary's 22 tools, and Claude Code executed one xerj_search and one xerj_def
call against an autoindexed repository. DeepSeek Harness 44.7.0 on Windows against
xerj v1.0.0-rc.99 loaded this row's - insert: block from a profile patch at
$DSH_HOME/profiles/desktop/cordis.patch.yml: in the contributor's run, the harness
spawned xerj mcp without an application restart, exposed mcp__xerj__xerj_search,
and executed a search. The tested entry was the profile patch; the dsh/ bundle
was not exercised. The run set XERJ_DISABLE_FEEDBACK=true and observed no automatic
field report. Contributor's verification and limits. Antigravity on Windows 11 (x86_64-pc-windows-msvc) against v1.0.0-rc.100 loaded this section's block at with : the client spawned , received 22 tools in (including and ), and executed one and one call against an autoindexed repository ().
Correcting an untested row is a welcome first PR.
Codex and the Grok CLI read TOML, so the entry is a table:
[mcp_servers.xerj]
command = "/abs/path/to/xerj"
args = ["mcp"]
OpenCode and Kilo Code take one command array, and Kilo uses the same shape one level shallower:
{"mcp": {"servers": {"xerj": {"type": "local", "command": ["/abs/path/to/xerj", "mcp"], "environment": {"XERJ_URL": "http://localhost:9200"}}}}
{"mcp": {"xerj": {"type": "local", "command": ["/abs/path/to/xerj", "mcp"]}}}
Continue and Goose read YAML:
# ~/.continue/config.yaml
mcpServers:
- name: xerj
command: /abs/path/to/xerj
args:
- mcp
# ~/.config/goose/config.yaml
extensions:
xerj:
type: stdio
cmd: /abs/path/to/xerj
args:
- mcp
enabled: true
Amp keeps the key namespaced, VS Code and Copilot need servers plus a transport, Zed calls them
context servers, and OpenClaw nests them under mcp:
{"amp.mcpServers": {"xerj": {"command": "/abs/path/to/xerj", "args": ["mcp"]}}}
{"servers": {"xerj": {"type": "stdio", "command": "/abs/path/to/xerj", "args": ["mcp"]}}}
{"context_servers": {"xerj": {"command": "/abs/path/to/xerj", "args": ["mcp"], "env": {}}}}
{mcp: {servers: {xerj: {command: "/abs/path/to/xerj", args: ["mcp"], transport: "stdio"}}}}
DeepSeek Harness loads one MCP server as one plugin in $DSH_HOME/cordis.patch.yml, or in a profile patch at $DSH_HOME/profiles/<profile>/cordis.patch.yml. The same block is the profile bundle in dsh/, and its command is xerj on PATH. If that file is only [], replace the array with the block below; appending under [] is not valid YAML. env and cwd are optional stdio fields (args defaults to [], env to {}, cwd to ""). When the node is not running --insecure, add XERJ_AUTH: "ApiKey <key>" under env:
# $DSH_HOME/cordis.patch.yml
- insert:
- id: mcp-xerj
name: '@deepseek-ai/dsh-mcp-client'
config:
serverName: xerj
transport: stdio
command: xerj
args:
- mcp
env:
XERJ_URL: "http://localhost:9200"
cwd: ""
Three candidates from the original list are deliberately absent. Aider has no MCP client support at
all (Aider-AI/aider#3314 is still open). T3 Code is
the other direction: it hosts an MCP endpoint other agents connect to, and its sessions inherit
their servers from the harness underneath. Grok Bot has no documented local stdio registration; the
Grok CLI does, which is the grok row above.
On main, 24 tools are exposed when Hub tools are enabled (25 with --facts-model-dir, which adds xerj_facts_ask); the rc.101 release has 23 and 24, because xerj_sources arrives in rc.102. Twelve are thin proxies over an endpoint XERJ already serves:
For xerj_search, pass {"prefix":"project","query":"retry policy"} to search
project-*, matching xerj search --prefix project. prefix is resolved by
the same function as the CLI: the prefix xerj autoindex --prefix stores, so
My_Proj searches my-proj-*. Use index for an explicit index name or
pattern instead; non-empty index and prefix cannot be combined. An empty
prefix, a prefix with no [a-z0-9] character, or a prefix that cannot name an
index is an error. With neither argument, the tool searches ax-*.
For new callers, query is a plain string and structured ES query-DSL goes in
query_dsl — {"index":"docs","query_dsl":{"match_all":{}}}. Do not combine
both fields. The published schema separates the types to simplify tool calls
(#2549); it cannot guarantee a model emits valid JSON. Existing callers may
still pass an object in query, and omitting both fields retains match_all
browsing. Prefer an explicit query_dsl clause when browsing is intended.
Search tools and xerj batch-search default to an 8,000 UAX #29 word budget
when max_tokens is omitted or null. A larger integer requests more evidence;
accounting explains any trimming. This is not a byte cap or a model/BPE token
limit. Errors and mandatory xerj_search staleness hints are outside this
budget and remain visible.
xerj_def finds local definitions with the same query, ranking and location deduplication
as xerj def --json. For example, {"symbol":"Cache","prefix":"project","kind":"struct","lang":"rust","k":5}
searches project-*; index can replace prefix, with the same scope rules as xerj_search.
symbol also accepts capability phrases. k must be 1–50 (default 5); optional max_tokens
caps the response. Results are ES JSON: declarations live in hits.hits[]._source.code,
locations in ax_path/path and line/start_line; older file hits may use fields._passage.
hits.total counts engine matches before deduplication. No match returns an empty hit list.
As in the CLI, kind/language filters are ignored when no target index maps the field
or any target index maps it as numeric. Ranking is lexical. This tool queries the configured node;
xerj_hub_def instead looks up definitions in a named Hub pack.
Two more are not plain proxies: xerj_map returns the per-index field map from the autoindex
catalog (trimmed to a byte cap, otherwise verbatim), and xerj_code_search runs the xerj code
reference-coding pipeline (tools/xerj-code/) and returns its rendered text.
xerj_index_budget counts a folder and estimates its index against the node's disk budget
before xerj autoindex hashes anything (it does not index). xerj_autoindex_remove drops
one autoindexed folder after listing it, and deletes nothing unless confirm is true.
xerj_status probes the node banner, cluster health and visible indices with
up to three bounded reads. A missing autoindex catalog does not mean the node
has no data; refused or malformed probes remain errors or unknown status.
xerj_batch_search retrieves ordered batches with per-item outcomes and citations.
xerj_corpus_feedback rates a hub corpus.
xerj_field_report files a field report, and xerj_hub_find, xerj_hub_def and xerj_hub_ask
answer from Corpus Hub packs already on the machine (the xerj ask flow).
xerj mcp --help prints the same config block and the full option list. The machine-readable
tool schemas are published at
xerj.org/docs/agents/schemas/mcp-tools.json,
generated from a live tools/list, never hand-written, and gated in CI
(scripts/mcp-schema-check.sh) plus a unit test, so the published list cannot drift from the
served one.
Why it exists
Agents burn their context window reading files. The PHP in WordPress core is about 5.2 million tokens, or 26 full context windows, so an agent cannot simply read it. Grep does not solve this either, because a grep hit is a line, and judging that line means opening the whole file.
Querying an index with passages and a projected _source costs kilobytes per question (a default _search returns whole documents) instead. In an AI security audit of WordPress core, an agent worked across 1,492 PHP files on roughly 26,000 tokens, which is what it takes to load about half a percent of the tree.
Use cases
- Reference coding: your coding agent retrieves how peer projects already solved it instead of re-deriving. Measured on four purpose-built libraries: 2.7x fewer output tokens than grep-driven coding (26x vs memory alone, which solved 11 of 16); users have told us ~5x in real development.
- Code search and security audits: AST-aware indexing, so an agent finds a function instead of a line.
- AI search and RAG: full-text, vector and hybrid retrieval in one query, with no separate vector database.
- Agent memory: durable recall with a knowledge graph over your own documents.
- Log analytics and observability: logs, metrics and traces in one engine.
- Elasticsearch replacement: same wire protocol, one binary.
Runnable examples live in recipes/ and docs/examples/.
Elasticsearch compatibility
XERJ implements the Elasticsearch REST API: indices, documents, bulk, search, aggregations,
mappings, kNN, scroll, reindex and the _cat endpoints. Kibana and the official client
libraries connect to it directly. One boundary worth knowing before you point export tooling
at it: scroll is a bounded up-front snapshot, not a segment-walking cursor, so a query whose
exact total exceeds the snapshot window is refused with a 400 rather than silently
truncated. Use search_after for result sets of any size
(the cap).
The conformance suite runs on every commit and currently passes 1444 of 1454 cases (0 failed, 10 skipped). It lives
in engine/tests/es-compat-yaml, and the remaining gaps are
listed there rather than hidden. XERJ is compatible with the API. It is not a
reimplementation of Elasticsearch internals, and it is not a fork.
Benchmarks
XERJ is benchmarked head to head against Elasticsearch 8.13.4 across ingest, full-text search, aggregations, vector search, and reads issued under a concurrent write flood. The latest closed-loop run scores 55 wins, 26 ties, 4 losses, including 1.72x ingest throughput and a 1.61x smaller on-disk footprint.
All four losses are the same gap: read p99 while a high-rate writer runs. It is
written up in full rather
than left out. Results and methodology are at
xerj.org/benchmarks and the harness is in
demo/playbooks (bench-matrix.mjs); the Elasticsearch start and runner
scripts still need to be published before anyone can rerun it end to end. Treat any number
you cannot reproduce with skepticism, including ours.
Code indexing
Source files are parsed with tree-sitter and indexed AST-aware — symbols plus full text — so an agent retrieves a function, not a line. On an Apache Lucene checkout (6,113 source files, 51 MB) single-threaded in-process extraction runs at ~1,500 files/s (13 MB/s); the tree-sitter parse is ~72% of that and is the per-file floor. Reproduce it with:
XERJ_EXTRACT_BENCH=/path/to/repo cargo test -p xerj-autoindex --release \
extract_bench -- --nocapture --ignored
Re-indexing is incremental: a file whose content is byte-identical to the last indexed generation skips the parse entirely and carries its committed result forward, so a re-run after editing a handful of files does work proportional to what changed, not to the whole repo — the common edit-and-rerun / CI case, typically over 95% unchanged. The full content hash still runs to detect changes, so total re-index time is floored by hashing, not parsing.
Build from source
You need a stable Rust toolchain.
git clone https://github.com/xerj-org/xerj
cd xerj/engine
cargo build --release -p xerj-server
./target/release/xerj --insecure --data-dir ~/xerj-data
The x86-64 release builds target x86-64-v2 (SSE4.2; set per target in
engine/.cargo/config.toml). The HNSW dot-product kernel also compiles an AVX2+FMA
variant and selects it at runtime when the CPU has both features. RUSTFLAGS overrides
that file, so a local build can opt into the host CPU:
RUSTFLAGS="-C target-cpu=native" cargo build --release -p xerj-server
To run the conformance suite against a running server:
cargo run --release -p es-yaml-runner -- --dir tests/es-compat-yaml/yaml
Documentation
- Guides and API reference
- Recipes for common tasks
- Roadmap: what ships today versus what is coming, verified against the release binary. Release-by-release view: milestones; live status: project board; standing pointer: pinned roadmap issue
- Changelog
Reference pages for individual subsystems, written against the source and including the limits each one does not lift:
- Second brain for the relationship layer over indexed documents:
the
/_graphroutes, evidence on links, the eight detectors and the two-hop cap. - Console reader for the console's corpus home, the reader for any record (an email with its attachments, a PDF page) and read-only guest mode, with the rendering rules that keep a hostile document inert and what it does not render.
- Scripting for the Painless subset, where scripts run, and the resource limits that bound them.
- Snapshot and restore for the supported subset of the snapshot API, and what restore replaces.
- Sharing for
xerj share: read-only guest access to one indexed folder through a link and a passcode, what a guest can and cannot reach, and the threat model. - Security model for authentication, the reserved
.xerj-memory-*namespace, API keys and what is not enforced. - Reranking for the optional
reranksearch stage: an external judge reorders the top hits by a 0-1 score that sorts well but is not calibrated (nDCG@10 measured on BEIR SciFact, NFCorpus and FiQA). It is the only search-time feature that sends document text off the node (proxy embeddings and the WAL tap also send text when an operator configures them; the page lists every outbound connection a node can open), it needs an operator-configured provider key, and its ranking quality with the real model is not verified. - The
/_factsAPI — request/response for writing facts and asking questions: the raw-versus-verified split (final_status/final_answer), the counting cascade, the revision/phase note, and the error set. - Get the model and serve it — install, obtain the
bundle, start the node with
--facts-model-dir, write a fact, ask, and verify, one step at a time. - Model bundles — the layout the engine loads, its env wiring, and the release asset it ships as.
Contributing
Pull requests are welcome from humans and AI agents alike; see
Join the community above. The baseline contribution is one short field
report: a plain gh pr create adding one markdown file under
user-feedback/16-agent-field-reports/.
Field-report-only PRs are CLA-exempt, so they merge without a signature.
For code changes, see CONTRIBUTING.md for the workflow and CLA.md for the contributor licence agreement, which is one signature per contributor rather than one per pull request. Agents opening a PR should state that an agent wrote it and follow AGENTS.md / .github/AI_CONTRIBUTIONS.md.
Bugs and feature requests go to GitHub issues.