Model-Agnostic Wide Reranker (slm-rerank)
The Wide Reranker is an ultra-high-throughput, prefill-dominant semantic filter and reranker for codebases and document collections. Originally designed for Liquid Foundation Models (LFM 2.5 8B-A1B), it is now completely AI model agnostic, supporting:
- LFM 2.5 (ChatML with
<think>\n</think>\nbypass) - Qwen (2.5 / 3 / 3.8 / QwQ)
- Gemma (2 / 3 turn-based formats)
- RWKV (v5 / v6 / v7 raw completion RNN format)
- Any OpenAI-compatible provider (vLLM, Ollama, llama.cpp, TabbyAPI)
Key Architectural Highlights
1. Model / Provider Adapter Architecture (slm_rerank/adapters.py)
- Base
ModelProfile/ProviderAdapterinterface:name: identifier (e.g.'lfm','qwen','gemma','rwkv','openai')format_prompt(query, chunk_content, file_path, symbol, intent) -> strextract_yes_no_logprobs(response_payload) -> Tuple[float, float, bool]length_normalization_exponent: model-specific tuning (LFM 0.15, Qwen 0.12, Gemma 0.12, RWKV 0.10)default_base_url: default local port (LFM :8034, Qwen :8033, Ollama :11434, RWKV :8000)
- Built-in profiles:
LFMProfile: ChatML with<think>\n</think>\nbypass.QwenProfile: Qwen ChatML<|im_start|>without think tags (unless QwQ).GemmaProfile: Gemma format (<start_of_turn>user\n...<start_of_turn>model\n).RWKVProfile: Clean raw completion format optimized for RNN/linear state attention.GenericOpenAIProfile: Compatible with vLLM, Ollama, llama.cpp, TabbyAPI.
2. Dynamic Token String & Vocabulary Discovery
- Hardcoded integer token IDs are eliminated; token text matching works directly from
top_logprobs(e.g.token.strip().lower() in ('yes', 'no')). - Automatically probes
/tokenizeat startup to dynamically discover model-specific Yes/No token IDs, falling back to normalized text matching.
3. Model Isolation in Persistent SQLite Cache
- SQLite cache key is isolated by active model profile: $$\text{cache_key} = \text{sha256}(f"{\text{model_id}}:{\text{normalized_query}}:{\text{chunk_hash}}")$$
- Stored locally at
~/.cache/slm-rerank/cache.db. - Switching between LFM, Qwen, Gemma, or RWKV guarantees zero score cross-contamination.
5. Multi-Port Auto-Discovery (8033–8040)
- Scans active models across dedicated ports
8033..8040(e.g. 8033 Qwen, 8034 LFM 2.5, 8035-8040 others) in parallel (< 100ms). - Seamlessly resolves target port based on requested model profile or active hardware.
- Scans ports on a single host, never the network. The default host is loopback, so a model server on another machine is found only when you name it — see Remote & Multi-Machine Setup. When nothing answers, discovery reports why instead of returning an endpoint that is not there.
6. Smart Ripgrep Candidate Discovery
- Runs when no paths or globs are passed:
slm-rerank -q "auth token"finds its own candidates. - Recall is a ranked union of two signals — terms matching a file's path and terms matching its body — never one gated behind the other. A file named after what you asked for stays reachable even when a dozen test files mention the same words.
- Candidates are ranked by how many distinct terms hit (a path hit counts double), with test files ranked below implementations, then truncated to the limit. Ordering is by relevance, not directory traversal.
- A path term matches whole tokens, split on separators and camelCase.
sharedoes not matchshared/, andlaydoes not matchreplay. - Query terms are stemmed with each stem kept beside its root (
migration→migrat,migrate;chunking→chunk;classes→class), so a term cap never severs a stem from the word it came from. - Requires
rg(ripgrep) orgiton PATH; ripgrep is preferred and also reaches untracked files.
7. AST "Ghost Stubs" / Context Skeletons (--stub / --slice)
- Drastically reduces frontier model context window consumption: preserves imports, types, and the target chunk while collapsing non-relevant sibling functions into 1-line stubs (
folded N lines).
8. Architecture-Aware Boundary Slicing (--by-slice)
- Automatically clusters reranked code by architectural domain slice (
features/<slice>,modules/<slice>,packages/<slice>).
9. Lexical Prior on the Final Score
- Tier-1 scores every candidate lexically. A query term that names the file or a symbol it declares counts most, then a directory match, then mentions in the body.
- The final
scoreadds that evidence to the model's verdict in logit space:logit(score) = logit(rawScore) + z. Herezis the chunk's lexical score, standardized over the scored chunks and clipped to ±3. A spread below 3 points counts as noise. rawScorestays the model's own score, and a chunk the model could not score stays at 0.- The prior costs no model call.
Installation & Quickstart
Option A: npm / npx (Zero Python, Zero PyTorch — Ideal for Laptops & Remote LAN)
You can run slm-rerank directly on any machine with Node.js >= 20 without installing Python:
# Direct execution via npx (points to local or LAN GPU endpoint)
npx slm-rerank -q "user authentication token" -e "http://<gpu-host-ip>:8034/v1" src/**/*.ts
# Or install globally
npm install -g slm-rerank
Option B: Python (Rich Terminal Table & Local Hardware Server)
cd /home/shawry/Documents/GitHub/slm-rerank
pip install -e . --break-system-packages
MCP Server (Claude Code & other MCP clients)
slm-rerank ships a zero-dependency, pure Node.js MCP server speaking JSON-RPC 2.0 over
stdio — no Python, no mcp SDK, no PyTorch required.
# Run the server directly (stdin/stdout JSON-RPC)
npx slm-rerank --mcp
# Register it with Claude Code
claude mcp add slm-reranker -- npx -y slm-rerank --mcp
# Point it at a remote LAN GPU box
claude mcp add slm-reranker -- npx -y slm-rerank --mcp --host 192.168.1.50
It exposes one tool, rerank_codebase, which auto-discovers a live SLM endpoint across
ports 8033–8040 and returns ranked file:line citations plus a JSON candidate manifest:
Query Expansion (opt-in, --expand)
Deterministic recall matches words and their stems. It reaches migrate from migration,
but it will never reach bridge from harness — and a file whose code says buildBridge
is invisible to a query that says "interop harness".
--expand (CLI) or "expand": true (MCP) asks the local model for the vocabulary the query
is missing, then feeds those terms into candidate discovery alongside the literal ones:
slm-rerank --query "interop harness" --expand
# 🧠 Expanded query with: communication, protocol, serialization, wrapper, bridge, adapter, ...
# → finds src/zz-adapter.ts, whose body says buildBridge and marshalRow
It is strictly additive and strictly optional:
- Expanded terms are weighted at half a literal term, so a synonym can add a file to the candidate set but never displace one the query named outright.
- Every failure — no server, timeout, malformed answer — returns no terms, and recall proceeds on the deterministic list. Expansion can widen recall; it can't break it.
- Decoding is greedy (
temperature: 0), so a query always yields the same terms. Results are cached for 30 days in~/.cache/slm-rerank/expansions.json, shared between the Node and Python implementations.
Measured effect
On ten queries against this repository where the deterministic path already finds the target file: 1 improved, 9 unchanged, 0 worse. Expansion is not a general win — it is a rescue for vocabulary mismatch. On a fixture repo where the target shares no words with the query, the same query goes from not found at all to rank #1:
Cost is one model call, roughly 280ms on an 8B LFM, plus a few extra searches (~40ms).
Known limitation: the model sometimes answers with prose or reads a word differently than you meant ("harness" as test harness). Generic and stop words are filtered out, and the half-weight keeps the rest from doing damage, but expansion quality is bounded by the model.
Fast answers and the usage log (v0.10.0)
--fast: the lexical ranking, with no model call
--fast returns the Tier-1 lexical order and never asks for a model server. It is for a planning
agent that needs a list of files. It prints 10 results unless -k gives another count.
slm-rerank -q "where is the S3 presigned URL signed" --fast -k 5
--rerank-top: the model scores the head only
The model scores the first 16 chunks of the lexical order, not all of the chunks that Tier-1 keeps.
--rerank-top <n> changes the count. --full still scores every kept chunk. The fused score judges
the head against every kept candidate, the same as the prompt hook does.
The slice cache
The slices of each file are stored by the SHA-1 of its content, under chunks/ in the user cache
folder. The cache is never in the working tree of the searched repository, so its gates never see
it.
SLM_RERANK_CACHE_DIR overrides the folder.
The usage log and slm-rerank stats
Each CLI query and each prompt-hook decision appends one JSON line to usage.jsonl in the cache
folder. A line holds the time, the query (its first 500 characters), the caller, the latency, and
the top paths. A hook line also holds the isCodeQuestion decision and its outcome.
slm-rerank stats # query count, median latency by caller, share of hook prompts that queried
slm-rerank stats --json
Measured on quality-control-mono
The query set is 10 questions whose answer files were checked by hand, such as "where does the router set app.tenant_id". A hit is an answer file among the top 5 results. The local server is LFM2.5-8B-A1B Q4_K_M on an RTX 4060.
Cold means an empty slice cache. The 0.9.1 cold query also woke the model from idle sleep, which adds about 2.6 s; the 0.10.0 cold queries ran on a loaded model. Warm means a loaded model, a full slice cache, and a query that the server has not seen. The server keeps prompt states, so a repeated query costs about 1.3 s.
The slice cache saves about 50 ms of a --fast query. The rg searches of discovery take most of
the 0.4 s that remains.
The Claude Code prompt hook
slm-rerank/hooks/claude-prompt-context is a UserPromptSubmit hook. It adds file:line citations
only when isCodeQuestion(prompt) is true: a source path, a code-form symbol, error text, a code
fence, or a question that locates code. It stays silent on a prompt about process, status, or
tools. A local shim imports it from the pinned package. The shim's own exit guards the import
only, so the package owns the deadline and logs a query that reaches it:
const importGuard = setTimeout(() => process.exit(0), 5000);
try {
const { runPromptHook } = await import("slm-rerank/hooks/claude-prompt-context");
clearTimeout(importGuard);
await runPromptHook();
} catch {
process.exit(0);
}
Grounded code review (v0.8.0)
The reranker answers which chunk is relevant with a calibrated probability. It writes no prose, so it cannot invent a finding. A generative model can, and a review that reports an invented finding is worse than no review.
slm-rerank-review closes that gap. It retrieves with the same calibrated scorer, asks the model
for candidate findings through the native /completion endpoint only (/v1/chat/completions
applies a chat template and is not usable here), then checks every candidate against the physical
file before reporting it.
What is guaranteed, and what is not
Three checks run, in order. The first two are deterministic:
- Evidence gate. The quoted evidence must be a line of the cited chunk. A fabricated quote is dropped. A multi-line quote collapsed onto one line still matches.
- Identifier gate. Every code identifier the finding names must exist in the cited chunk. A
finding about
onSkipin a file that has noonSkipis dropped. - Support gate (best effort). The calibrated binary scorer judges whether the evidence proves the claim, through a dedicated judge prompt rather than the retrieval prompt.
- Contradiction gate (best effort veto). A second judge asks whether the code contradicts
the claim. A high score vetoes the finding. This catches a real case the support gate missed:
"the state variable is not initialized" against
useState(0).
Every failure path returns fewer findings rather than raising, so a missing or confused model degrades the review; it never fabricates one.
Measured on the local 8B model
The semantic gates are a filter, not a guarantee. On a five-case labeled set (SEMANTIC_CASES in
review_bench.py), the local LFM2.5 judge rejected every case: precision stayed 100% but recall
fell to 0%. Support probabilities were near 0.02 for true and false claims alike, so the score
cannot separate them. The contradiction judge shows weak separation only (false 0.56 / 0.31 /
0.22 against true 0.36 / 0.15).
Treat the semantic gates as advisory until a stronger judge model is available. The evidence and identifier gates are the deterministic guarantee.
Usage
# Review files or globs, with all three gates
slm-rerank-review "frontend/src/**/*.tsx" --query "find real defects" --top 5
# Evidence and identifier gates only, no semantic judge
slm-rerank-review src/app.ts --no-support --no-contradiction
# Deterministic gates plus the semantic judges (default)
slm-rerank-review src/app.ts --top 5
# Machine-readable report
slm-rerank-review src/app.ts --json
Reasons a candidate is not reported are stable strings: EMPTY_EVIDENCE, EVIDENCE_TOO_SHORT,
EVIDENCE_NOT_FOUND, CLAIM_SYMBOL_NOT_IN_CHUNK, CLAIM_NOT_SUPPORTED, SUPPORT_UNVERIFIED,
CHUNK_UNREADABLE.
Trust benchmark
slm-rerank-review-bench # deterministic, no model needed
slm-rerank-review-bench --live # also run retrieval and generation against the local model
The offline corpus carries findings the local model actually produced during the quality-control-mono guide-tour review, each with evidence that is absent from the target file. The benchmark proves the deterministic gates report every supported finding and drop every fabricated one:
Reported precision is 100% with the gates and 18.18% without them.
Remote & Multi-Machine Setup
Discovery probes ports on one host. Running the reranker on a laptop while the GPU box
runs llama-server elsewhere means pointing it at that box explicitly — nothing is
auto-detected across the network.
On the machine serving the model, bind to all interfaces (not just loopback):
llama-server -m LFM2.5-8B-A1B-Q8_0.gguf --host 0.0.0.0 --port 8034
On the client machine, set one environment variable:
# Full endpoint — skips port scanning entirely
export SLM_ENDPOINT=http://192.168.1.220:8034/v1
# Or just the host — ports 8033-8040 are then scanned on that box
export SLM_HOST=192.168.1.220
For Claude Code, put it in the MCP registration itself:
claude mcp add slm-reranker -e SLM_ENDPOINT=http://192.168.1.220:8034/v1 -- npx -y slm-rerank --mcp
# Equivalent, scanning the remote port range instead of pinning one endpoint
claude mcp add slm-reranker -- npx -y slm-rerank --mcp --host 192.168.1.220
Environment Variables
Both the Node and Python implementations read the same variables, in this order:
A --base-url / -e argument beats both. A pinned endpoint URL beats --host, and is used
as-is even when a specific --model is requested, on the assumption that you named the
server deliberately.
CLI Usage
Model Selection & Auto-Detection
# Auto-detect model from default port 8034 (LFM)
slm-rerank --query "auth middleware" src/**/*.ts
# Explicitly target Qwen on port 8033
slm-rerank --query "auth middleware" src/**/*.ts --model qwen --base-url http://localhost:8033/v1
# Target Gemma on Ollama
slm-rerank --query "find cache key" src/**/*.py --model gemma --base-url http://localhost:11434/v1
# Target RWKV on port 8000
slm-rerank --query "linear attention state" src/**/*.py --model rwkv -e http://localhost:8000/v1
Piping from Git / Find
git ls-files "*.py" | slm-rerank --query "brier score calibration" --threshold 0.65
Cache Management
# Bypass cache for fresh live evaluation
slm-rerank --query "auth" src/*.py --no-cache
# Clear cache entries
slm-rerank --clear-cache --query "auth" src/*.py
Configuration File (~/.config/reranker/config.yaml)
model: qwen
base_url: http://localhost:8033/v1
endpoints:
lfm: http://localhost:8034/v1
qwen: http://localhost:8033/v1
gemma: http://localhost:11434/v1
rwkv: http://localhost:8000/v1
openai: http://localhost:11434/v1
concurrency: 4
timeout: 45.0
Python API
import asyncio
from slm_rerank import Reranker, QwenProfile
async def main():
# Model-agnostic initialization (auto-detects or uses explicit profile)
reranker = Reranker(model="qwen", base_url="http://localhost:8033/v1")
response = await reranker.rerank(
query="progress trigger handler",
candidates=["backend/src/features/progress/trigger.ts"],
threshold=0.65,
)
for item in response.results:
print(f"[{item.score*100:.1f}%] {item.file_path}:{item.citation.start_line} ({item.symbol})")
if __name__ == "__main__":
asyncio.run(main())
