Agenten stärken
Installation vom Herausgeber. Kein PlugMCP-Endpunkt.
Host a copy with our keys and logs.
Noch kein paketiertes Release in der Registry. Folgen Sie der README des Repositorys.
Remote-URL oder ein Befehl auf Ihrem Rechner? Worin sich die beiden Installationen unterscheiden
Auf PlugMCP ausführen
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
StartenWebShift is a Rust library and MCP server that shifts noisy web pages into clean, right-sized text for LLM consumption.
Raw HTML is mostly junk: scripts, ads, navigation menus, cookie banners, tracking pixels. Feeding it directly to an LLM floods the context window with tens of thousands of useless tokens and leaves no room for reasoning. WebShift strips all that noise, sterilizes the text, and enforces strict size budgets so the model receives only the content that matters.
Depending on the features you enable, WebShift can be four things:
text-map does preserve the DOM structure for text rewriting use cases.)Question
|
+- (optional) LLM query expansion -> multiple search variants
|
+- Search via backend (SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing, HTTP)
|
+- Deduplicate + filter binary URLs
|
+- Streaming fetch with per-page size cap
|
+- HTML cleaning -> plain text (noise elements, scripts, nav removed)
|
+- Unicode/BiDi sterilization
|
+- BM25 deterministic reranking
| +- (optional) LLM-assisted tier-2 reranking
|
+- Budget-aware truncation across all sources
|
+- (optional) LLM Markdown summary with inline citations
|
+- Structured JSON output
For a detailed explanation of each pipeline stage, BM25 parameters, adaptive budget allocation, and real compression metrics see Under the Hood. For the full configuration reference (TOML, env vars, CLI args) see Configuration. For ready-to-use examples see Use Cases.
cargo install webshift-mcp
The binary is called mcp-webshift.
cargo install --path crates/webshift-mcp
# Full pipeline (search + fetch + clean + rerank)
webshift = "0.2"
# Cleaner + fetcher only (no search backends)
webshift = { version = "0.2", default-features = false }
# Text-map only (extract/replace text nodes in HTML)
webshift = { version = "0.2", default-features = false, features = ["text-map"] }
# Everything including LLM features
webshift = { version = "0.2", features = ["llm"] }
The easiest option is SearXNG — free, self-hosted, no API key:
docker run -d -p 8080:8080 searxng/searxng
No Docker? Use a cloud backend — see Search backends.
{
"mcpServers": {
"webshift": {
"command": "mcp-webshift",
"args": ["--default-backend", "searxng"]
}
}
}
That's it. The agent now has webshift_query, webshift_fetch, and webshift_onboarding.
For client-specific setup see docs/integrations/.
webshift_query parametersResolution order (highest priority first):
--default-backend, --brave-api-key, etc.WEBSHIFT_* prefixwebshift.toml (current dir, then ~/webshift.toml)[server]
max_query_budget = 32000 # total char budget across all sources
max_result_length = 8000 # per-page char cap
max_total_results = 20 # hard cap on results per call
max_download_mb = 1 # streaming cap per page (MB)
search_timeout = 8 # seconds
results_per_query = 5
oversampling_factor = 2
adaptive_budget = "auto" # "auto" | "on" | "off" — budget allocation mode
[backends]
default = "searxng"
[backends.searxng]
url = "http://localhost:8080"
[backends.brave]
api_key = "BSA-..."
[backends.tavily]
api_key = "tvly-..."
[backends.exa]
api_key = "..."
[backends.serpapi]
api_key = "..."
engine = "google" # google | bing | duckduckgo | yandex
[backends.google]
api_key = "..."
cx = "..." # Custom Search Engine ID
[backends.bing]
api_key = "..."
market = "en-US"
[backends.http]
url = "https://my-search.example.com/api/search"
query_param = "q"
count_param = "limit"
results_path = "data.items" # dot-path to results array in JSON response
title_field = "title"
url_field = "link"
snippet_field = "description"
[backends.http.headers]
"Authorization" = "Bearer my-token"
[llm]
enabled = false
base_url = "http://localhost:11434/v1" # OpenAI-compatible
api_key = ""
model = "gemma3:27b"
timeout = 60
expansion_enabled = true
summarization_enabled = true
llm_rerank_enabled = false
For every setting with all three config methods (TOML, env vars, CLI args)
and plain-language descriptions, see the full Configuration Reference.
Ready-to-use config examples are in Use Cases and examples/.
WEBSHIFT_DEFAULT_BACKEND=searxng
WEBSHIFT_SEARXNG_URL=http://localhost:8080
WEBSHIFT_BRAVE_API_KEY=BSA-xxx
WEBSHIFT_GOOGLE_API_KEY=xxx
WEBSHIFT_GOOGLE_CX=xxx
WEBSHIFT_BING_API_KEY=xxx
WEBSHIFT_LLM_ENABLED=true
WEBSHIFT_LLM_BASE_URL=http://localhost:11434/v1
WEBSHIFT_LLM_MODEL=gemma3:27b
All opt-in — disabled by default, no data leaves your machine unless enabled.
Cross-language normalization (bonus): when BM25 reranking surfaces pages in foreign languages (e.g. Chinese, Japanese, Arabic), the LLM summarizer still produces the final report in the prompt language. The agent receives clean, readable output regardless of the language mix in the source pages.
Works with any OpenAI-compatible API (OpenAI, Ollama, vLLM, LM Studio, etc.):
[llm]
enabled = true
base_url = "http://localhost:11434/v1"
model = "gemma3:27b"
Always active — the core value proposition:
use webshift::{Config, clean, fetch, query};
// Clean raw HTML — cap output at 8000 chars
let result = clean("<html><body><p>Hello world</p></body></html>", 8000);
println!("{}", result.text);
// Pass 0 to disable the per-page cap entirely (no truncation)
let full = clean("<html><body><p>Hello world</p></body></html>", 0);
assert!(!full.truncated);
// Fetch and clean a single page
let config = Config::default();
let page = fetch("https://example.com", &config).await?;
// Full search pipeline
let results = query(&["rust async programming"], &config).await?;
for source in &results.sources {
println!("[{}] {} — {} chars", source.id, source.title, source.content.len());
}
// Backend partial failures (CAPTCHA, rate-limits, engine outages) are surfaced
// in `warnings` rather than as Err. Empty sources + non-empty warnings means
// "all backends blocked" — distinct from a legitimate "no matches found".
if results.sources.is_empty() && !results.warnings.is_empty() {
eprintln!("all backends failed: {:?}", results.warnings);
}
Extract text nodes, manipulate them (translate, rewrite, simplify), and rebuild the HTML with structure, attributes, and links intact.
use webshift::{extract_text_nodes, replace_text_nodes, TextReplacement};
let html = r#"<p>Hello <a href="https://example.com">world</a></p>"#;
let map = extract_text_nodes(html);
// map.nodes = [(0, "Hello"), (1, "world")]
let replacements = vec![
TextReplacement { id: 0, text: "Ciao".into() },
TextReplacement { id: 1, text: "mondo".into() },
];
let result = replace_text_nodes(html, &replacements).unwrap();
// → <p>Ciao <a href="https://example.com">mondo</a></p>
// href untouched, tags intact, only text changed.
Requires features = ["text-map"]. See Use Cases #11 for a full translation example.
WebShift is in beta. Core functionality is stable and the server is used daily, but the API surface may still change before 1.0.
Feedback is very welcome. If something doesn't work as expected, behaves oddly, or you have a use case that isn't covered:
Bug reports, configuration questions, and feature requests all help shape the roadmap.
Contributions are welcome! Please see CONTRIBUTING.md for detailed guidelines on:
MIT License — see LICENSE for details.
Need help? Check the documentation or open an issue on GitHub.
Rust MCP server for denoised web search — fetch, clean, and rerank web content for AI agents.
Tools
Noch keine Tool-Liste. Remote-Server erhalten eine nach dem ersten Health-Check.
FAQ
Kopieren Sie auf dieser Seite die stdio-Konfiguration für Cursor, Claude Desktop, Claude Code oder VS Code. Sie nutzt die Installation des Herausgebers, keinen PlugMCP-Proxy.
In diesem Eintrag ist keine erforderliche Umgebungsvariable aufgeführt.
Webshift läuft lokal über stdio.
Zuletzt geprüft: 2026-11-10. Rust MCP server for denoised web search — fetch, clean, and rerank web content for AI agents.