接入你的智能体
從發布者安裝。不是 PlugMCP 端點。
遠端 URL 還是本機指令? 兩種安裝方式的差異
Connect AI agents and engineers to Apache Spark History Server for intelligent job analysis, performance monitoring, and investigation
[!IMPORTANT]
✨ NEW — Spark History Server CLI is now available
A standalone Go binary that queries Spark History Server directly from your terminal — no MCP, no AI framework, no daemon process. Inspect jobs, compare runs, investigate failures, and script against the Spark REST API.
This project provides two interfaces to your Spark History Server data:
shs) — For Engineers & ScriptsA standalone Go binary. Query your Spark History Server directly from the terminal, shell scripts, or CI/CD pipelines. Also works as a skill for coding agents like Claude Code and Kiro.
# Auto-detect latest version, OS, and architecture
VERSION=$(curl -s https://api.github.com/repos/kubeflow/mcp-apache-spark-history-server/releases | grep -m1 '"tag_name": "cli/' | cut -d'"' -f4 | sed 's|cli/||')
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m)
[ "$ARCH" = "x86_64" ] && ARCH="amd64"
[ "$ARCH" = "aarch64" ] && ARCH="arm64"
curl -sSL "https://github.com/kubeflow/mcp-apache-spark-history-server/releases/download/cli%2F${VERSION}/shs-${VERSION}-${OS}-${ARCH}.tar.gz" | tar xz
sudo mv shs /usr/local/bin/
# Generate a config file
shs setup config > config.yaml # then set your Spark History Server URL
# Explore applications
shs apps
shs jobs -a APP_ID --status failed
shs stages -a APP_ID --sort duration
shs compare apps --app-a APP1 --app-b APP2
# Use as a skill with Claude Code or Kiro
shs setup skill > ~/.claude/skills/spark-history.md
CLI documentation for full usage, or check out a real-world example of Claude Code comparing two TPC-DS 3TB benchmark runs.
An MCP (Model Context Protocol) server that exposes Spark History Server data as tools for AI agents. Agents query your Spark infrastructure using natural language — the server handles tool selection, multi-server routing, and structured data retrieval.
Use the MCP server when you want an AI agent to conduct multi-step investigations, synthesize findings across tools, or answer natural-language questions about your Spark applications.
The server supports MCP protocol versions 2026-07-28 and 2025-11-25 (and earlier) at the same time. It negotiates the version with each client, so no configuration is needed.
# Run directly with uvx (no install needed)
uvx --from mcp-apache-spark-history-server spark-mcp
# Or install with pip
uv tool install mcp-apache-spark-history-server
spark-mcp
The package is published to PyPI.
Register the server with a single command. Both examples run it over stdio via uvx.
With no config file present, the server defaults to a Spark History Server at http://localhost:18080;
point it elsewhere with a config file or SHS_SERVERS__LOCAL__URL.
Claude Code (claude mcp add):
claude mcp add --env SHS_MCP__TRANSPORT=stdio --env SHS_SERVERS__LOCAL__URL=http://localhost:18080\
--transport stdio spark-history \
-- uvx --from mcp-apache-spark-history-server spark-mcp
Kiro CLI (kiro-cli mcp add):
kiro-cli mcp add --name spark-history --command uvx \
--args --from --args mcp-apache-spark-history-server --args spark-mcp \
--env SHS_MCP__TRANSPORT=stdio --env SHS_SERVERS__LOCAL__URL=http://localhost:18080
Verify in either client with claude mcp list / kiro-cli mcp list, then ask the agent to "list the available Spark applications."
The server also ships prompts — guided, multi-step workflows you run as a command. In Claude Code: /mcp__spark-history__investigate_failure <app_id>. In Kiro CLI: /prompts investigate_failure (or @investigate_failure). See Prompts for the full list and arguments.
The commands above have two layers: the client's own options and the arguments/environment forwarded to spark-mcp. spark-mcp itself takes a single flag, --config / -c; everything else is set through SHS_* environment variables.
For example, to point at a remote Spark History Server with an explicit config file:
# Claude Code
claude mcp add --env SHS_MCP__TRANSPORT=stdio --transport stdio spark-history \
-- uvx --from mcp-apache-spark-history-server spark-mcp --config ~/.config/spark-mcp/config.yaml
# Kiro CLI
kiro-cli mcp add --name spark-history --command uvx \
--args --from --args mcp-apache-spark-history-server --args spark-mcp \
--args --config --args ~/.config/spark-mcp/config.yaml \
--env SHS_MCP__TRANSPORT=stdio
Basic configuration below. Create a file named config.yaml:
servers:
local:
default: true
url: "http://your-spark-history-server:18080"
auth: # optional
username: "user"
password: "pass"
include_plan_description: false # include SQL plans by default (default: false)
mcp:
transport: "streamable-http" # or: stdio
port: "18888"
debug: false
The server looks for its config file in the following order and uses the first one it finds:
--config / -c flag (e.g. spark-mcp --config /path/to/config.yaml)SHS_MCP_CONFIG environment variable./config.yaml in the current working directory~/.config/spark-mcp/config.yaml (honors $XDG_CONFIG_HOME when set)If none exist, the server starts with built-in defaults that can be overridden by SHS_* environment variables. When a path is given explicitly via the flag or SHS_MCP_CONFIG but the file is missing, the server fails fast instead of falling back.
Tip for MCP clients: when the server is launched by an MCP client (Claude Desktop, Kiro, etc.), the working directory is not guaranteed, so a
./config.yamlmay not be found. Prefer--config/SHS_MCP_CONFIG, or place the file at~/.config/spark-mcp/config.yaml.
Configurations can be overridden with environment variables. Nesting levels are
separated by a double underscore (__), so field names and server names may
themselves contain single underscores (e.g. SHS_SERVERS__MY_SERVER__URL maps
to servers.my_server.url).
SHS_MCP__PORT Port for MCP server (default: 18888)
SHS_MCP__TRANSPORT Transport mode: streamable-http or stdio
SHS_MCP__DEBUG Enable debug mode (default: false)
SHS_MCP__ADDRESS Bind address (default: localhost)
SHS_SERVERS__*__URL URL for a specific server
SHS_SERVERS__*__AUTH__USERNAME
SHS_SERVERS__*__AUTH__PASSWORD
SHS_SERVERS__*__AUTH__TOKEN
SHS_SERVERS__*__VERIFY_SSL
SHS_SERVERS__*__SSL_CA_CERT Path to a PEM CA bundle for a specific server
SHS_SERVERS__*__TIMEOUT
SHS_SERVERS__*__EMR_CLUSTER_ARN
SHS_SERVERS__*__INCLUDE_PLAN_DESCRIPTION
Configure multiple Spark History Servers and route queries to specific ones:
servers:
production:
default: true
url: "http://prod-spark-history:18080"
auth:
username: "user"
password: "pass"
staging:
url: "http://staging-spark-history:18080"
Agents can target a specific server per query:
"Get application
<app_id>from the production server"
For a private CA or TLS-inspecting proxy, keep verify_ssl: true and point
ssl_ca_cert to a mounted PEM bundle:
servers:
local:
ssl_ca_cert: "/etc/ssl/custom-ca/ca-bundle.pem"
Set the same path with SHS_SERVERS__LOCAL__SSL_CA_CERT. The CA certificates
in the bundle are added to the container's default trust roots, so include the
issuing CA and intermediates.
graph TB
subgraph Clients
A[🤖 AI Agent / LLM]
B[👩💻 Engineer / Script / CI]
C[🔧 Coding Agent - Claude Code / Kiro]
end
subgraph "Kubeflow Spark AI Toolkit"
D[⚡ MCP Server]
E[🛠️ CLI - shs]
end
subgraph "Spark History Servers"
F[🔥 Production]
G[🔥 Staging / Dev]
end
A -->|MCP Protocol| D
B -->|Terminal commands| E
C -->|shs skill file| E
D -->|REST API| F
D -->|REST API| G
E -->|REST API| F
E -->|REST API| G
Automatically available when AWS credentials and region are configured. See IAM setup guide.
get_job_bottlenecks + list_stages + compare_job_performancelist_jobs + get_stagecompare_job_performance + compare_job_environmentslist_sql_executions + get_sql_execution + compare_sql_executionsBeyond tools, the server exposes MCP prompts — reusable, multi-step investigation workflows that your agent runs as a single command. Each prompt expands into a guided sequence of tool calls (which the agent still executes one at a time), so you get a consistent, evidence-driven analysis without having to remember the right tool order.
serveris optional: when omitted, the prompt searches every configured server for the application(s). Passserver="<name>"only to target a specific server or disambiguate an id that exists on more than one.
MCP prompts surface as slash commands with the format /mcp__<server>__<prompt> (note the double underscores), where <server> is the name you registered the server under (spark-history in the examples above). Type / to discover them, then pass arguments space-separated after the command:
# Investigate a failed application
/mcp__spark-history__investigate_failure spark-cc4d615a5e6b4500b8eb1e9deb48cb4e
# Target a specific configured server
/mcp__spark-history__investigate_failure spark-cc4d615a5e6b4500b8eb1e9deb48cb4e production
# Compare two applications
/mcp__spark-history__compare_applications spark-app-A spark-app-B
List and run prompts with the /prompts command, or type @ then Tab to autocomplete. Run a prompt by name and supply its arguments:
# Open the prompt picker (shows prompts from all MCP servers)
/prompts
# Run a prompt directly
/prompts investigate_failure
# Or use the @ shortcut
@investigate_failure
Kiro CLI prompts you for arguments interactively when the prompt declares them, so you can run /prompts investigate_failure and provide app_id (and optionally server) when asked.


Deploy the MCP server using Helm:
helm install spark-history-mcp ./deploy/kubernetes/helm/mcp-apache-spark-history-server/
# Production configuration
helm install spark-history-mcp ./deploy/kubernetes/helm/mcp-apache-spark-history-server/ \
--set replicaCount=3 \
--set autoscaling.enabled=true
See deploy/kubernetes/helm/ for full configuration options.
When deployed in Kubernetes, connect Claude Desktop via mcp-remote:
kubectl port-forward svc/spark-history-mcp 18888:18888
git clone https://github.com/kubeflow/mcp-apache-spark-history-server.git
cd mcp-apache-spark-history-server
# Install Task runner
brew install go-task # macOS; see https://taskfile.dev/installation/ for others
# MCP Server
task install # install Python dependencies
task start-spark-bg # start Spark History Server with sample data
task start-mcp-bg # start MCP server
task start-inspector-bg # open MCP Inspector at http://localhost:6274
task stop-all
# CLI
cd skills/cli
task build # build ./bin/shs
task test # unit tests
task test-e2e # e2e tests (starts/stops Docker SHS automatically)
task start-shs # start SHS with CLI e2e sample data
Using this project? Add your organization to ADOPTERS.md and help grow the community.
See CONTRIBUTING.md for guidelines.
Apache License 2.0 — see LICENSE.
Built for use with Apache Spark™ History Server. Not affiliated with or endorsed by the Apache Software Foundation.
Connect your Spark infrastructure to AI agents and engineers
Built by the community, for the community 💙
MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.
工具
尚無工具清單。遠端伺服器會在首次健康檢查後取得工具清單。
常見問題
在本頁複製適用於 Cursor、Claude Desktop、Claude Code 或 VS Code 的 stdio 設定。它使用發布者的安裝方式,而不是 PlugMCP 代理伺服器。
這筆記錄未列出必要的環境變數。
MCP Apache Spark History Server 透過 stdio 在本機執行。
上次檢查時間:2026-11-17。 MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.