We evaluated a code graph analysis MCP server on all 500 tasks from SWE-bench Verified using Claude Sonnet as the base agent. Key findings:
Metric
Baseline
MCP-Augmented
Change
Resolution Rate
49.8%
42.4%
-14.9%
Tool Calls
—
—
-42.3%
Tokens Used
—
—
-14.0%
Cost per Task
—
—
-15.2%
MCP tools alter the agent's exploration strategy, trading general-purpose search for opinionated shortcuts. The effect varies by codebase: the server helped on 1 of 12 repositories and hurt on 10, revealing an efficiency-resolution tradeoff that developers should evaluate before deploying MCP tools in production.
Methodology: Paired comparison experiments with Docker-isolated task environments, pinned dependencies, and identical model configurations. The only variable is the presence of MCP tools.
@software{newell2025mcpbr,
author = {Newell, Grey},
title = {mcpbr: Benchmarking Model Context Protocol Servers on Software Engineering Tasks},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.18627369},
url = {https://doi.org/10.5281/zenodo.18627369}
}
Supported Benchmarks
mcpbr supports 30+ benchmarks across 10 categories through a flexible abstraction layer:
MCPToolBench++ - Large-scale MCP tool use evaluation across 45+ categories. Tests tool discovery, selection, invocation, and result interpretation.
GSM8K - Grade-school math word problems testing chain-of-thought reasoning with numeric answer matching.
# Run SWE-bench Verified (default)
mcpbr run -c config.yaml
# Run any benchmark
mcpbr run -c config.yaml --benchmark humaneval -n 20
mcpbr run -c config.yaml --benchmark gsm8k -n 50
mcpbr run -c config.yaml --benchmark cybergym --level 2
# List all available benchmarks
mcpbr benchmarks
See the benchmarks guide for details on each benchmark and how to configure them.
Overview
This harness runs two parallel evaluations for each task:
MCP Agent: LLM with access to tools from your MCP server
Baseline Agent: LLM without tools (single-shot generation)
By comparing these, you can measure the effectiveness of your MCP server for different software engineering tasks. See the MCP integration guide for tips on testing your server.
Regression Detection
mcpbr includes built-in regression detection to catch performance degradations between MCP server versions:
Key Features
Automatic Detection: Compare current results against a baseline to identify regressions
Detailed Reports: See exactly which tasks regressed and which improved
Multi-Channel Alerts: Send notifications via Slack, Discord, or email
How It Works
A regression is detected when a task that passed in the baseline now fails in the current run. This helps you catch issues before deploying new versions of your MCP server.
# First, run a baseline evaluation and save results
mcpbr run -c config.yaml -o baseline.json
# Later, compare a new version against the baseline
mcpbr run -c config.yaml --baseline-results baseline.json --regression-threshold 0.1
# With notifications
mcpbr run -c config.yaml --baseline-results baseline.json \
--regression-threshold 0.1 \
--slack-webhook https://hooks.slack.com/services/YOUR/WEBHOOK/URL
Use Cases
CI/CD Integration: Automatically detect regressions in pull requests
Version Comparison: Compare different versions of your MCP server
Performance Monitoring: Track MCP server performance over time
Team Notifications: Alert your team when regressions are detected
Note: The npm package requires Python 3.11+ and the mcpbr Python package (pip install mcpbr)
via pip
# Install from PyPI
pip install mcpbr
# Or install from source
git clone https://github.com/greynewell/mcpbr.git
cd mcpbr
pip install -e .
# Or with uv
uv pip install -e .
Note for Apple Silicon users: The harness automatically uses x86_64 Docker images via emulation. This may be slower than native ARM64 images but ensures compatibility with all SWE-bench tasks.
Quick Start
Option 1: Use Example Configurations (Recommended)
Get started in seconds with our example configurations:
# Set your API key
export ANTHROPIC_API_KEY="your-api-key"
# Run your first evaluation using an example config
mcpbr run -c examples/quick-start/getting-started.yaml -v
This runs 5 SWE-bench tasks with the filesystem server. Expected runtime: 15-30 minutes, cost: $2-5.
Explore 25+ example configurations in the examples/ directory:
# Config is auto-created on first run
mcpbr run -n 1
# Or explicitly generate a config file first
mcpbr init
Edit the configuration to point to your MCP server:
mcp_server:
command: "npx"
args:
- "-y"
- "@modelcontextprotocol/server-filesystem"
- "{workdir}"
env: {}
provider: "anthropic"
agent_harness: "claude-code"
model: "sonnet" # or full name: "claude-sonnet-4-5-20250929"
dataset: "SWE-bench/SWE-bench_Lite"
sample_size: 10
timeout_seconds: 300
max_concurrent: 4
# Optional: disable default logging (logs are saved to output_dir/logs/ by default)
# disable_logs: true
Run the evaluation:
mcpbr run --config config.yaml
Infrastructure Modes
mcpbr supports running evaluations on different infrastructure platforms, allowing you to scale evaluations or offload compute-intensive tasks to cloud VMs.
Local (Default)
Run evaluations on your local machine:
infrastructure:
mode: local # default
This is the default mode - evaluations run directly on your machine using local Docker containers.
Azure VM
Run evaluations on Azure Virtual Machines with automatic provisioning and cleanup:
mcpbr includes a built-in Claude Code plugin that makes Claude an expert at running benchmarks correctly. The plugin provides specialized skills and knowledge about mcpbr configuration, execution, and troubleshooting.
Installation Options
You have three ways to enable the mcpbr plugin in Claude Code:
Option 1: Clone Repository (Automatic Detection)
When you clone this repository, Claude Code automatically detects and loads the plugin:
git clone https://github.com/greynewell/mcpbr.git
cd mcpbr
# Plugin is now active - try asking Claude:
# "Run the SWE-bench Lite eval with 5 tasks"
Best for: Contributors, developers testing changes, or users who want the latest unreleased features.
Option 2: npm Global Install (Planned for v0.4.0)
Install the plugin globally via npm for use across any project:
Note: The npm package is not yet published. This installation method will be available in a future release. Track progress in issue #265.
Best for: Users who want plugin features available in any directory.
Option 3: Claude Code Plugin Manager (Planned for v0.4.0)
Install via Claude Code's built-in plugin manager:
Open Claude Code settings
Navigate to Plugins > Browse
Search for "mcpbr"
Click Install
Note: Plugin manager installation is not yet available. This installation method will be available after plugin marketplace submission. Track progress in issue #267.
Best for: Users who prefer a GUI and want automatic updates.
Installation Comparison
Method
Availability
Auto-updates
Works Anywhere
Latest Features
Clone Repository
Available now
Manual (git pull)
No (repo only)
Yes (unreleased)
npm Global Install
Planned (not yet released)
Via npm
Yes
Yes (published)
Plugin Manager
Planned (not yet released)
Automatic
Yes
Yes (published)
What You Get
The plugin includes three specialized skills that enhance Claude's ability to work with mcpbr:
1. run-benchmark
Expert at running evaluations with proper validation and error handling.
Capabilities:
Validates prerequisites (Docker running, API keys set, config files exist)
Constructs correct mcpbr run commands with appropriate flags
Handles errors gracefully with actionable troubleshooting steps
Monitors progress and provides meaningful status updates
Example interactions:
"Run the SWE-bench Lite benchmark with 10 tasks"
"Evaluate my MCP server using CyberGym level 2"
"Test my config with a single task"
2. generate-config
Generates valid mcpbr configuration files with benchmark-specific templates.
Capabilities:
Ensures required {workdir} placeholder is included in MCP server args
Validates MCP server command syntax
Provides templates for different benchmarks (SWE-bench, CyberGym, MCPToolBench++)
Suggests appropriate timeouts and concurrency settings
Example interactions:
"Generate a config for the filesystem MCP server"
"Create a config for testing my custom MCP server"
"Set up a CyberGym evaluation config"
3. swe-bench-lite
Quick-start command for running SWE-bench Lite evaluations.
Capabilities:
Pre-configured for 5-task evaluation (fast testing)
Includes sensible defaults for output files and logging
Perfect for demonstrations and initial testing
Automatically sets up verbose output for debugging
Example interactions:
"Run a quick SWE-bench Lite test"
"Show me how mcpbr works"
"Test the filesystem server"
Benefits
When using Claude Code with the mcpbr plugin active, Claude will automatically:
Verify Docker is running before starting evaluations
Check for required API keys (ANTHROPIC_API_KEY)
Generate valid configurations with proper {workdir} placeholders
Use correct CLI flags and avoid deprecated options
Provide contextual troubleshooting when issues occur
Follow mcpbr best practices for optimal results
Troubleshooting
Plugin not detected in cloned repository:
Ensure you're in the repository root directory
Verify the claude-code.json file exists in the repo
mcpbr supports configurable timeouts for MCP server operations to handle different server types and workloads.
Configuration Fields
Field
Description
Default
startup_timeout_ms
Timeout in milliseconds for MCP server startup
60000 (60s)
tool_timeout_ms
Timeout in milliseconds for MCP tool execution
900000 (15 min)
These fields map to the MCP_TIMEOUT and MCP_TOOL_TIMEOUT environment variables used by Claude Code. See the Claude Code settings documentation for more details.
Example Configuration
mcp_server:
command: "npx"
args: ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"]
startup_timeout_ms: 60000 # 60 seconds for server to start
tool_timeout_ms: 900000 # 15 minutes for long-running tools
Common Timeout Values
Different server types require different timeout settings based on their operational characteristics:
Server Type
startup_timeout_ms
tool_timeout_ms
Notes
Fast (filesystem, git)
10000 (10s)
30000 (30s)
Local operations with minimal overhead
Medium (web search, APIs)
30000 (30s)
120000 (2m)
Network I/O with moderate latency
Slow (code analysis, databases)
60000 (60s)
900000 (15m)
Complex processing or large datasets
When to adjust timeouts:
Increase startup_timeout_ms if your server takes longer to initialize (e.g., loading large models, establishing database connections)
Increase tool_timeout_ms if your tools perform long-running operations (e.g., codebase analysis, file processing, AI inference)
Decrease timeouts for fast servers to fail quickly on connection issues
Custom Agent Prompt
You can customize the prompt sent to the agent using the agent_prompt field:
agent_prompt: |
Fix the following bug in this repository:
{problem_statement}
Make the minimal changes necessary to fix the issue.
Focus on the root cause, not symptoms.
Use {problem_statement} as a placeholder for the SWE-bench issue text. You can also override the prompt via CLI with --prompt.
Evaluation Parameters
Parameter
Default
Description
provider
anthropic
LLM provider
agent_harness
claude-code
Agent backend
benchmark
swe-bench
Benchmark to run (swe-bench, cybergym, or mcptoolbench)
By default, mcpbr consolidates all outputs into a single timestamped directory:
.mcpbr_run_20260126_133000/
├── config.yaml # Copy of configuration used
├── evaluation_state.json # Task results and state
├── logs/ # Detailed MCP server logs
│ ├── task_1_mcp.log
│ ├── task_2_mcp.log
│ └── ...
└── README.txt # Auto-generated explanation
This makes it easy to:
Archive results: tar -czf results.tar.gz .mcpbr_run_*
Clean up: rm -rf .mcpbr_run_*
Share: Just zip one directory
You can customize the output directory:
# Custom output directory
mcpbr run -c config.yaml --output-dir ./my-results
# Or in config.yaml
output_dir: "./my-results"
Note: The --output-dir CLI flag takes precedence over the output_dir config setting. This ensures that the README.txt file in the output directory reflects the final effective configuration values after all CLI overrides are applied.
Markdown Report (--report)
Generates a human-readable report with:
Summary statistics
Per-task results table
Analysis of which tasks each agent solved
Per-Instance Logs (--log-dir)
Logging is enabled by default to prevent data loss. Detailed execution traces are automatically saved to output_dir/logs/ unless disabled.
To disable logging:
# Via CLI flag
mcpbr run -c config.yaml --disable-logs
# Or in config file
disable_logs: true
Creates a directory with detailed JSON log files for each task run. Filenames include timestamps to prevent overwrites:
This is useful for debugging failed runs or analyzing agent behavior in detail.
JUnit XML Output (--output-junit)
The harness can generate JUnit XML reports for integration with CI/CD systems like GitHub Actions, GitLab CI, and Jenkins. Each task is represented as a test case, with resolved/unresolved tasks mapped to pass/fail states.
mcpbr run -c config.yaml --output-junit junit.xml
The JUnit XML report includes:
Test Suites: Separate suites for MCP and baseline evaluations
Test Cases: Each task is a test case with timing information
Failures: Unresolved tasks with detailed error messages
Properties: Metadata about model, provider, benchmark configuration
System Output: Token usage, tool calls, and test results per task
pipeline {
agent any
stages {
stage('Benchmark') {
steps {
sh 'pip install mcpbr'
sh 'mcpbr run -c config.yaml --output-junit junit.xml'
}
}
}
post {
always {
junit 'junit.xml'
}
}
}
The JUnit XML format enables native test result visualization in your CI/CD dashboard, making it easy to track benchmark performance over time and identify regressions.
Load Tasks: Fetches tasks from the selected benchmark (SWE-bench, CyberGym, or MCPToolBench++) via HuggingFace
Create Environment: For each task, creates an isolated Docker environment with the repository and dependencies
Run MCP Agent: Invokes Claude Code CLI inside the Docker container, letting it explore and generate a solution (patch or PoC)
Run Baseline: Same as MCP agent but without the MCP server
Evaluate: Runs benchmark-specific evaluation (test suites for SWE-bench, crash detection for CyberGym, tool use accuracy for MCPToolBench++)
Report: Aggregates results and calculates improvement
Pre-built Docker Images
The harness uses pre-built SWE-bench Docker images from Epoch AI's registry when available. These images come with:
The repository checked out at the correct commit
All project dependencies pre-installed and validated
A consistent environment for reproducible evaluations
The agent (Claude Code CLI) runs inside the container, which means:
Python imports work correctly (e.g., from astropy import ...)
The agent can run tests and verify fixes
No dependency conflicts with the host machine
If a pre-built image is not available for a task, the harness falls back to cloning the repository and attempting to install dependencies (less reliable).
The architecture uses Protocol-based abstractions for providers, harnesses, and benchmarks, making it easy to add support for additional LLM providers, agent backends, or software engineering benchmarks in the future. See the API reference and benchmarks guide for more details.
On ARM64 Macs, x86_64 Docker images run via emulation which is slower. This is normal. If you're experiencing issues, ensure you have Rosetta 2 installed:
Ensure the Claude Code CLI is installed and in your PATH:
which claude # Should return the path to the CLI
Development
# Install dev dependencies
pip install -e ".[dev]"
# Run unit tests
pytest -m "not integration"
# Run integration tests (requires API keys and Docker)
pytest -m integration
# Run all tests
pytest
# Lint
ruff check src/
Creating Releases
We use an automated workflow for releases. See the Release Guide for full details.
Quick start for maintainers:
# Patch release (bug fixes) - most common
gh workflow run release.yml -f version_bump=patch
# Minor release (new features)
gh workflow run release.yml -f version_bump=minor
# Major release (breaking changes)
gh workflow run release.yml -f version_bump=major
For AI agents: See the AI Agent Guide for a quick reference.
The workflow automatically:
Bumps version in pyproject.toml
Syncs version to all package files
Creates git tag and GitHub release
Triggers PyPI and npm publication
Roadmap
We're building the defacto standard for MCP server benchmarking! Our v1.0 Roadmap includes 200+ features across 11 strategic categories:
अभी टूल सूची नहीं है। रिमोट सर्वर को पहले हेल्थ चेक के बाद सूची मिलती है।
FAQ
सवाल
Mcpbr कैसे इंस्टॉल करें?
इस पेज पर Cursor, Claude Desktop, Claude Code या VS Code के लिए stdio कॉन्फ़िग कॉपी करें। यह पब्लिशर का इंस्टॉल इस्तेमाल करता है, PlugMCP प्रॉक्सी नहीं।
क्या Mcpbr के लिए API key चाहिए?
इस रिकॉर्ड में कोई ज़रूरी एनवायरनमेंट वेरिएबल दर्ज नहीं है।
Mcpbr कौन सा ट्रांसपोर्ट इस्तेमाल करता है?
Mcpbr stdio पर लोकल चलता है।
Mcpbr की पिछली जाँच कब हुई थी?
पिछली जाँच 2026-10-27 को हुई। Benchmark your MCP server.