MCP Failure Lab
Reproduce MCP timeouts, cancellation races, transport loss, and invalid responses with repeatable tests and CI reports.
Documentation · Compatibility · Project page
Example output (duration varies):
$ npx mcp-failure-lab demo
MCP Failure Lab — Demo
Running a real 500ms delay scenario...
Scenario: Deterministic delay demo
Outcome: success
Duration: ~500 ms
Assertions: passed
Why this exists
Real MCP clients behave differently when things break. A timeout may leave a connection usable; an interrupted response may close it. A malformed reply may be rejected by one client and accepted by another.
Failure Lab makes those cases repeatable so you can check both the failed call and what happens next.
Real-world findings
- Python SDK #3522: Python
mcp2.2.0 stayed closed after an interrupted HTTP response, rejected the next request, and raised anExceptionGroupduring cleanup. - Rust SDK #1283:
rmcp3.4.0 accepted an invalid response containing bothresultanderrorover stdio and HTTP. - Ruby SDK #589:
mcp1.6.1 accepted responses missingjsonrpcor declaringjsonrpc: "1.0"over stdio and HTTP in all three repeats. The Ruby report records all 60 executions; the results and upstream finding are included in the Observatory export. - Java SDK 2.0.1 accepted
jsonrpc: "1.0"over both transports (#1156). Over stdio, responses missingjsonrpcor containing bothresultanderrorleft the nextpingtiming out (#1157). See the Java report. - Duplicate-response comparison: all five tested SDKs completed the next
ping. TypeScript reported the duplicate through its error callback; the other harnesses surfaced no call-level duplicate error.
The SDK comparisons describe specific tested versions, not every release. See the versioned reports for reproduction steps and limitations.
Quick start
Requires Node.js 22.19.0 or newer and npm.
npx mcp-failure-lab demo
The demo needs no API key, external server, or global installation. To test your own MCP client, connect it to Failure Lab over stdio or local Streamable HTTP:
npx mcp-failure-lab serve
# Or:
npx mcp-failure-lab serve --transport http
The stdio process waits for a client; it is not an interactive terminal command. Configure your
client to launch it, or follow the getting started guide.
Press Ctrl+C to stop a manually started server.
HTTP defaults to http://127.0.0.1:3000/mcp. It provides neither authentication nor TLS;
do not expose it to an untrusted network.
Test your own MCP server
External targets support HTTP and stdio. After the
repository setup,
install the official GitHub MCP server with its executable on PATH and set
GITHUB_PERSONAL_ACCESS_TOKEN in your environment. Then run:
npm run dev -- run examples/scenarios/github-get-me.json \
--target examples/targets/github-stdio.json
The example calls GitHub's read-only get_me tool. Its target configuration uses envFrom
to pass the token from your environment rather than storing it in JSON.
For your own scenario and
target configuration, use:
npx mcp-failure-lab run scenario.json --target target.json
External runs check tool results, deadlines, and adapter setup and cleanup. They do not inject faults into another server; Failure Lab is not a proxy.
See the external targets guide for prerequisites, credential handling, and configuration.
What Failure Lab can break
The built-in server exposes these tools for testing client behavior:
| Tool | Behavior |
|---|---|
ping | Returns a deterministic health response |
protocol_ping_liveness | Sends a bounded protocol ping during its in-flight call |
delay | Waits for a bounded duration before returning |
hang | Remains pending until the client cancels |
disconnect | Interrupts the active transport while a request is in flight |
malformed_message | Violates one selected JSON-RPC response rule exactly once |
duplicate_response | Sends the same JSON-RPC response twice for one request |
response_after_cancellation | Sends one late response for a cancelled stdio request |
session_loss | Invalidates the caller's legacy HTTP session |
oversized_response | Returns a deterministic text payload up to 256 KiB |
invalid_progress | Sends an unknown, regressing, or late progress notification |
Protocol ping is distinct from the ping tool. Built-in scenarios default to MCP 2026-07-28;
protocol liveness requires scenario protocolVersion: "2025-11-25".
Both public transports accept legacy clients.
See the fault tools reference for timing bounds, activation, cancellation, cleanup, and transport limits.
oversized_response accepts sizeBytes from 0 through 262,144. It generates only ASCII x
characters, so the text size is exact in UTF-8 bytes and contains no external data. The cap
applies on both stdio and HTTP. Choose a size below your client's configured response limit to
check success, then a size above that limit (but no more than 262,144) to check client rejection.
The included 64 KiB scenario checks the exact
returned text size and a follow-up request. The JSON-RPC and transport framing add bytes around
the text, so a client's whole-message limit may trigger before sizeBytes reaches its threshold.
The integration test confirms that a stdio client with a 4 KiB message limit accepts a 1 KiB
payload and rejects an 8 KiB payload.
invalid_progress requires a client progress token and accepts unknown-token,
regressing-progress, or after-completion. These are protocol notifications associated with
one tool call; they are not tool result content. The unknown-token variant sends one notification
with a different token. The regressing variant sends progress 2 followed by 1. The late variant
sends one notification after the matching response on stdio or in the HTTP response stream.
These violate request-token correlation, monotonic progress, and the completed-request lifecycle,
respectively.
Each variant is bounded and consumed once. See the
regressing,
unknown-token, and
after-completion scenarios.
The built-in runner can assert callback values with expect.progressValues; external target
adapters do not report progress notifications and cannot use that assertion.
How recovery testing works
Trigger a fault, record its outcome, then make a second request on the same connection. A detected fault does not prove recovery; the follow-up must succeed too.
Scenario observe calls run after the primary call, including errors and timeouts.
For example, duplicate-response.json
calls duplicate_response, then uses ping to check that the connection remains usable:
# From a repository checkout
npm run dev -- run examples/scenarios/duplicate-response.json
This checks post-fault behavior, not an automatic recovery policy. A stdio disconnect terminates the server process and requires a new process and connection.
See Scenarios for outcome, duration, result, and observer assertions.
SDK / transport evidence
MCP Failure Observatory collects SDK and transport evidence. The repository reports preserve tested versions, methods, and limitations.
Results vary by SDK, version, transport, and protocol. A passing run is evidence for that combination, not a guarantee for every client.
CI usage
Save a scenario file with explicit expectations and timeouts, then produce a JUnit report:
npx mcp-failure-lab run scenario.json --report junit > junit.xml
Pin the Failure Lab package version in CI so upgrades do not change the test environment unexpectedly.
Exit codes: 0 means expectations passed, 1 means the scenario could not be loaded or executed,
and 2 means an assertion failed. An expected timeout can pass; an unexpected success can fail.
See Reporting for JSON, JUnit, and lifecycle diagnostics.
Documentation
Detailed guides live at mcplab.dev: Getting started · CLI · Architecture · Troubleshooting.
Contributing
See CONTRIBUTING.md for setup, tests, and the contribution workflow. Planned work is tracked in GitHub Issues.
If Failure Lab helps you test an MCP integration, consider starring the repository.