MST (MCP Server Tester)
Testing, evals, and conformance for servers. MST is published as , and its CLI is .
Power your agent
Yayıncının kendi komutu. PlugMCP uç noktası değildir.
Host a copy with our keys and logs.
npx -y @gleanwork/[email protected]Claude Code'un yerel komutlar için kurulum bağlantısı yok. Aşağıdaki komutu kopyalayın ve bir terminalde çalıştırın.
claude mcp add gleanwork-mcp-server-tester-github -- npx -y @gleanwork/mcp-server-tester@1.1.1Bir terminalde çalıştırın.
{ "mcpServers": { "gleanwork-mcp-server-tester-github": { "command": "npx", "args": [ "-y", "@gleanwork/[email protected]" ] } }}claude_desktop_config.json
Uzak URL mi, yoksa makinenizde bir komut mu? İki kurulum arasındaki fark
PlugMCP’de çalıştır
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
Başlat@gleanwork/mcp-server-testermstWrite deterministic Playwright tests against your MCP tools, or run data-driven eval datasets — including LLM-based evaluation of tool discoverability.
The mcp Playwright fixture connects to your MCP server (stdio or HTTP) and exposes a high-level API for calling tools and asserting responses. Custom matchers keep assertions readable.
import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';
test('read_file returns file contents', async ({ mcp }) => {
const result = await mcp.callTool('read_file', { path: '/tmp/test.txt' });
expect(result).toContainToolText('Hello, world');
expect(result).not.toBeToolError();
});
test('server exposes required tools', async ({ mcp }) => {
const tools = await mcp.listTools();
expect(tools.map((t) => t.name)).toContain('read_file');
});
Playwright tests are fast, deterministic, and designed for CI. Use them for regression testing, schema validation, and protocol conformance. The framework includes built-in conformance checks for the MCP spec. mst generate records tool calls as a spec file you can edit (see CLI).
Available matchers:
Eval datasets define cases as JSON: an input the client under test acts on, and assertions about what it did. Run them with runEvalDataset() in a Playwright test, on the mst client, or with mst run on any client (Claude Code, Cowork, ChatGPT). Checks on a single tool's response belong in Playwright tests, above.
{
"name": "file-ops",
"cases": [
{
"id": "read-config",
"input": "What version is set in the app's config file?",
"assertions": {
"toolsTriggered": {
"calls": [
{
"name": "read_file",
"required": true
}
]
},
"containsText": "1.0.0"
}
},
{
"id": "summarize-readme",
"input": "Summarize the README in two sentences",
"assertions": {
"toolsTriggered": {
"calls": [
{
"name": "read_file",
"required": true
}
]
}
},
"judges": [{ "type": "rubric", "rubric": "groundedness" }]
}
]
}
import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';
import { loadEvalDataset, runEvalDataset } from '@gleanwork/mcp-server-tester';
test('file operations eval', async ({ mcp }, testInfo) => {
const dataset = await loadEvalDataset('./data/evals.json');
const result = await runEvalDataset(
{ dataset, client: 'mst', model: 'claude-haiku-4-5', defaultTrials: 5 },
{ mcp, testInfo }
);
expect(result.passed).toBe(result.total);
});
Supported assertions:
Judges go in the case's judges list, beside assertions: each scores the answer against a rubric or reference, such as { "type": "rubric", "rubric": "groundedness" } or a plugin judge "acme/judge/completeness".
In LLM client mode, a real LLM receives your server's tool list and a natural language prompt, then decides which tools to call. This tests whether your tool names, descriptions, and input schemas are clear enough for autonomous use — a different question from whether the tools return correct output.
A case with input runs on the client. In a Playwright test, runEvalDataset names the client and model ({ dataset, client: 'mst', model: 'claude-haiku-4-5' }), and the mst client uses the test's MCP connection; an eval config names them with client and model. A case can set its own client, model and clientOptions.
{
"id": "find-config",
"input": "Find the application config file and return its contents",
"assertions": {
"toolsTriggered": {
"calls": [
{
"name": "read_file",
"required": true
}
]
}
}
}
LLM client mode makes real API calls and produces non-deterministic results. Use trials to run a case multiple times and measure pass rate rather than expecting 100% on a single run. See the LLM Client Guide for configuration and cost management.
Requires Node.js 22+. These docs describe 2.0, which is in beta:
npm install --save-dev @gleanwork/mcp-server-tester@beta @playwright/test
npm install @gleanwork/mcp-server-tester installs 1.x, whose docs are at v1.1.1. Upgrading: Migration guide (v1.x → v2.0).
LLM-as-judge with the default Anthropic judge needs the Anthropic SDK:
npm install --save-dev @anthropic-ai/sdk
The SDK client's packages (ai and the @ai-sdk/* providers) are optional dependencies, installed by default; add them yourself if you install with --omit=optional.
npx @gleanwork/mcp-server-tester@beta init
The CLI wizard creates a playwright.config.ts, example tests, and a sample eval dataset configured for your server. See the CLI Guide for all options.
Point the framework at your MCP server in playwright.config.ts:
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './tests',
reporter: [['list'], ['@gleanwork/mcp-server-tester/reporters/mcpReporter']],
projects: [
{
name: 'my-server',
use: {
mcpConfig: {
transport: 'stdio',
command: 'node',
args: ['server.js'],
},
},
},
],
});
For HTTP servers, set transport: 'http' and serverUrl. For servers that require OAuth, see the Transports Guide and CLI Guide for authentication setup, including CI/CD token management.
Plain stdio is supported by Claude CLI, Vercel AI SDK, ChatGPT Work/Codex, and Cowork evaluation clients, including Cowork on macOS and Linux. Cowork also supports private file-backed native proxies and mixed HTTP/stdio server sets. See evaluation client transport support for client-specific fields, environment rules, and setup ownership.
Install AI skills to help your coding assistant write tests and evals, and improve tool metadata:
npx skills add -g gleanwork/mcp-server-tester
This installs skills globally so they're available across all your projects. Four skills are included:
Compatible with Claude Code, Cursor, Windsurf, Copilot, and 40+ other AI agents.
The examples/ directory contains complete working examples:
For projects with multiple datasets, use an eval config. Dataset
paths are shorthand for tagged file sources, while clients, metrics, judges,
result stores, and other extensions are built-ins or come from plugins
(referenced as <namespace>/<kind>/<name>, such as acme/judge/completeness):
{
"name": "tool-selection-search",
"datasets": [
{
"type": "file",
"path": "evalsets/search.json"
}
],
"servers": {
"prod": {
"transport": "http",
"serverUrl": "https://example.com/mcp"
}
},
"client": "mst",
"clientOptions": {
"provider": "anthropic"
},
"metrics": ["passed", "tool_count"],
"results": {
"store": {
"type": "file",
"dir": ".mcp-test-results"
}
}
}
Run one eval config or a bounded batch:
npx mst run \
--config eval.json \
--plugins ./plugins \
--dry-run
npx mst batch \
--config-dir configs \
--workers 4 \
--skip-existing \
--dry-run
Use variants to compare server sets or client configurations. A variant can override
servers, client, model, client options, tool maps, input templates, metrics, and judges.
The canonical execution primitives remain EvalDataset, EvalCase,
MCPConfig, and runEvalDataset.
Plugins add dataset sources, clients, judges, pairwise judges, metrics, result stores, connectors, environments and shared configs. A plugin
is a plain default-exported object in ESLint's shape, and its extensions are
referenced as <namespace>/<kind>/<name>. See
Plugins.
Secrets remain environment-variable or plugin-owned runtime inputs and do not
belong in committed eval configs.
These MCP protocol features are not currently supported. These are deliberate scope decisions, not bugs:
listPrompts, getPrompt)callTool waits for the complete response)If any of these affect your use case, please open an issue.
MIT
Playwright-based testing and eval framework for MCP servers with LLM-as-a-judge
{ "mcpServers": { "gleanwork-mcp-server-tester-github": { "command": "npx", "args": [ "-y", "@gleanwork/[email protected]" ] } }}Araçlar
Henüz araç listesi yok. Uzak sunucular ilk sağlık kontrolünden sonra bir liste alır.
SSS
Bu sayfadan Cursor, Claude Desktop, Claude Code veya VS Code için stdio yapılandırmasını kopyalayın. PlugMCP proxy'sini değil, yayıncının kurulumunu kullanır.
Bu kayıtta gerekli bir ortam değişkeni listelenmiyor.
MCP Server Tester yerel olarak stdio üzerinden çalışır.
Son kontrol: 2026-10-28. Playwright-based testing and eval framework for MCP servers with LLM-as-a-judge