MST (MCP Server Tester)
Testing, evals, and conformance for servers. MST is published as , and its CLI is .
에이전트 강화
게시자의 자체 명령입니다. PlugMCP 엔드포인트가 아닙니다.
Host a copy with our keys and logs.
npx -y @gleanwork/[email protected]Claude Code에는 로컬 명령용 설치 링크가 없습니다. 아래 명령을 복사해 터미널에서 실행합니다.
claude mcp add gleanwork-mcp-server-tester-github -- npx -y @gleanwork/mcp-server-tester@1.1.1터미널에서 실행합니다.
{ "mcpServers": { "gleanwork-mcp-server-tester-github": { "command": "npx", "args": [ "-y", "@gleanwork/[email protected]" ] } }}claude_desktop_config.json
원격 URL인가요, 내 컴퓨터에서 실행하는 명령인가요? 두 설치 방식의 차이
PlugMCP에서 실행
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
실행@gleanwork/mcp-server-testermstWrite deterministic Playwright tests against your MCP tools, or run data-driven eval datasets — including LLM-based evaluation of tool discoverability.
The mcp Playwright fixture connects to your MCP server (stdio or HTTP) and exposes a high-level API for calling tools and asserting responses. Custom matchers keep assertions readable.
import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';
test('read_file returns file contents', async ({ mcp }) => {
const result = await mcp.callTool('read_file', { path: '/tmp/test.txt' });
expect(result).toContainToolText('Hello, world');
expect(result).not.toBeToolError();
});
test('server exposes required tools', async ({ mcp }) => {
const tools = await mcp.listTools();
expect(tools.map((t) => t.name)).toContain('read_file');
});
Playwright tests are fast, deterministic, and designed for CI. Use them for regression testing, schema validation, and protocol conformance. The framework includes built-in conformance checks for the MCP spec. mst generate records tool calls as a spec file you can edit (see CLI).
Available matchers:
Eval datasets define cases as JSON: an input the client under test acts on, and assertions about what it did. Run them with runEvalDataset() in a Playwright test, on the mst client, or with mst run on any client (Claude Code, Cowork, ChatGPT). Checks on a single tool's response belong in Playwright tests, above.
{
"name": "file-ops",
"cases": [
{
"id": "read-config",
"input": "What version is set in the app's config file?",
"assertions": {
"toolsTriggered": {
"calls": [
{
"name": "read_file",
"required": true
}
]
},
"containsText": "1.0.0"
}
},
{
"id": "summarize-readme",
"input": "Summarize the README in two sentences",
"assertions": {
"toolsTriggered": {
"calls": [
{
"name": "read_file",
"required": true
}
]
}
},
"judges": [{ "type": "rubric", "rubric": "groundedness" }]
}
]
}
import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';
import { loadEvalDataset, runEvalDataset } from '@gleanwork/mcp-server-tester';
test('file operations eval', async ({ mcp }, testInfo) => {
const dataset = await loadEvalDataset('./data/evals.json');
const result = await runEvalDataset(
{ dataset, client: 'mst', model: 'claude-haiku-4-5', defaultTrials: 5 },
{ mcp, testInfo }
);
expect(result.passed).toBe(result.total);
});
Supported assertions:
Judges go in the case's judges list, beside assertions: each scores the answer against a rubric or reference, such as { "type": "rubric", "rubric": "groundedness" } or a plugin judge "acme/judge/completeness".
In LLM client mode, a real LLM receives your server's tool list and a natural language prompt, then decides which tools to call. This tests whether your tool names, descriptions, and input schemas are clear enough for autonomous use — a different question from whether the tools return correct output.
A case with input runs on the client. In a Playwright test, runEvalDataset names the client and model ({ dataset, client: 'mst', model: 'claude-haiku-4-5' }), and the mst client uses the test's MCP connection; an eval config names them with client and model. A case can set its own client, model and clientOptions.
{
"id": "find-config",
"input": "Find the application config file and return its contents",
"assertions": {
"toolsTriggered": {
"calls": [
{
"name": "read_file",
"required": true
}
]
}
}
}
LLM client mode makes real API calls and produces non-deterministic results. Use trials to run a case multiple times and measure pass rate rather than expecting 100% on a single run. See the LLM Client Guide for configuration and cost management.
Requires Node.js 22+. These docs describe 2.0, which is in beta:
npm install --save-dev @gleanwork/mcp-server-tester@beta @playwright/test
npm install @gleanwork/mcp-server-tester installs 1.x, whose docs are at v1.1.1. Upgrading: Migration guide (v1.x → v2.0).
LLM-as-judge with the default Anthropic judge needs the Anthropic SDK:
npm install --save-dev @anthropic-ai/sdk
The SDK client's packages (ai and the @ai-sdk/* providers) are optional dependencies, installed by default; add them yourself if you install with --omit=optional.
npx @gleanwork/mcp-server-tester@beta init
The CLI wizard creates a playwright.config.ts, example tests, and a sample eval dataset configured for your server. See the CLI Guide for all options.
Point the framework at your MCP server in playwright.config.ts:
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './tests',
reporter: [['list'], ['@gleanwork/mcp-server-tester/reporters/mcpReporter']],
projects: [
{
name: 'my-server',
use: {
mcpConfig: {
transport: 'stdio',
command: 'node',
args: ['server.js'],
},
},
},
],
});
For HTTP servers, set transport: 'http' and serverUrl. For servers that require OAuth, see the Transports Guide and CLI Guide for authentication setup, including CI/CD token management.
Plain stdio is supported by Claude CLI, Vercel AI SDK, ChatGPT Work/Codex, and Cowork evaluation clients, including Cowork on macOS and Linux. Cowork also supports private file-backed native proxies and mixed HTTP/stdio server sets. See evaluation client transport support for client-specific fields, environment rules, and setup ownership.
Install AI skills to help your coding assistant write tests and evals, and improve tool metadata:
npx skills add -g gleanwork/mcp-server-tester
This installs skills globally so they're available across all your projects. Four skills are included:
Compatible with Claude Code, Cursor, Windsurf, Copilot, and 40+ other AI agents.
The examples/ directory contains complete working examples:
For projects with multiple datasets, use an eval config. Dataset
paths are shorthand for tagged file sources, while clients, metrics, judges,
result stores, and other extensions are built-ins or come from plugins
(referenced as <namespace>/<kind>/<name>, such as acme/judge/completeness):
{
"name": "tool-selection-search",
"datasets": [
{
"type": "file",
"path": "evalsets/search.json"
}
],
"servers": {
"prod": {
"transport": "http",
"serverUrl": "https://example.com/mcp"
}
},
"client": "mst",
"clientOptions": {
"provider": "anthropic"
},
"metrics": ["passed", "tool_count"],
"results": {
"store": {
"type": "file",
"dir": ".mcp-test-results"
}
}
}
Run one eval config or a bounded batch:
npx mst run \
--config eval.json \
--plugins ./plugins \
--dry-run
npx mst batch \
--config-dir configs \
--workers 4 \
--skip-existing \
--dry-run
Use variants to compare server sets or client configurations. A variant can override
servers, client, model, client options, tool maps, input templates, metrics, and judges.
The canonical execution primitives remain EvalDataset, EvalCase,
MCPConfig, and runEvalDataset.
Plugins add dataset sources, clients, judges, pairwise judges, metrics, result stores, connectors, environments and shared configs. A plugin
is a plain default-exported object in ESLint's shape, and its extensions are
referenced as <namespace>/<kind>/<name>. See
Plugins.
Secrets remain environment-variable or plugin-owned runtime inputs and do not
belong in committed eval configs.
These MCP protocol features are not currently supported. These are deliberate scope decisions, not bugs:
listPrompts, getPrompt)callTool waits for the complete response)If any of these affect your use case, please open an issue.
MIT
Playwright-based testing and eval framework for MCP servers with LLM-as-a-judge
{ "mcpServers": { "gleanwork-mcp-server-tester-github": { "command": "npx", "args": [ "-y", "@gleanwork/[email protected]" ] } }}도구
아직 도구 목록이 없습니다. 원격 서버는 첫 상태 확인 후에 도구 목록이 생깁니다.
FAQ
이 페이지에서 Cursor, Claude Desktop, Claude Code 또는 VS Code용 stdio 설정을 복사합니다. PlugMCP 프록시가 아니라 게시자의 설치 방식을 사용합니다.
이 레코드에는 필수 환경 변수가 등록되어 있지 않습니다.
MCP Server Tester은(는) stdio를 통해 로컬에서 실행됩니다.
마지막 확인: 2026-10-28. Playwright-based testing and eval framework for MCP servers with LLM-as-a-judge