Plumloom Autoeval Oss

Open-source CLI and MCP server for evaluating AI applications and gating releases with reliability evidence.

1# Plumloom Autoeval Oss

README.mdMCP server

Transport: stdio
Status: Local

Power your agent

Add this server

Install from the publisher. Not a PlugMCP endpoint.

Run on PlugMCP

Host a copy with our keys and logs.

No packaged release in the registry yet. Follow the repository README.

RepositoryPublisher installStep-by-step: add this in Claude

Remote URL or a command on your machine? How the two installs differ

Run on PlugMCP

Host a copy with our keys and logs.

A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.

Launch

Server facts

Local

Transport: stdio
License: Apache-2.0
Last checked: 2026-10-28

Stars: 33
Tools: —
Uptime: —

Tools

No tool list yet. Remote servers get one after the first health check.

FAQ

Questions

How do I install Plumloom Autoeval Oss?

Copy the stdio config for Cursor, Claude Desktop, Claude Code, or VS Code on this page. It uses the publisher's install, not a PlugMCP proxy.

Does Plumloom Autoeval Oss need an API key?

No required environment variable is listed on this record.

What transport does Plumloom Autoeval Oss use?

Plumloom Autoeval Oss runs locally over stdio.

When was Plumloom Autoeval Oss last checked?

Last checked 2026-10-28. Open-source CLI and MCP server for evaluating AI applications and gating releases with reliability evidence.

From the publisher

Documentation

Demo

Quickstart, validate the setup, and see passing and failing release gates.

Autoeval is an open-source CLI and MCP server that turns committed evaluation files into versioned, gated checks on AI-agent behavior, scored by the Plumloom backend. It runs from your terminal, your CI, or your coding agent.

[!NOTE] A single evaluation tells you what happened. Repeated evaluation tells you what would happen again.

LLM outputs and judge scores vary between runs, so one passing score is one sample. Autoeval reports reliability evidence — confidence intervals and consistency statistics — so a release decision rests on more than a single number.

What you can evaluate

	Scenario	Conversation	Agent trace
Supported evaluation context	✅	✅	✅
Multiple trials — confidence intervals and consistency statistics	✅	—	—

[!IMPORTANT] Multiple trials and statistical reliability evidence apply to scenario evaluations. Autoeval also supports conversation and agent-trace evaluation contexts.

Where it runs


`>_ CLI`	Author, validate, and run eval files; read scored results from the terminal.
`⑂ CI release gate`	A suite manifest lists evals and thresholds. `autoeval suite gate` exits non-zero on anything other than `PASS`.
`◇ Coding agents`	A local stdio MCP server exposes the same action layer as the CLI.
`⚿ AI providers`	Connect OpenAI or Together.ai so Plumloom can run evaluations.
Eval files and the suite manifest are the only inputs you author and commit. Autoeval submits runs to Plumloom, reads back scored results, and the gate turns them into a CI pass/fail.

Quickstart

Requirements: Node.js 22.13+ and a Plumloom CLI key (pl_sk_…). pnpm 11 only if building from source.

Connect an AI provider (OpenAI or Together.ai).
Create a Plumloom API key.

npm install --global @plumloom/cli
autoeval login
autoeval quickstart

quickstart resolves a workspace and judge model, grades a bundled sample end to end, and prints the result. Autoeval connects to https://api.plumloom.ai automatically. → Quickstart guide

pnpm install --frozen-lockfile && pnpm build

In the examples below, replace autoeval with node packages/cli/dist/cli.js when running from this repository.

File-driven evaluation

An evaluation file uses the configured-run JSON accepted by autoeval eval validate. Artifact and reference-document paths resolve relative to the evaluation file.

{
  "methodology": { "judgeModelId": "<judge-model-id>", "evaluatorInstructions": "..." },
  "configuration": {
    "contextName": "Support conversation",
    "contextType": "conversation",
    "artifactFile": "./data/conversation.json",
    "referenceDocumentFiles": ["./policy/refunds.md"]
  }
}

autoeval eval validate --input support.autoeval.json
autoeval eval create-from --workspace <workspace-id> --input support.autoeval.json --run

Model-ID fields take UUIDs; copy enabled IDs from autoeval models. → Write and run your own evaluation file

Gate a release in CI

A suite manifest names a workspace, lists evaluation files, and declares release thresholds.

workspace: <workspace-id>
gate:
  minOverall: 4.0
  minScenario: 3.5
evals:
  - ./evals/refund-policy.autoeval.json
  - file: ./evals/support-conversation.autoeval.json
    gate:
      metrics:
        factuality: 4.0
        relevance: 3.8

autoeval doctor     --manifest autoeval.suite.yaml   # read-only pre-flight
autoeval suite run  --manifest autoeval.suite.yaml   # run, no release judgment
autoeval suite gate --manifest autoeval.suite.yaml   # apply thresholds

Verdict	`PASS`	`FAIL` / `INCONCLUSIVE`	`ERROR`
Exit code	`0`	`1`	`5`

Commit the manifest, every referenced eval file, and any artifacts or reference documents those evals load. Start from examples/suite/autoeval.suite.yaml and its fixtures under examples/smoke/fixtures/, then use autoeval workspace list and autoeval models to replace the synthetic IDs.

Copy .github/workflows/autoeval-suite-gate.yml.example into your repository for a ready-to-use workflow. See Gate a release in CI and Release gating for CI.

For same-repository pull requests, the example workflow also updates one context-aware Autoeval Reliability Report comment per suite and matrix job, showing the gate verdict alongside available Standard, Judgment, and Repeatability evidence. Comment failures never replace the gate verdict or exit code. Fork pull requests are excluded because they do not receive repository secrets.

autoeval workspace list
autoeval models
autoeval results <evaluation-id> <run-id>
autoeval --json results <evaluation-id> <run-id>

See the command guide, CLI reference, developer guide, and .env.example. Smoke workflows: examples/smoke/ · sample configs: examples/evals/.

Import OpenAI evaluations

Add your OpenAI key once in the Plumloom UI under AI Providers & Models. The CLI uses only your existing Plumloom CLI key; it never requests an OpenAI key or calls OpenAI directly.

autoeval import status
autoeval import connect
autoeval import review --search rubric
autoeval import run --all-new
autoeval import list
autoeval import insights --workspace <workspace-uuid>
autoeval import show <plumloom-eval-uuid> --show-outputs

Imports create a Plumloom copy and never write back to OpenAI. See the OpenAI import guide for every command, exit code, safety limit, and local/staging verification commands. This import surface is CLI-only.

Detail results use --page and --size (sent as backend page_size). Imported-run histories show the backend's run dates and result totals. Removal reports success only after the backend confirms deleted: true.

Human detail output uses numbered runs and shows unknown for missing model names. Add --json for the full redacted record, including raw_json. Result rows come from the backend's nested results.items page. Omit --run for the newest run, or use the Plumloom run UUID from the run history, not an eval_... source evaluation ID.

Insights use spaced sections and readiness markers. The human flaky-graders panel shows only backend-flagged entries; JSON retains the full insights envelope and grader evaluation IDs.

Interactive import review, import run, import show, and import evals display loading progress on stderr; --json stays machine-readable. Import runs show batch progress for IDs, text files, and all-new selections. Expanded detail output uses terminal-width input/expected/output columns with wrapped content. Ctrl+C cancels review without displaying a late response.

Use it from your coding agent

MCP server — a local stdio server exposing the same action layer as the CLI. Credentials resolve from AUTOEVAL_API_KEY, then the OS credential store; it never prompts. → MCP guide · Codex & Claude Code

Claude Code plugin

claude plugin marketplace add AlchemyInCode/plumloom-autoeval-oss
claude plugin install plumloom-autoeval@plumloom

Gemini CLI extension (skill-only for now: install and trigger supported, end-to-end execution not yet)

gemini extensions install https://github.com/AlchemyInCode/plumloom-autoeval-oss

→ Agent skill guide · Security and permissions

Free Autoeval Pilot

[!TIP] Limited to ten teams. Complimentary hands-on onboarding, evaluation support, and model access. Applications are considered in the order received. Sign up for the Free Pilot →

Documentation

Get started	Integrate	Operate
Docs home	MCP server	Release gating
Quickstart	Codex & Claude Code	Troubleshooting
Command guide	Agent skill	Security policy
Developer guide	Agent skill security	Contributing
CLI reference	DeepSeek Harness

Support

Privacy policy: Plumloom Privacy Policy
Support: [email protected]

Plumloom Autoeval Oss

AlchemyInCode33starsClaim this server

Open-source CLI and MCP server for evaluating AI applications and gating releases with reliability evidence.

Install Run on PlugMCPClaim this server

GitHub

Run on PlugMCP

1# Plumloom Autoeval Oss

README.mdMCP server

Transport: stdio
Status: Local

Power your agent

Add this server

Install from the publisher. Not a PlugMCP endpoint.

Run on PlugMCP

Host a copy with our keys and logs.

No packaged release in the registry yet. Follow the repository README.

RepositoryPublisher installStep-by-step: add this in Claude

Remote URL or a command on your machine? How the two installs differ

Run on PlugMCP

Host a copy with our keys and logs.

A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.

Launch

Server facts

Local

Transport: stdio
License: Apache-2.0
Last checked: 2026-10-28

Stars: 33
Tools: —
Uptime: —

Tools

No tool list yet. Remote servers get one after the first health check.

FAQ

Questions

How do I install Plumloom Autoeval Oss?

Copy the stdio config for Cursor, Claude Desktop, Claude Code, or VS Code on this page. It uses the publisher's install, not a PlugMCP proxy.

Does Plumloom Autoeval Oss need an API key?

No required environment variable is listed on this record.

What transport does Plumloom Autoeval Oss use?

Plumloom Autoeval Oss runs locally over stdio.

When was Plumloom Autoeval Oss last checked?

Last checked 2026-10-28. Open-source CLI and MCP server for evaluating AI applications and gating releases with reliability evidence.

From the publisher

Documentation

Demo

Quickstart, validate the setup, and see passing and failing release gates.

[!NOTE] A single evaluation tells you what happened. Repeated evaluation tells you what would happen again.

LLM outputs and judge scores vary between runs, so one passing score is one sample. Autoeval reports reliability evidence — confidence intervals and consistency statistics — so a release decision rests on more than a single number.

What you can evaluate

	Scenario	Conversation	Agent trace
Supported evaluation context	✅	✅	✅
Multiple trials — confidence intervals and consistency statistics	✅	—	—

[!IMPORTANT] Multiple trials and statistical reliability evidence apply to scenario evaluations. Autoeval also supports conversation and agent-trace evaluation contexts.

Where it runs


`>_ CLI`	Author, validate, and run eval files; read scored results from the terminal.
`⑂ CI release gate`	A suite manifest lists evals and thresholds. `autoeval suite gate` exits non-zero on anything other than `PASS`.
`◇ Coding agents`	A local stdio MCP server exposes the same action layer as the CLI.
`⚿ AI providers`	Connect OpenAI or Together.ai so Plumloom can run evaluations.
Eval files and the suite manifest are the only inputs you author and commit. Autoeval submits runs to Plumloom, reads back scored results, and the gate turns them into a CI pass/fail.

Quickstart

Requirements: Node.js 22.13+ and a Plumloom CLI key (pl_sk_…). pnpm 11 only if building from source.

Connect an AI provider (OpenAI or Together.ai).
Create a Plumloom API key.

npm install --global @plumloom/cli
autoeval login
autoeval quickstart

quickstart resolves a workspace and judge model, grades a bundled sample end to end, and prints the result. Autoeval connects to https://api.plumloom.ai automatically. → Quickstart guide

pnpm install --frozen-lockfile && pnpm build

In the examples below, replace autoeval with node packages/cli/dist/cli.js when running from this repository.

File-driven evaluation

An evaluation file uses the configured-run JSON accepted by autoeval eval validate. Artifact and reference-document paths resolve relative to the evaluation file.

{
  "methodology": { "judgeModelId": "<judge-model-id>", "evaluatorInstructions": "..." },
  "configuration": {
    "contextName": "Support conversation",
    "contextType": "conversation",
    "artifactFile": "./data/conversation.json",
    "referenceDocumentFiles": ["./policy/refunds.md"]
  }
}

autoeval eval validate --input support.autoeval.json
autoeval eval create-from --workspace <workspace-id> --input support.autoeval.json --run

Model-ID fields take UUIDs; copy enabled IDs from autoeval models. → Write and run your own evaluation file

Gate a release in CI

A suite manifest names a workspace, lists evaluation files, and declares release thresholds.

workspace: <workspace-id>
gate:
  minOverall: 4.0
  minScenario: 3.5
evals:
  - ./evals/refund-policy.autoeval.json
  - file: ./evals/support-conversation.autoeval.json
    gate:
      metrics:
        factuality: 4.0
        relevance: 3.8

autoeval doctor     --manifest autoeval.suite.yaml   # read-only pre-flight
autoeval suite run  --manifest autoeval.suite.yaml   # run, no release judgment
autoeval suite gate --manifest autoeval.suite.yaml   # apply thresholds

Verdict	`PASS`	`FAIL` / `INCONCLUSIVE`	`ERROR`
Exit code	`0`	`1`	`5`

Copy .github/workflows/autoeval-suite-gate.yml.example into your repository for a ready-to-use workflow. See Gate a release in CI and Release gating for CI.

autoeval workspace list
autoeval models
autoeval results <evaluation-id> <run-id>
autoeval --json results <evaluation-id> <run-id>

See the command guide, CLI reference, developer guide, and .env.example. Smoke workflows: examples/smoke/ · sample configs: examples/evals/.

Import OpenAI evaluations

Add your OpenAI key once in the Plumloom UI under AI Providers & Models. The CLI uses only your existing Plumloom CLI key; it never requests an OpenAI key or calls OpenAI directly.

autoeval import status
autoeval import connect
autoeval import review --search rubric
autoeval import run --all-new
autoeval import list
autoeval import insights --workspace <workspace-uuid>
autoeval import show <plumloom-eval-uuid> --show-outputs

Insights use spaced sections and readiness markers. The human flaky-graders panel shows only backend-flagged entries; JSON retains the full insights envelope and grader evaluation IDs.

Use it from your coding agent

Claude Code plugin

claude plugin marketplace add AlchemyInCode/plumloom-autoeval-oss
claude plugin install plumloom-autoeval@plumloom

Gemini CLI extension (skill-only for now: install and trigger supported, end-to-end execution not yet)

gemini extensions install https://github.com/AlchemyInCode/plumloom-autoeval-oss

→ Agent skill guide · Security and permissions

Free Autoeval Pilot

[!TIP] Limited to ten teams. Complimentary hands-on onboarding, evaluation support, and model access. Applications are considered in the order received. Sign up for the Free Pilot →

Documentation

Get started	Integrate	Operate
Docs home	MCP server	Release gating
Quickstart	Codex & Claude Code	Troubleshooting
Command guide	Agent skill	Security policy
Developer guide	Agent skill security	Contributing
CLI reference	DeepSeek Harness

Support

Privacy policy: Plumloom Privacy Policy
Support: [email protected]