Canary Lab

Canary Lab

A local wrapper that coordinates Playwright with Claude or Codex.

Your Claude or Codex agent investigates the app, writes Playwright tests, and repairs application failures. Playwright executes the tests. Canary Lab starts services, manages run state, captures evidence, and records the verdict. The evaluation report connects requirements to tests, captured evidence, and the result.

Quick Start

You need Node.js 22.12 or newer and npm 9 or newer.

1. Create a Canary Lab workspace. This is separate from the app repo you want to test:

   npx canary-lab init my-lab
   

   init installs the workspace dependencies and Chromium, then registers the Canary Lab skills and Model Context Protocol (MCP) connection for supported Claude and Codex clients. The workspace holds test suites and run evidence; your app stays in its own repo.

2. Restart Claude or Codex so it discovers the skill and connection. In its chat, ask it to test a specific flow using your app's absolute path:

   /canary-lab /absolute/path/to/your-app "checkout flow"
   

3. Follow the Flight (the end-to-end evaluation) in your agent or open the Canary Lab interface. When the report is ready, open its evaluation archive to review the test results and captured evidence.

The connection starts the local Canary Lab service when needed. To open the interface yourself:

cd my-lab
npx canary-lab ui

If /canary-lab or the Canary Lab exec tool is missing, run npx canary-lab setup --force from the workspace and restart your agent. Setup reports each saved connection separately; see configuration repair for skill migration, backups, and verification outcomes.

What Happens During a Flight

/canary-lab starts one resumable Flight:

scan repo → create suite → collect requirements → author and map tests → run and repair → export evaluation → check readiness for parallel runs

The agent works in your current Claude or Codex session. Canary Lab manages Flight state and validates each stage against saved evidence.

• Requirements stay reviewable. Add a product requirements document, link a local file, gather relevant repo documents, or infer requirements from the branch diff.
• Coverage is traceable. The coverage percentage comes from test mappings to requirements and paths.
• Repairs are usually isolated. Runs use separate Git checkouts when available. Before a suite is prepared for parallel runs, worktree creation failure can make a run use the app's working copy; Canary Lab warns you.
• Progress survives interruption. Run /canary-lab again with the same repo to continue the existing Flight.
• The evaluation is the deliverable. The archive preserves the recorded verdict, including failed or aborted runs, test evidence, any captured browser media, and per-test reasoning. It is available before the separate Parallel setup stage finishes.

How Canary Lab Determines the Verdict

Canary Lab derives pass or fail from the Playwright results and their saved evidence. The agent uses that evidence to diagnose failures, repair application code, and request a rerun. Repairs preserve the tests, and the verdict reflects the recorded run.

Canary Lab adds:

• Reviewable run evidence. Test results and captured artifacts support the recorded verdict.
• Controlled concurrency. After Canary Lab verifies that the app accepts assigned ports, runs can use separate ports and Git worktrees. Conflicting work waits in a queue.
• Safe environment switching. Environment files are backed up before changes and restored when the run ends.

Choose the Right Skill

Use /canary-lab for the complete evaluation. Use a focused skill when you need only one part:

The skills share the same workspace, runs, evidence, and UI. Work started in one surface appears in the others.

Use the UI or CLI Directly

The agent skill is the normal interactive path. The same Flight can also start from the Flights view or from a terminal:

npx canary-lab flight /absolute/path/to/your-app "checkout flow"

Use the CLI for shell automation or when you want Canary Lab to conduct the Flight outside an existing agent conversation. Calling it again follows an active Flight or resumes a paused one. A completed Flight needs --redo or --from-stage to run again.

The UI and MCP server share one configurable port, 7421 by default. Choose another during setup with npx canary-lab init my-lab --port 8200, or change it later in Project Settings. The ui --port option is not supported.

Custom MCP clients should connect to http://localhost:<port>/mcp?profile=compact.

Playwright Without a New Test Language

Canary Lab suites use normal Playwright tests plus a service configuration that names your existing development commands. Canary Lab starts each service, waits for health checks, and tags the output so failures map back to the correct test. Once the app is prepared for parallel runs, Canary Lab can assign free ports to concurrent runs.

See Suite Folders for the file structure and examples, or Guide for the complete run and repair workflow.

New workspaces include demonstrations for a prepared repair loop, a bare repo Flight, and focused coverage and authoring workflows. Delete the demo files after exploring them.

Works With Docker Compose

Use Docker Compose for infrastructure such as Postgres or Redis, and let Canary Lab start application services with their normal development commands. This keeps hot reload available during repair.

Measured Repair Speed

In the latest-result view of the bundled storefront demo, median repair time was 71 seconds with Canary Lab and 132 seconds without it. Median output-token use was 63% lower.

We tested Claude Code and Codex on two scenarios with seeded bugs. The table uses five pairs per agent/scenario, with one requested fresh Claude three-service pair replacing the original slower pair. This selected replacement is disclosed in the benchmark report; the original evidence remains preserved.

All 42 executed attempts passed the independent evaluator, including the original and fresh pairs. The 40 selected attempts above show lower elapsed time and token use; dollar cost was not measured. The original campaign's activity breakdown attributes most of its time difference to fewer agent steps for service startup and test execution.

This is a small sample on Canary Lab's own demo, with one pair replaced after observing its result. The refreshed numbers are descriptive, and no new confidence interval is claimed for that group. See Benchmark for the method, limits, and every selected pair.

How It Compares

Use Canary Lab when you need service orchestration, environment switching, backend logs, requirement coverage, or a saved evaluation report. Plain npx playwright test is enough when you only need to run a suite. Harness data stays local; repair agents may still need network access.

Updating a Workspace

With Node.js 22.12 or newer:

npm install --save-dev canary-lab@latest
npx canary-lab upgrade

Restart Canary Lab and connected agent apps afterwards so they load the refreshed skills and connection path. Existing suite folders under features/ and personal agent files are preserved.

Requirements

• Node.js 22.12 or newer and npm 9 or newer.
• Chromium for Playwright, installed automatically by canary-lab init unless installation is skipped.
• A local UI server, started automatically through the agent connection or manually with npx canary-lab ui.
• Optional repair agents: supported Claude or Codex command-line clients on PATH.

node-pty gives each service a real terminal. Its prebuilt binaries require no normal installation-time compilation. A postinstall step restores the execute bit on its helper binary when needed; it does nothing on Windows or when node-pty is absent.

Limitations

• Repairs are only as useful as your service logs.
• Environment runs overwrite target files while active. If a process is killed during backup or restore, reopen the UI and use the environment controls to recover.
• Environment values are not validated, so stale configuration can surface as unclear failures.
• Linux and Windows workflows are not polished yet.

Documentation

License

MIT