The verification layer for the agentic coding era.
AI ships code in minutes — verifying it hasn't. testsprite opens your live app, uses it like a real user, and shows your coding agent exactly what broke — so it fixes its own work before a bug ever reaches you.
Proof, in public: verification beats model size. With this CLI in the loop, the cheapest model in the field shipped the most correct app on an open leaderboard — 89%, at half the cost of the priciest one. See the leaderboard →
⭐ Help us reach more developers and grow the TestSprite community. Star this repo!
What is it?
TestSprite is the AI testing platform 100,000+ teams use to test their software, frontend and backend — in the cloud, against the live product, not mocks. This repo is its official CLI.
It puts that platform in your coding agent's hands: the agent verifies every behavior it ships, and what broke comes back as one self-consistent bundle it can act on — no dashboard scraping. Humans drive the same surface from a terminal or CI.
⭐️ Star the Repository
If you find testsprite useful, a GitHub Star ⭐️ would be greatly appreciated — it helps other builders (and their agents) find the project, and stars notify you about new releases.
Quickstart
Requires Node.js 20.19+, 22.13+, or 24+. (No global install? npx @testsprite/testsprite-cli works too.)
npm install -g @testsprite/testsprite-cli
testsprite setup
testsprite setup prompts for your API key, verifies it, and installs the verification-loop skill for the coding agents this project uses — one command, so your agent is wired to verify its own work.
Which agent it installs for. Each of the eight supported targets — claude, cursor, cline, windsurf, antigravity, kiro, copilot, codex — lands its skill in a different place, so setup works out which you use: the agent calling it, if it identifies itself, together with every agent whose config is already in the project. It installs for all of them and prints what it found — in a terminal it shows you that list first, so you can accept it or narrow it before anything is written. If nothing identifies an agent, it installs for claude and says so — that is the one case where you want --agent <target> to name one yourself. Check the result any time with testsprite agent status.
Non-interactive (CI / onboarding scripts):
TESTSPRITE_API_KEY=sk-... testsprite setup --from-env --yes
Pointing a coding agent (Claude Code, Cursor, Codex, Cline, …) at TestSprite? Have it run
testsprite setupfirst — that installs the verification skill, so the agent knows how to create, run, and triage tests on its own (instead of guessing from this README).setupreports which agents it installed for; add--agent <target>if that list is not the one you wanted. New here? Start with the getting-started overview.
Privacy note: interactive runs check the npm registry at most once per 24 h to offer a "new version available" notice — package name only, never your key or data;
TESTSPRITE_NO_UPDATE_NOTIFIER=1disables it. The backend also advertises its minimum supported CLI version — a below-floor CLI prints a one-line upgrade advisory on stderr, and a too-old client may be rejected with exit 14 (CLIENT_TOO_OLD). Details in DOCUMENTATION.md → Update notice.
From there, the loop runs on its own — an example session, typed by the coding agent:
# 1 — describe the behavior you want to guarantee, run it, wait
testsprite test create --project proj_8f0f6 --type frontend \
--plan-from ./checkout-flow.plan.json --run --wait --output json
# → exits 1: the run failed
# 2 — pull ONE self-consistent failure bundle
testsprite test failure get test_3a9f21c7 --out ./.testsprite/failure
# 3 — the agent reads the bundle, fixes the code, then replays
testsprite test rerun test_3a9f21c7 --wait --output json
# → exits 0: passed. The test now lives in your durable suite.
./checkout-flow.plan.json (the --plan-from argument above) is a JSON file describing the test in plain language — no browser code required. This is the exact, byte-identical output of testsprite test create --plan-template (also embedded verbatim in test create --help):
{
"$schema": "https://raw.githubusercontent.com/TestSprite/testsprite-cli/v0.4.0/schemas/plan.schema.json",
"projectId": "prj_abc123",
"type": "frontend",
"name": "Login rejects an empty password",
"planSteps": [
{
"type": "action",
"description": "Navigate to /login and submit the form with an empty password"
},
{
"type": "assertion",
"description": "Verify an inline error says the password is required"
}
]
}
Get this exact skeleton without hand-copying it (and without the risk of it drifting from your installed version — see below): testsprite test create --plan-template. Full field reference (including the {{...}}-placeholder caveat, size caps, and the $schema hook for live editor validation): Plan file format.
The
$schemaURL above is pinned to the CLI version that generated this doc (v0.4.0) —--plan-template's live output always pins to your installed version instead, which is what actually resolves. If you're reading this on a later release, run the command yourself rather than trusting this snippet verbatim.
Don't want to write the first tests yourself? For a deployed target, ask TestSprite to propose them. Local projects use the authored-plan flow below. testsprite test plan generate --project <id> explores your app (or reads the API spec you uploaded with project docs upload), drafts test cases, and stages them for review — nothing is written to your disk. You read the printed table and keep what you want:
testsprite test plan generate --project proj_8f0f6 # stages proposals for review
testsprite test plan accept --project proj_8f0f6 # → real tests
testsprite test plan accept --project proj_8f0f6 --only prop_2 prop_5 # or just these
It only runs what the project is still missing, so a second call picks up where the first left off. Generation spends workspace credits per stage that actually runs (the result line reports the spend), and a PRD uploaded with project docs upload --role prd bills 0.5 credits for its embedding — an API spec is free. Full walkthrough, including what happens on Ctrl-C and every precondition error: Generate commands.
Prefer to configure each step by hand (or learn the surface offline with --dry-run first)? See Manual setup and Install & verify.
Testing an environment on your machine
Frontend tests can reach your running app through a TestSprite tunnel, including on the Free plan (ordinary run credits apply).
Use an API key with run:tunnel; the CLI names the missing scope if the key cannot open a tunnel. Existing keys receive this scope on backend v0.18.0 or later unless deliberately narrowed.
testsprite project create --type frontend --name "My local app" --local 3000
testsprite test create --plan-from ./checkout.plan.json --project <project-id>
testsprite test run <test-id> # the saved loopback URL opens a tunnel and waits
testsprite test run <id> <id> --max-concurrency 5
testsprite test run --all --project <project-id>
# Or keep one tunnel open in terminal A (no port argument)
testsprite tunnel start --ttl 3600
# Terminal B: borrow the clientId printed above; keep terminal A running
testsprite test run <test-id> --tunnel-client <client-uuid>
testsprite tunnel list
testsprite tunnel stop <client-uuid>
# Stop every live tunnel on this account only when you intend to revoke them all
testsprite tunnel stop --all --confirm
Run several local frontend tests in one invocation to share one tunnel binding. The batch runs up to 5 tests at once by default; --max-concurrency accepts 1–10. --all --local skips backend tests. An owned batch cancels unfinished runs on Ctrl-C or tunnel loss; JSON output lists every requested test in input order, including tests that did not start. The minted client id is printed before the tunnel connects.
Hit the tunnel limit? Run testsprite tunnel list, then testsprite tunnel stop <client-uuid> for an unused binding, or testsprite tunnel stop --all --confirm to revoke them all. An older server without listing returns exit 4; a key without run:tunnel returns exit 3.
A second tunnel start or process using the same credential takes over, and the first exits 10.
An environment is one origin (scheme, host and port) plus its sign-in configuration. Frontend runs automatically open a tunnel for a saved localhost, 127.0.0.1 or [::1] URL: test run, test rerun, test create --run, test create-batch --run and testlist run use this behavior. --env <name> selects the environment; omit it for the project default. --no-wait disables automatic tunnels; drop it to run through one. If environment reads are unavailable, the command keeps the previous server behavior. Several fresh-run IDs require a tunnel; public targets retain the exit-5 refusal. A named public environment used against a local origin prints a sign-in advisory on stderr. Reruns and test-list runs share one timeout including capacity waits and require the server to confirm tunnel support; an older server triggers a cancellation attempt and a clear error.
--local <port> saves new project and environment URLs as http://localhost:<port>. Use --url http://127.0.0.1:<port> or --url http://[::1]:<port> when the host matters (--local-host is deprecated). Dev servers can be slow through a tunnel, so for heavy apps test a production build (npm run build plus your framework's preview or start command).
The tunnel uses verified TLS. On a network that re-signs TLS, set NODE_EXTRA_CA_CERTS=/path/to/ca.pem to your organisation's root CA. If the tunnel keeps failing to connect for 60 seconds, the command exits 10, and an owned run is cancelled and refunded.
On an owned tunnel, Ctrl-C or a timeout cancels unfinished runs by default, and a run cancelled before it finishes is refunded. A run on a borrowed --tunnel-client detaches on Ctrl-C instead. Local projects skip plan generation, so write plans with test create --plan-from. Portal runs stay blocked, at no charge, until the project has a public URL. See the local-testing reference for the details.
Commands
The earlier command names —
init,auth configure,auth whoami,auth logout— still work as hidden, deprecated aliases (each prints a one-line notice pointing at the new name), so existing scripts keep running.auth configurenow runs the fullsetup(it also installs the skill).
📚 Full reference — every command, flag, and example: DOCUMENTATION.md, including configuration & profiles, scripting, and exit codes.
Why a CLI for coding agents?
- 🧪 Tests like a real user. Runs against a live browser or API in the cloud — real clicks, real navigation, real assertions. Not a mock.
- 🤖 Agent-shaped output.
test failure getreturns one bundle — the failing step, its neighbors, screenshots, DOM snapshots, the test source, a root-cause hypothesis, and a recommended fix target — all sharing a singlesnapshotId. The CLI refuses to stitch data from two different runs, so an agent never reasons over a frankenstein context. - ♻️ A loop, not a one-shot.
create → run → failure get → fix → rerun— every pass is banked, not thrown away. - 📐 Scriptable & deterministic. Stable
--output jsoncontract, predictable exit codes, and a--dry-runthat exercises the full code path offline with canned data. - 🚦 CI-native.
testsprite ci init githubwrites a PR gate;testsprite ci run --env <name> --target-url "$PREVIEW_URL"tests each pull request's preview deployment as one execution and exits with the verdict. The job summary is the execution report — the same content as the GitHub App's PR comment — plus::error::annotations on the checks tab, step outputs,--summary-filefor a machine summary and--report junitfor a JUnit sidecar. In any other CI,testsprite ci run --project <id> --env <name> --target-url <url>still runs the suite as one execution and makes the exit code the verdict — the pull-request context, draft skip and job summary are GitHub Actions only. Details → - 🔌 One command to onboard your agent.
testsprite setupdrops ready-made skill files into your repo — for the agents it detects you using — so your coding agent knows how to drive the loop on its own.
How it works
Every time your agent changes code, it asks one question: is this behavior already covered by the suite?
- Not yet covered →
testsprite test create— describe the new behavior, run it. - Already covered →
testsprite test rerun— replay the existing tests, so nothing that used to work breaks silently. - Something fails →
testsprite test failure get— one self-consistent bundle; the agent fixes the code and reruns.
Every pass is banked into a durable suite, so coverage compounds as the project grows — a lasting record of every requirement it has ever gotten right, far bigger than any context window.
flowchart TD
A["🤖 Your coding agent<br/>Claude Code · Codex · Antigravity · Kimi · Cursor · Trae …"]
D{"behavior already<br/>covered by the suite?"}
B["<b>testsprite test create</b><br/>new behavior → new test"]
R["<b>testsprite test rerun</b><br/>replay the existing tests"]
C{{"☁️ TestSprite testing agent<br/>runs the test like a real user against<br/>real browsers & real APIs on Cloud"}}
F["<b>testsprite test failure get</b><br/>ONE self-consistent bundle:<br/>failing step · screenshots · DOM ·<br/>root-cause · recommended fix"]
S[("📚 Durable integration suite<br/>grows with every pass")]
A -->|"writes / changes code"| D
D -->|"no — new behavior"| B
D -->|"yes"| R
B --> C
R --> C
C -->|"pass ✅"| S
C -->|"fail ❌"| F
F -->|"agent reads the bundle<br/>& fixes the code"| A
S -.->|"defines what's covered"| D
The cloud is a black box on purpose: your agent describes intent and reads results. It never has to know how the test was driven — only what a real user experienced.
Proved in public
On CoderCup — an open leaderboard where frontier coding agents build the same app under the same rules, with TestSprite as the referee — the cheapest model in the field shipped the most correct app on the board: 89%, at half the cost of the priciest one.
That's the point of all of this: you no longer need the biggest, most expensive model to ship software you can trust — top-tier quality, without paying top-tier prices, within reach of every team.
Getting help
- 📚 CLI reference — DOCUMENTATION.md
- 🌐 Platform docs — testsprite.com/docs
- 🐛 Issues & feature requests — GitHub issues
- 💬 Quick questions — Discord, or
testsprite --help/testsprite test run --helpright in your terminal - 📝 Changelog — CHANGELOG.md
Contributing
Contributions are welcome — the CLI is plain TypeScript/Node (20.19+, 22.13+, or 24+), tested with Vitest, built with tsc. Getting a dev loop running takes a minute:
git clone https://github.com/TestSprite/testsprite-cli.git
cd testsprite-cli && npm install
npm run build # tsc → dist/ ; try it: node dist/index.js --help
npm test # unit suite — no network or credentials needed
npm run lint:fix # ESLint
npm run typecheck # tsc --noEmit
Pull requests target the main branch. The full guide — build from source, test tiers, CI checks, branches, project layout — is in CONTRIBUTING.md.
Support — if you need any help, we're responsive on Discord, and feel free to email us at [email protected] too.
License
Apache-2.0 © TestSprite
