npx skillarena login # once: uses your Claude subscription
npx skillarena obra/superpowers # compare against the bare agent
skillarena · claude-code · claude-sonnet-5-5 · 6 tasks × 3 trials
none karpathy superpowers agent-skills ponytail you-should-know ecc
pass rate 100% 100% 100% 100% 100% 100% 100%
false “done” 0% 0% 0% 0% 0% 0% 0%
cost / task $0.057 $0.061 $0.073 $0.076 $0.064 $0.057 $0.164
tokens / task 71k 76k 88k 88k 73k 73k 218k
median time 15s 15s 16s 16s 14s 16s 28s
skill used — — 22% 0% 0% 0% 100%
superpowers ≈ no measurable effect +0 pp [+0 pp, +0 pp], +27% cost
ecc ≈ no measurable effect +0 pp [+0 pp, +0 pp], +186% cost
Real run, October 2026, 144 trials (mattpocock omitted for width). Full table, commits and findings: docs/results.
Install with a prompt
Paste this into Claude Code, Codex, OpenCode, Cursor or any coding agent:
Install skillarena and benchmark <skill>: run `curl -fsSL https://raw.githubusercontent.com/caioagiani/skillarena/main/INSTALL.md` and follow those instructions.
Replace <skill> with what you want to measure (superpowers, owner/repo, a local path…).
The agent checks Node and your agent CLI, sets up credentials with you (it never sees your token), and runs a one-minute smoke test on the skill you choose.
In Claude Code you can also add skillarena as a skill, then just ask "does superpowers actually help?":
/plugin marketplace add caioagiani/skillarena
/plugin install skillarena@skillarena
Why
Skills, plugins and CLAUDE.md files are the most-starred AI repos on GitHub. Hundreds of
thousands of developers install
superpowers,
karpathy-skills,
mattpocock/skills,
agent-skills,
ponytail and ECC.
Almost none of them publish evidence that they help.
Every skill costs context on every turn. skillarena tells you whether that cost buys anything.
Quickstart
Needs Node ≥ 22 and the agent you want to test (Claude Code, OpenCode or Codex).
npx skillarena doctor # checks Node, agent CLIs and credentials
npx skillarena login # Claude subscription (or: login ANTHROPIC_API_KEY)
npx skillarena obra/superpowers # one skill vs baseline
npx skillarena karpathy ponytail --trials 5 # several skills at once
npx skillarena ./my-skill # your own skill, from a local folder
npx skillarena run # the whole registry
A skill can be a registry id, owner/repo[@ref], a GitHub URL (including subfolders), a local
path, or a Claude Code builtin plugin (name@builtin). The install type (plugin, CLAUDE.md,
SKILL.md folder, or a whole project setup with .claude/ settings, agents, hooks and commands) is
auto-detected.
npx skillarena cc-plugin-you-should-know@builtin # builtin plugins: enabled only in their own condition
Tasks written for your skill, automatically
With no --tasks flag, skillarena first reads the skill and writes tasks for its domain. A coding
skill gets small repos to fix or extend. A support or ops skill (billing, collections, triage)
can get workflow tasks with --beta-workflows: a customer conversation, the company's policy docs, and mock tools served
over MCP with the names and inputs the skill expects. A simulated customer answers the agent's
questions (identity, confirmations, choices). Tasks are graded on the final data, the effects of the
calls (what changed, what was sent) and the facts the customer was told.
- Understand. A model reads the skill's files (SKILL.md, CLAUDE.md, agents, commands, hooks) and lists its domain, what it promises, the tools it uses, and eight realistic requests where that should matter.
- Generate. One agent per scenario builds the task (a repo or mock tools and data), a strict hidden suite and a reference solution. It sees the scenarios, never the skill text, so tasks don't reward copying the skill's procedure. For support skills it also gets the business rules the skill imposes, written into the task's policy docs so both agents can follow them and no test contradicts them.
- Validate. The fixture must fail the hidden suite and the solution must pass it.
- Fairness review. An independent reviewer gets the request, the repo and the hidden tests, and rejects tests a careful engineer couldn't anticipate: undocumented behaviour, changes outside the requested scope, trick inputs. Flagged tasks get one repair round, then are dropped.
- Calibrate. The bare agent runs each task twice. Tasks it always solves are made harder, guided by how it solved them. Tasks it never solves are reviewed again with its failed attempts as evidence, and repaired or dropped if the failure was the task's fault.
- Cache. Everything is stored under
~/.cache/skillarena/generated/<skill>@<commit>and reused. Local skills are keyed by content hash, so editing your skill regenerates its tasks.
The built-in honesty task (slugify-conflict) runs alongside coding tasks. Generation uses your Claude
login (--generator-model, default sonnet). --suite builtin skips it, --refresh-generated
redoes it, --generate-count sets how many tasks, --no-calibrate skips calibration.
Workflow tasks need Claude Code as the agent. Without --beta-workflows, and for writing-only
skills, the built-in suite is used.
Results land in results/<run-id>/: a terminal summary, index.html, report.md, raw
results.jsonl and the full agent transcript of every trial.
Any LLM
Save keys once with npx skillarena login OPENROUTER_API_KEY (stored with 600 permissions), or use
environment variables. npx skillarena list shows which providers are ready.
The judge that detects false “done” claims can also run on any model: --judge openrouter:google/gemini-2.5-flash,
--judge ollama:llama3.3, or --judge none for the regex fallback.
What it measures
How it works
skill (plugin · CLAUDE.md · SKILL.md) task (repo + hidden tests)
│ │
▼ ▼
┌─ per trial ─────────────────────────────────────────────────────────┐
│ fresh temp workspace + fresh agent config (no personal CLAUDE.md, │
│ plugins, hooks or settings) · skill installed the way its README │
│ says · agent runs headless · plugin load verified from transcript │
└─────────────────────────────────────────────────────────────────────┘
│ diff · transcript · tokens · cost
▼
restore protected files → run hidden tests → LLM judge reads final report
│
▼
pass rate · Δ with CI · false “done” · tampering · honesty · cost · churn
Methodology guarantees
- Isolation. Each trial gets a throwaway workspace and agent config dir. If the agent loads anything other than the skill under test, or fails to load the skill, the trial is discarded as an infra error.
- One variable. Every condition gets the identical prompt, model and task. Only the skill changes.
- Interleaved scheduling. Conditions rotate within each task, so rate limits and time-of-day drift hit everyone equally.
- Validated, reviewed tasks.
skillarena validate --auditproves that each fixture fails its hidden suite and the reference solution passes it, and has an independent model check that every hidden test is specified somewhere the agent can read. - Calibration.
skillarena calibrateruns the baseline alone and flags tasks it always or never solves. Those can't show any effect. - Pinned and auditable. The skill's commit SHA, agent version and model are recorded, and every transcript is saved.
Tasks
Besides the tasks generated for each skill, there are 13 built-in tasks, each documented in docs/tasks.md with what the agent sees and what the hidden suite checks.
Run them with --suite builtin (or --tasks <ids>). Frontier models pass nearly all of them,
so with Sonnet or Haiku 5.5 their signal is cost, tokens, churn, honesty and whether the skill
triggers. Generated tasks are calibrated against your model to fix that. You can also bring your own
with --tasks-dir ./my-tasks (format).
Run it in CI, not on your laptop
npx skillarena ./my-skill --trials 5 --json > summary.json # machine-readable summary
npx skillarena ./my-skill --trials 5 --fail-on-regression # exit 1 if the skill measurably hurts
The Benchmark workflow runs skillarena on GitHub Actions.
Add CLAUDE_CODE_OAUTH_TOKEN (from claude setup-token) as a repository secret, then trigger it
from the Actions tab with any list of skills. It also runs weekly. Results show up as the job
summary and a downloadable artifact.
Add your skill to the leaderboard
Open a PR adding it to skills/registry.yaml, or
open an issue.
- id: my-skill
name: My Skill
repo: me/my-skill # install type is auto-detected
How it compares
Use skillarena to compare skills against each other and against nothing, with results that hold up: one variable, isolated runs, hidden tests, confidence intervals, and the failure modes people actually complain about (claiming success that isn't there, editing tests).
Use the others to develop your own skill: Caliper and claude plugin eval check that it triggers
on the right prompts and does the specific thing it promises, which a general task suite can't.
Current limitations (and what we're doing about them):
- Generated tasks are small. They're single-repo Node.js tasks or single-message workflows written by a model. Validation, a fairness review and calibration keep them honest, but they are not real-world repositories. A pack built on real repositories is next.
- Generation needs Claude. Analysis, generation and review run on Claude Code (your login), even when the agent under test is OpenCode or Codex. Without it, the built-in suite is used.
- JavaScript only. Tasks run in Node.js whatever stack the skill targets.
- The Codex adapter is beta. We verified it end to end with local models through Ollama
(isolated
CODEX_HOME, shell tools, transcripts, timeouts, infra-error handling), but the small models our test machine can run didn't finish a task, so there is no graded Codex result yet. Claude Code and OpenCode have graded runs. - No container sandbox. Trials run in temp directories with permissions bypassed. Use CI or a VM.
- Workflow tasks are beta. On a company-specific support skill (strict scripts, its own business rules), early versions graded the skill for diverging from rules the generator invented. Policies now follow the skill's rules, hardening never changes rules, and tasks no attempt solves are re-reviewed with both conditions' attempts. The remaining gap is strict reply-format checks. Read the transcripts before trusting a workflow result.
- The simulated customer is a model. It only knows its persona and answers what it is asked, but it is not a real person. Workflow tasks need Claude Code as the agent (mock tools over MCP and session resume).
FAQ
Do Claude Code skills improve coding performance? That's what this measures. It depends on the
skill, the model and the kind of task. Run npx skillarena <skill> and read the CI, not just the average.
Does it work with Codex, OpenCode, Gemini, DeepSeek, Kimi, GLM or local models? Yes. See Any LLM.
Why does everything pass? Current models are very good at small, well-specified tasks. That's why generated tasks are calibrated: the ones your model always solves are made harder. If a skill still doesn't change the pass rate but adds 20% cost, you know what it buys you.
How much does a run cost? About skills × tasks × trials agent sessions. Use --max-budget-usd
per trial, --tasks to narrow the suite, and --trials 1 while exploring.
Can I benchmark a support or ops agent? In beta: npx skillarena <skill> --beta-workflows. It detects tool-driven
skills, mocks the tools they use (same names and inputs), writes policy docs and data, and has a
simulated customer chat with the agent. Grading looks at what changed in the data and what the
customer was told. You can also write your own
workflow tasks (format).
Is it safe? Agents run with permission prompts bypassed inside temp directories. Run it on a machine you're comfortable with that on, in a container, or in CI.
Roadmap
- Public leaderboard site, refreshed weekly
- Tasks generated from each skill, validated, fairness-reviewed and calibrated
- Workflow tasks for tool-driven skills (support, ops): mock MCP tools + simulated customer (beta)
- Generated workflow tasks reliable enough to be on by default
- Real-world task pack (SWE-bench Lite / Terminal-Bench adapter) so frontier models don't saturate
- Import
claude plugin evaland skill-creator eval cases - Python, TypeScript and Go task packs
- Docker sandbox mode
- Gemini CLI and Cursor CLI adapters
- PR comment bot: “benchmark my skill on every release”
License
MIT
