- name
SKILL.md
技能檔案保留在發布者的儲存庫中。
在終端機中貼上複製的指令。它會將這個 SKILL.md 安裝到對應的代理中。
npx skills add agentscope-ai/OpenJudge --skill 00-meta-eval -a cursor -y在終端機中貼上複製的指令。它會將這個 SKILL.md 安裝到對應的代理中。
npx skills add agentscope-ai/OpenJudge --skill 00-meta-eval -a claude-code -y在終端機中貼上複製的指令。它會將這個 SKILL.md 安裝到對應的代理中。
npx skills add agentscope-ai/OpenJudge --skill 00-meta-eval -a codex -y在終端機中貼上複製的指令。它會將這個 SKILL.md 安裝到對應的代理中。
npx skills add agentscope-ai/OpenJudge --skill 00-meta-eval -a grok -y在 Grok Bot 聊天中傳送複製的訊息。Bot 會從 GitHub 取得這個 SKILL.md,並加入你所有 Bot 共用的技能中。
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/eval_pipeline/00-meta-eval來自發布者
Entry router for the eval skill collection. You diagnose what the user has and route them to the right sub-skill. You don't do evaluation yourself — you're the triage desk.
Each sub-skill is self-contained: it carries inline the data shapes, statistics, and data principles it needs, so it can be installed and used on its own.
You MUST create a task for each item and complete them in order:
Ask these 4 questions (all at once — don't drip-feed):
To route you to the right evaluation skill, I need to understand your situation:
1. What data do you have?
a) Agent traces / production logs
b) Product spec / design docs
c) Nothing yet — starting from scratch
2. Do you have human labels?
a) Yes, ≥50 labeled examples
b) Some, but fewer than 50
c) None
3. What are the stakes?
a) Low — internal experimentation, exploring options
b) Production — customer-facing, quality matters
c) Regulated — compliance requirements, audit trail needed
4. How well do you know this evaluation domain?
a) Very well — have clear standards and criteria
b) Somewhat — general idea but need structure
c) Not well — exploring what "good" even means
Shortcut rule: data_form + label_status already determine the entry workflow
(see triage table). The moment those two are clear — even if stakes and domain knowledge
are not — give the provisional recommendation AND ask the remaining questions in the same
message. stakes and user_prior refine the downstream path (how much calibration rigor,
how fast a path), not the entry point. Never make the user wait a round-trip for a route you
can already determine.
Example: "no logs, no labels" → recommend 08-bootstrap now, and ask stakes/domain to tune
the roadmap. Don't reply with only the questionnaire.
Match the user's situation to a sub-skill:
These are local workflows under skills/eval_pipeline/, not packages to install — "use"
a workflow means open and follow that sub-skill.
After diagnosis, respond with:
Diagnosis: data=[data_form] | labels=[label_status] | stakes=[value or "asking"] | domain=[value or "asking"]
Recommended workflow: `[skill-name]` (provisional if stakes/domain unknown)
Why: [one sentence explaining the routing decision from data_form + label_status]
What this workflow will do: [one sentence about the output — e.g., "produces an
OpenJudge-compatible dataset with stratified sampling"]
To refine the path, also tell me: [stakes / domain knowledge, if still unknown]
Recommend exactly ONE workflow as the immediate next step. Do NOT list a second workflow as a current action — that splits the user's focus. If they ask "what comes after," point them to the Canonical Workflow below as a map for later, explicitly framed as "once you finish [recommended workflow]," not as a second thing to do now.
A ? marks a field you are still asking about. Give the recommendation now; refine later.
Most evaluation builds follow this order. Use it to sequence sub-skills and to state preconditions — recommend the next workflow only when its inputs exist.
1. 00-meta-eval route to the right entry workflow
2. entry point:
- have traces/spec → 01-eval-design (build the dataset)
- nothing at all → 08-bootstrap (uncalibrated v0 + roadmap to labels)
3. 02-metric-design select graders, build the GradingRunner pipeline
4. RUN the evaluation (produces scores; needed before any A/B or calibration)
5. 03-align-human ONLY once ≥50 human labels exist — calibrate before any
production gate. Production stakes REQUIRE this step.
6. scenario module (as needed):
- 05-rag-eval retrieval vs generation diagnosis
- 06-prompt-regression REQUIRES paired baseline+candidate outputs on shared
queries — do not route here before both prompts have
been run and their outputs collected
- 07-redteam policy-first safety + over-refusal
7. 04-eval-report synthesize maturity + ship readiness
Precondition rules to enforce when routing:
06-prompt-regression until the user has run both the baseline
and candidate prompts and has their outputs paired by query. Comparing prompts that
haven't been run yet is impossible.03-align-human calibration.
If stakes are production/regulated and no labels exist, the path MUST explicitly include
two steps before any ship decision: (1) collect ≥50 human labels, (2) run 03-align-human
to calibrate. State both steps every time production is in scope — an unlabeled system is
never production-ready, no matter how good the scores look.01-eval-design → 02-metric-design
→ run → collect labels → 03-align-human. Do not jump to bootstrap (you have data) or to
prompt-regression (no paired outputs yet).If you catch yourself thinking:
All of these mean: Stop. Return to the diagnostic questions.
01-eval-design.>
---name: meta-evaldescription: >---# skills/eval_pipeline/00-meta-eval/SKILL.md常見問題
從發布者下載 skills/eval_pipeline/00-meta-eval/SKILL.md,或複製路徑後開啟儲存庫。PlugMCP 只收錄這項技能,不代管檔案。
發布者將它存放在 https://github.com/agentscope-ai/OpenJudge。
Claude、Cursor、Codex 和 Grok Build 可以讀取你從發布者儲存庫加入的 SKILL.md。在聊天中傳送技能的 GitHub 資料夾連結,Grok Bot 就會安裝它。