- name
SKILL.md
स्किल फ़ाइल पब्लिशर की रिपॉज़िटरी में ही रहती है।
कॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add agentscope-ai/OpenJudge --skill 00-meta-eval -a cursor -yकॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add agentscope-ai/OpenJudge --skill 00-meta-eval -a claude-code -yकॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add agentscope-ai/OpenJudge --skill 00-meta-eval -a codex -yकॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add agentscope-ai/OpenJudge --skill 00-meta-eval -a grok -yकॉपी किया गया संदेश Grok Bot चैट में भेजें। Bot यह SKILL.md GitHub से लाता है और उसे उन स्किल में जोड़ देता है जो आपके सभी Bot साझा करते हैं।
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/eval_pipeline/00-meta-evalपब्लिशर से
Entry router for the eval skill collection. You diagnose what the user has and route them to the right sub-skill. You don't do evaluation yourself — you're the triage desk.
Each sub-skill is self-contained: it carries inline the data shapes, statistics, and data principles it needs, so it can be installed and used on its own.
You MUST create a task for each item and complete them in order:
Ask these 4 questions (all at once — don't drip-feed):
To route you to the right evaluation skill, I need to understand your situation:
1. What data do you have?
a) Agent traces / production logs
b) Product spec / design docs
c) Nothing yet — starting from scratch
2. Do you have human labels?
a) Yes, ≥50 labeled examples
b) Some, but fewer than 50
c) None
3. What are the stakes?
a) Low — internal experimentation, exploring options
b) Production — customer-facing, quality matters
c) Regulated — compliance requirements, audit trail needed
4. How well do you know this evaluation domain?
a) Very well — have clear standards and criteria
b) Somewhat — general idea but need structure
c) Not well — exploring what "good" even means
Shortcut rule: data_form + label_status already determine the entry workflow
(see triage table). The moment those two are clear — even if stakes and domain knowledge
are not — give the provisional recommendation AND ask the remaining questions in the same
message. stakes and user_prior refine the downstream path (how much calibration rigor,
how fast a path), not the entry point. Never make the user wait a round-trip for a route you
can already determine.
Example: "no logs, no labels" → recommend 08-bootstrap now, and ask stakes/domain to tune
the roadmap. Don't reply with only the questionnaire.
Match the user's situation to a sub-skill:
These are local workflows under skills/eval_pipeline/, not packages to install — "use"
a workflow means open and follow that sub-skill.
After diagnosis, respond with:
Diagnosis: data=[data_form] | labels=[label_status] | stakes=[value or "asking"] | domain=[value or "asking"]
Recommended workflow: `[skill-name]` (provisional if stakes/domain unknown)
Why: [one sentence explaining the routing decision from data_form + label_status]
What this workflow will do: [one sentence about the output — e.g., "produces an
OpenJudge-compatible dataset with stratified sampling"]
To refine the path, also tell me: [stakes / domain knowledge, if still unknown]
Recommend exactly ONE workflow as the immediate next step. Do NOT list a second workflow as a current action — that splits the user's focus. If they ask "what comes after," point them to the Canonical Workflow below as a map for later, explicitly framed as "once you finish [recommended workflow]," not as a second thing to do now.
A ? marks a field you are still asking about. Give the recommendation now; refine later.
Most evaluation builds follow this order. Use it to sequence sub-skills and to state preconditions — recommend the next workflow only when its inputs exist.
1. 00-meta-eval route to the right entry workflow
2. entry point:
- have traces/spec → 01-eval-design (build the dataset)
- nothing at all → 08-bootstrap (uncalibrated v0 + roadmap to labels)
3. 02-metric-design select graders, build the GradingRunner pipeline
4. RUN the evaluation (produces scores; needed before any A/B or calibration)
5. 03-align-human ONLY once ≥50 human labels exist — calibrate before any
production gate. Production stakes REQUIRE this step.
6. scenario module (as needed):
- 05-rag-eval retrieval vs generation diagnosis
- 06-prompt-regression REQUIRES paired baseline+candidate outputs on shared
queries — do not route here before both prompts have
been run and their outputs collected
- 07-redteam policy-first safety + over-refusal
7. 04-eval-report synthesize maturity + ship readiness
Precondition rules to enforce when routing:
06-prompt-regression until the user has run both the baseline
and candidate prompts and has their outputs paired by query. Comparing prompts that
haven't been run yet is impossible.03-align-human calibration.
If stakes are production/regulated and no labels exist, the path MUST explicitly include
two steps before any ship decision: (1) collect ≥50 human labels, (2) run 03-align-human
to calibrate. State both steps every time production is in scope — an unlabeled system is
never production-ready, no matter how good the scores look.01-eval-design → 02-metric-design
→ run → collect labels → 03-align-human. Do not jump to bootstrap (you have data) or to
prompt-regression (no paired outputs yet).If you catch yourself thinking:
All of these mean: Stop. Return to the diagnostic questions.
01-eval-design.>
---name: meta-evaldescription: >---# skills/eval_pipeline/00-meta-eval/SKILL.mdFAQ
पब्लिशर से skills/eval_pipeline/00-meta-eval/SKILL.md डाउनलोड करें, या पाथ कॉपी करके रिपॉज़िटरी खोलें। PlugMCP स्किल को सूचीबद्ध करता है, फ़ाइल होस्ट नहीं करता।
पब्लिशर इसे https://github.com/agentscope-ai/OpenJudge पर रखता है।
Claude, Cursor, Codex और Grok Build वह SKILL.md पढ़ सकते हैं जिसे आप पब्लिशर की रिपॉज़िटरी से जोड़ते हैं। जब आप चैट में स्किल के GitHub फ़ोल्डर का लिंक भेजते हैं, तो Grok Bot उसे इंस्टॉल कर देता है।