- name
SKILL.md
Le fichier du skill reste dans le dépôt de l'éditeur.
Collez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 01-eval-design -a cursor -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 01-eval-design -a claude-code -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 01-eval-design -a codex -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 01-eval-design -a grok -yEnvoyez le message copié dans un chat Grok Bot. Le Bot récupère ce SKILL.md sur GitHub et l'ajoute aux skills que partagent tous vos Bots.
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/eval_pipeline/01-eval-designDe l'éditeur
Design high-quality evaluation datasets that measure what actually matters for your
application. You extract evaluation dimensions from business context, structure them
into stratified test cases, and output datasets ready for OpenJudge GradingRunner.
You MUST create a task for each item and complete them in order:
After you have a dataset, validate coverage with the bundled, tested script
(scripts/coverage_check.py, standard library only, no OpenJudge dependency) before
trusting any per-slice metric:
python scripts/coverage_check.py --dataset eval-data/dataset.jsonl
It reports per-dimension and per-(dimension × stratum) counts, flags thin cells
(< 5 per dimension, < 10 per cell), checks the adversarial share (≥ 10%), and returns a
verdict (adequate / thin_coverage; exit 0 if adequate). --self-test to verify it.
Read the user's agent traces to identify what can go wrong:
order_accuracy dimension.Read the spec / design doc and extract:
Ask the user to describe (in one go, not question-by-question):
Briefly describe:
- Who uses this system and what do they ask it to do?
- What are 3 examples of a perfect response?
- What are 3 examples of an unacceptable response?
- What failures keep you up at night?
- Are there any hard red lines the system must never cross?
# Write this into the user's project as eval-design.md frontmatter
scenario: "Customer support chatbot for e-commerce"
stakes: production
dimensions:
- id: order_accuracy
criterion: "Order number, status, and tracking info must match the backend"
priority: P0
source: trace_failure_cluster
- id: tone_appropriateness
criterion: "Response tone matches customer sentiment"
priority: P1
source: spec
- id: no_hallucination
criterion: "No fabricated policies, prices, or product features"
priority: P0
source: hard_red_line
A flat random sample hides systematic failures. Stratify by difficulty so your eval detects degradation where it matters most.
Don't guess. Use this rule: for per-stratum TPR/TNR to be meaningful, each stratum needs at least 10 samples (binomial CI at n=10, p=0.5 → half-width ~±15%). For production use, target 30+ per stratum (CI narrows to ~±9%).
# Minimum viable: 10 samples × 3 strata = 30 per dimension
# Production target: 30 samples × 3 strata = 90 per dimension
For each eval dimension, cover four types of cases (adapted from community practice):
Use 3-5 different prompt templates to generate diverse synthetic inputs. Diversity of the generation prompt matters more than the number of outputs — 5 prompts × 10 outputs each beats 1 prompt × 50 outputs.
Template examples:
1. "Generate a {scenario} query where the user {action} with {constraint}"
2. "Write a frustrated customer message about {failure_mode}"
3. "Create an ambiguous query that could mean either {intent_a} or {intent_b}"
4. "Generate a query in {non_english_language} about {domain}"
5. "Create a query with a typo/misspelling about {domain}"
Critical rule: You generate inputs ONLY. Never generate labels. Labels must come from real system output + human judgment (or deterministic rules). An LLM generating both inputs and labels creates a self-consistency loop with artificially inflated accuracy.
For each dimension, generate 3 types of adversarial inputs:
If human annotation is needed, provide a template:
## Annotation Task: [dimension_name]
**Criterion**: [what the dimension measures]
**Pass**: [concrete, observable conditions for pass]
**Fail**: [concrete, observable conditions for fail]
**Examples**:
- Input: "..." | Output: "..." | Judgment: Pass | Reason: ...
- Input: "..." | Output: "..." | Judgment: Fail | Reason: ...
**Edge cases**:
- If X happens but Y doesn't → [how to judge]
- If both A and B are present → [which takes priority]
Format the dataset for direct use with OpenJudge GradingRunner:
# The standard dataset format accepted by GradingRunner.arun()
dataset = [
{
"query": "Where is my order #12345?",
"response": "Your order #12345 was shipped on May 10 and is expected to arrive May 12.",
"reference_response": "Order #12345: shipped May 10, ETA May 12. Tracking: 1Z999AA10123456784.",
"context": "Order #12345 | Status: shipped | Date: 2026-05-10 | Carrier: UPS | Tracking: 1Z999AA10123456784",
"metadata": {
"difficulty": "easy",
"dimension": "order_accuracy",
"quadrant": "happy_path"
}
},
{
"query": "My package hasn't moved in 3 days, this is ridiculous",
"response": "I understand your frustration. Let me check tracking for your recent orders.",
"reference_response": None,
"context": "Customer has 2 active orders: #12345 (in transit, last scan 2026-05-09), #12346 (processing)",
"metadata": {
"difficulty": "boundary",
"dimension": "tone_appropriateness",
"quadrant": "boundary"
}
},
]
After running this skill:
After 01-eval-design:
02-metric-design: You have a dataset. Now select graders and build the evaluation pipeline.03-align-human: If you have human labels, calibrate your judge against them.08-bootstrap: If you're still exploring and want a quick v0 grader before full dataset design.>
---name: eval-designdescription: >---# skills/eval_pipeline/01-eval-design/SKILL.mdFAQ
Téléchargez skills/eval_pipeline/01-eval-design/SKILL.md auprès de l'éditeur, ou copiez le chemin et ouvrez le dépôt. PlugMCP répertorie le skill et n'héberge pas le fichier.
L'éditeur le maintient sur https://github.com/agentscope-ai/OpenJudge.
Claude, Cursor, Codex et Grok Build peuvent lire un SKILL.md que vous ajoutez depuis le dépôt de l'éditeur. Grok Bot l'installe lorsque vous envoyez dans un chat le lien du dossier GitHub du skill.