- name
SKILL.md
The skill file stays in the publisher's repository.
Paste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add agentscope-ai/OpenJudge --skill 04-eval-report -a cursor -yPaste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add agentscope-ai/OpenJudge --skill 04-eval-report -a claude-code -yPaste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add agentscope-ai/OpenJudge --skill 04-eval-report -a codex -yPaste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add agentscope-ai/OpenJudge --skill 04-eval-report -a grok -ySend the copied message in a Grok Bot chat. The Bot fetches this SKILL.md from GitHub and adds it to the skills all your Bots share.
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/eval_pipeline/04-eval-reportFrom the publisher
Synthesize everything from your evaluation journey into a comprehensive report. This skill is read-only — it analyzes what exists, doesn't create new graders or datasets.
You MUST create a task for each item and complete them in order:
Read eval-design.md and all runs/ directories. Build a timeline:
Timeline:
2026-04-15 01-eval-design → 5 failure modes → 3 dimensions from 200 traces
2026-04-18 02-metric-design → 4 graders configured (2 LLM + 1 rule + 1 executable)
2026-04-25 (evaluation run) → 90-sample stratified dataset scored
2026-05-01 03-align-human → 2 judges Phase 3, 1 Phase 2, 1 Phase 1 (TPR/TNR + kappa)
2026-05-10 07-redteam → safety audit not yet run
Report key metrics:
Rate the evaluation system across 5 dimensions:
Scoring rule: The overall maturity level is the minimum across dimensions (weakest link principle). If 4 dimensions are L3 but Safety is L1, the system is L1.
Find themes confirmed by multiple skills. Example:
Find where skills disagree. These are the most valuable findings:
What hasn't been touched by any skill?
Which principle/grader has the lowest pass rate? Where are failures clustering?
Compute Jaccard similarity between principle pairs — when sample A fails on principle X, does it also fail on principle Y? Highly correlated pairs (Jaccard > 0.5) likely share a root cause.
Which difficulty stratum performs worst across all principles? If boundary stratum TPR < 0.7 for 3 of 4 principles, boundary discrimination is a systemic weakness.
For each weakness area, classify the root cause:
This classification is critical — fixing a metric problem by changing the system (or vice versa) wastes effort.
Generate P0/P1/P2 actions. Each must include: priority, concrete action, current state, target state, expected impact, and the skill to use.
🔴 P0 | Calibrate hallucination judge
Current: TPR=0.74 (below 0.8 threshold)
Target: TPR >= 0.8
Impact: Judge becomes usable as production gate
Use: 03-align-human
🔴 P0 | Add boundary samples for hallucination
Current: n=6 boundary samples (CI half-width ±18%)
Target: n >= 20 (CI narrows to ±10%)
Impact: Reliable per-stratum TPR measurement
Use: 01-eval-design
🟡 P1 | Align tone_consistency judge
Current: kappa=0.72, bias=-0.15 (lenient)
Target: kappa >= 0.8, |bias| < 0.1
Impact: Reduce false positive rate ~15%
Use: 03-align-human
🟢 P2 | Enable auto-gate for factuality judge
Current: kappa=0.87, TPR=0.92, TNR=0.88
Impact: Eliminate 90% of human review for this dimension
Use: 03-align-human (mark Phase 3)
One page for non-technical stakeholders:
Executive Summary
=================
System: Customer support chatbot for e-commerce
Stakes: Production
Report Date: 2026-05-12
SHIP READINESS: Conditional
2 items must be resolved before production gate:
1. Hallucination judge TPR below threshold (0.74 < 0.8)
2. No safety/redteam evaluation has been run
TOP 3 RISKS:
1. Hallucination detection unreliable — severity: HIGH
The judge measuring whether the bot fabricates information itself has poor
recall (TPR=0.74), meaning ~26% of hallucinations go undetected.
Mitigation: Calibrate with more boundary labels (2-3 weeks).
2. Safety coverage missing — severity: MEDIUM
No jailbreak, injection, or PII leakage testing has been performed.
Mitigation: Run redteam skill this sprint (1-2 days).
3. Tone evaluation is biased lenient — severity: LOW
The tone judge systematically rates responses as better than humans do.
Mitigation: Refine judge prompt with borderline examples.
EVAL MATURITY: L2 (Developing) → Target L3 in 3-4 weeks
Strongest: Failure Discovery (L3)
Weakest: Safety Coverage (L1), Human Alignment (L2)
NEXT ACTIONS (this sprint):
[P0] Add boundary labels + recalibrate hallucination judge
[P0] Run initial redteam evaluation
[P1] Refine tone judge alignment
Each per-metric run you synthesize should be a runs/<skill>/<ts>/results.json row of the form:
{"metric": "order_accuracy", "mean": 0.82, "ci_95": [0.78, 0.86], "n": 90,
"by_stratum": {"easy": 0.95, "boundary": 0.71, "adversarial": 0.60},
"verdict": "pass | fail | insufficient_evidence"}
After 04-eval-report:
>
---name: eval-reportdescription: >---# skills/eval_pipeline/04-eval-report/SKILL.mdFAQ
Download skills/eval_pipeline/04-eval-report/SKILL.md from the publisher, or copy the path and open the repository. PlugMCP lists the skill and does not host the file.
The publisher keeps it at https://github.com/agentscope-ai/OpenJudge.
Claude, Cursor, Codex, and Grok Build can read a SKILL.md you add from the publisher's repository. Grok Bot installs it when you send the skill's GitHub folder link in a chat.