- name
SKILL.md
स्किल फ़ाइल पब्लिशर की रिपॉज़िटरी में ही रहती है।
कॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add agentscope-ai/OpenJudge --skill 04-eval-report -a cursor -yकॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add agentscope-ai/OpenJudge --skill 04-eval-report -a claude-code -yकॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add agentscope-ai/OpenJudge --skill 04-eval-report -a codex -yकॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add agentscope-ai/OpenJudge --skill 04-eval-report -a grok -yकॉपी किया गया संदेश Grok Bot चैट में भेजें। Bot यह SKILL.md GitHub से लाता है और उसे उन स्किल में जोड़ देता है जो आपके सभी Bot साझा करते हैं।
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/eval_pipeline/04-eval-reportपब्लिशर से
Synthesize everything from your evaluation journey into a comprehensive report. This skill is read-only — it analyzes what exists, doesn't create new graders or datasets.
You MUST create a task for each item and complete them in order:
Read eval-design.md and all runs/ directories. Build a timeline:
Timeline:
2026-04-15 01-eval-design → 5 failure modes → 3 dimensions from 200 traces
2026-04-18 02-metric-design → 4 graders configured (2 LLM + 1 rule + 1 executable)
2026-04-25 (evaluation run) → 90-sample stratified dataset scored
2026-05-01 03-align-human → 2 judges Phase 3, 1 Phase 2, 1 Phase 1 (TPR/TNR + kappa)
2026-05-10 07-redteam → safety audit not yet run
Report key metrics:
Rate the evaluation system across 5 dimensions:
Scoring rule: The overall maturity level is the minimum across dimensions (weakest link principle). If 4 dimensions are L3 but Safety is L1, the system is L1.
Find themes confirmed by multiple skills. Example:
Find where skills disagree. These are the most valuable findings:
What hasn't been touched by any skill?
Which principle/grader has the lowest pass rate? Where are failures clustering?
Compute Jaccard similarity between principle pairs — when sample A fails on principle X, does it also fail on principle Y? Highly correlated pairs (Jaccard > 0.5) likely share a root cause.
Which difficulty stratum performs worst across all principles? If boundary stratum TPR < 0.7 for 3 of 4 principles, boundary discrimination is a systemic weakness.
For each weakness area, classify the root cause:
This classification is critical — fixing a metric problem by changing the system (or vice versa) wastes effort.
Generate P0/P1/P2 actions. Each must include: priority, concrete action, current state, target state, expected impact, and the skill to use.
🔴 P0 | Calibrate hallucination judge
Current: TPR=0.74 (below 0.8 threshold)
Target: TPR >= 0.8
Impact: Judge becomes usable as production gate
Use: 03-align-human
🔴 P0 | Add boundary samples for hallucination
Current: n=6 boundary samples (CI half-width ±18%)
Target: n >= 20 (CI narrows to ±10%)
Impact: Reliable per-stratum TPR measurement
Use: 01-eval-design
🟡 P1 | Align tone_consistency judge
Current: kappa=0.72, bias=-0.15 (lenient)
Target: kappa >= 0.8, |bias| < 0.1
Impact: Reduce false positive rate ~15%
Use: 03-align-human
🟢 P2 | Enable auto-gate for factuality judge
Current: kappa=0.87, TPR=0.92, TNR=0.88
Impact: Eliminate 90% of human review for this dimension
Use: 03-align-human (mark Phase 3)
One page for non-technical stakeholders:
Executive Summary
=================
System: Customer support chatbot for e-commerce
Stakes: Production
Report Date: 2026-05-12
SHIP READINESS: Conditional
2 items must be resolved before production gate:
1. Hallucination judge TPR below threshold (0.74 < 0.8)
2. No safety/redteam evaluation has been run
TOP 3 RISKS:
1. Hallucination detection unreliable — severity: HIGH
The judge measuring whether the bot fabricates information itself has poor
recall (TPR=0.74), meaning ~26% of hallucinations go undetected.
Mitigation: Calibrate with more boundary labels (2-3 weeks).
2. Safety coverage missing — severity: MEDIUM
No jailbreak, injection, or PII leakage testing has been performed.
Mitigation: Run redteam skill this sprint (1-2 days).
3. Tone evaluation is biased lenient — severity: LOW
The tone judge systematically rates responses as better than humans do.
Mitigation: Refine judge prompt with borderline examples.
EVAL MATURITY: L2 (Developing) → Target L3 in 3-4 weeks
Strongest: Failure Discovery (L3)
Weakest: Safety Coverage (L1), Human Alignment (L2)
NEXT ACTIONS (this sprint):
[P0] Add boundary labels + recalibrate hallucination judge
[P0] Run initial redteam evaluation
[P1] Refine tone judge alignment
Each per-metric run you synthesize should be a runs/<skill>/<ts>/results.json row of the form:
{"metric": "order_accuracy", "mean": 0.82, "ci_95": [0.78, 0.86], "n": 90,
"by_stratum": {"easy": 0.95, "boundary": 0.71, "adversarial": 0.60},
"verdict": "pass | fail | insufficient_evidence"}
After 04-eval-report:
>
---name: eval-reportdescription: >---# skills/eval_pipeline/04-eval-report/SKILL.mdFAQ
पब्लिशर से skills/eval_pipeline/04-eval-report/SKILL.md डाउनलोड करें, या पाथ कॉपी करके रिपॉज़िटरी खोलें। PlugMCP स्किल को सूचीबद्ध करता है, फ़ाइल होस्ट नहीं करता।
पब्लिशर इसे https://github.com/agentscope-ai/OpenJudge पर रखता है।
Claude, Cursor, Codex और Grok Build वह SKILL.md पढ़ सकते हैं जिसे आप पब्लिशर की रिपॉज़िटरी से जोड़ते हैं। जब आप चैट में स्किल के GitHub फ़ोल्डर का लिंक भेजते हैं, तो Grok Bot उसे इंस्टॉल कर देता है।