- name
SKILL.md
Le fichier du skill reste dans le dépôt de l'éditeur.
Collez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 03-align-human -a cursor -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 03-align-human -a claude-code -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 03-align-human -a codex -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 03-align-human -a grok -yEnvoyez le message copié dans un chat Grok Bot. Le Bot récupère ce SKILL.md sur GitHub et l'ajoute aux skills que partagent tous vos Bots.
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/eval_pipeline/03-align-humanDe l'éditeur
Measure whether your automatic judge agrees with human judgment, detect where and why they disagree, and build a roadmap to reduce human review over time.
You MUST create a task for each item and complete them in order:
Don't hand-write the calibration statistics — that is exactly where subtle bugs hide. Run
the bundled, tested script (scripts/calibration.py, standard library only, no OpenJudge
dependency):
python scripts/calibration.py --pairs pairs.jsonl # one paired file, OR
python scripts/calibration.py --verdicts verdicts.jsonl --labels labels.jsonl --stratum-key difficulty
Paired rows look like {"id","judge":"pass|fail","human":"pass|fail","stratum"?} (judge/human
may also be 1/0). It prints the confusion matrix, TPR/TNR/F1 with bootstrap 95% CIs, Cohen's
kappa, Gwet's AC1 (auto-flags the kappa paradox), directional bias, per-stratum TPR/TNR, and
the calibration gate verdict (calibrated / not_calibrated / insufficient_evidence;
exit code 0 only if calibrated). --json for machine output, --self-test to verify it.
Always report and interpret the actual numbers the script returns — TPR/TNR (with their 95% CIs), Cohen's kappa, Gwet's AC1, directional bias, per-stratum TPR/TNR, and the gate verdict — never just state that you ran it. If you don't yet have the paired verdicts/labels, say exactly what's missing (e.g. the judge verdicts file, or N more labels per class).
The Steps below explain what each number means and how to act on it — read them to interpret the script's output. The inline snippets are the reference behind the script; you normally just run the script rather than re-implementing it.
Human labels live in labels/<grader_name>.jsonl, one row per judged sample. Keep them
separate from the dataset so they can be re-paired with any judge run:
{"id": "sample_017", "label": "pass", # "pass"|"fail" (or 1/0); joins to a dataset row id
"annotator": "alice", "rationale": "Order number matches context.",
"timestamp": "2026-06-20T10:00:00Z", "schema_version": 1}
Match judge verdicts with human labels:
import json
# Load human labels and judge verdicts
labels = {item["id"]: item["label"] for item in json.load(open("labels.jsonl"))}
verdicts = json.load(open("runs/verdicts-dev.jsonl"))
# Pair them
paired = []
for v in verdicts:
if v["id"] in labels:
paired.append({
"id": v["id"],
"judge": v["verdict"], # "pass" or "fail"
"human": labels[v["id"]], # "pass" or "fail"
})
print(f"Paired: {len(paired)}, Unmatched: {len(verdicts) - len(paired)}")
# Warn if severe class imbalance
pass_rate = sum(1 for p in paired if p["human"] == "pass") / len(paired)
if pass_rate > 0.8 or pass_rate < 0.2:
print(f"WARNING: Human label pass rate is {pass_rate:.0%} — "
"kappa may be paradoxically low. Use Gwet's AC1 as complement.")
Build a confusion matrix and compute per-stratum metrics:
from openjudge.analyzer.validation import (
AccuracyAnalyzer, F1ScoreAnalyzer,
FalsePositiveAnalyzer, FalseNegativeAnalyzer,
)
# Convert to OpenJudge-compatible dataset with labels
analysis_dataset = [
{"query": p.get("query", ""), "response": p.get("response", ""),
"label": 1 if p["human"] == "pass" else 0}
for p in paired
]
# Binary grader results (1=pass, 0=fail)
grader_results = [
GraderScore(name="judge", score=1.0 if p["judge"] == "pass" else 0.0, reason="")
for p in paired
]
accuracy = AccuracyAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
f1 = F1ScoreAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
fpr = FalsePositiveAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
fnr = FalseNegativeAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
# TPR = 1 - FNR, TNR = 1 - FPR
tpr = 1 - fnr.false_negative_rate
tnr = 1 - fpr.false_positive_rate
print(f"TPR={tpr:.2f}, TNR={tnr:.2f}, F1={f1.f1_score:.2f}")
# Per-stratum breakdown if difficulty data exists
for stratum in ["easy", "boundary", "hard"]:
stratum_data = [d for d in analysis_dataset
if d.get("metadata", {}).get("difficulty") == stratum]
if len(stratum_data) >= 10:
# Compute TPR/TNR per stratum
...
Why per-stratum matters: A judge with TPR=0.9 overall but TPR=0.5 on boundary cases is unreliable exactly where judgment matters most. This is the "progress illusion" (EMNLP 2025) — aggregate metrics hide stratum-level failure.
Use bootstrap resampling to quantify uncertainty:
import numpy as np
def bootstrap_ci(samples, metric_fn, n_iter=1000, ci=95):
"""Compute bootstrap confidence interval for a metric."""
n = len(samples)
values = []
for _ in range(n_iter):
idx = np.random.choice(n, n, replace=True)
resampled = [samples[i] for i in idx]
values.append(metric_fn(resampled))
lower = np.percentile(values, (100 - ci) / 2)
upper = np.percentile(values, 100 - (100 - ci) / 2)
return np.mean(values), lower, upper
tpr_mean, tpr_low, tpr_high = bootstrap_ci(
paired, lambda s: sum(1 for p in s if p["judge"] == "fail" and p["human"] == "fail")
/ max(1, sum(1 for p in s if p["human"] == "fail"))
)
kappa = (p_o - p_e) / (1 - p_e)
= 0.8: substantial — judge is consistent with humans
When 90% of samples are "pass," kappa can be paradoxically low even with high agreement. Gwet's AC1 corrects for this. If kappa and AC1 differ by > 0.15, report both and note the class imbalance effect.
bias = P(judge=fail | human=pass) - P(judge=pass | human=fail)
For samples where judge and human disagree, cluster them to find root causes:
judge_prompt_ambiguous: pass/fail definitions not clear for this casehuman_inconsistent: multiple human annotators disagreed on this sampletask_inherently_subjective: the dimension is fundamentally subjectivelabel_error: human label appears wrong, judge's reasoning is more convincingjudge_too_strict: judge applies criteria more harshly than humans intendjudge_too_lenient: judge overlooks issues humans catchPresent the top 3 patterns with 3 exemplars each so the user can decide whether to refine the judge prompt or accept the disagreement as inherent noise.
Present the alignment dashboard:
Alignment Results for [grader_name]:
TPR: 0.91 TNR: 0.88 F1: 0.90
Kappa: 0.87 AC1: 0.89 Bias: +0.03 (none)
95% CI (accuracy): [0.84, 0.93]
Per-stratum:
easy: TPR=0.96 TNR=0.94
boundary: TPR=0.82 TNR=0.79 ← weakest, monitor
hard: TPR=0.85 TNR=0.83
Bias checks:
✓ position: no bias detected
✓ verbosity: r=0.12 (clean)
✓ self-enh: judge model != target model
✓ progress: boundary gap 0.09 (acceptable)
✓ label drift: KL=0.08 (stable)
Phase: 3 (Auto-gate) — judge is calibrated and aligned
Disagreement patterns:
- 12 samples: judge slightly stricter on multi-step queries
Root cause: judge_prompt_ambiguous
Recommendation: add a multi-step borderline example to few-shot
Recommendation: [calibrated + aligned | needs refinement | not ready]
After 03-align-human:
04-eval-report: Generate a comprehensive report with maturity assessment.02-metric-design: If you need to redesign graders based on bias findings.01-eval-design: If disagreement patterns reveal dataset coverage gaps.>
---name: align-humandescription: >---# skills/eval_pipeline/03-align-human/SKILL.mdFAQ
Téléchargez skills/eval_pipeline/03-align-human/SKILL.md auprès de l'éditeur, ou copiez le chemin et ouvrez le dépôt. PlugMCP répertorie le skill et n'héberge pas le fichier.
L'éditeur le maintient sur https://github.com/agentscope-ai/OpenJudge.
Claude, Cursor, Codex et Grok Build peuvent lire un SKILL.md que vous ajoutez depuis le dépôt de l'éditeur. Grok Bot l'installe lorsque vous envoyez dans un chat le lien du dossier GitHub du skill.