- name
SKILL.md
Tệp skill vẫn nằm trong kho mã của nhà phát hành.
Dán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add agentscope-ai/OpenJudge --skill 03-align-human -a cursor -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add agentscope-ai/OpenJudge --skill 03-align-human -a claude-code -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add agentscope-ai/OpenJudge --skill 03-align-human -a codex -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add agentscope-ai/OpenJudge --skill 03-align-human -a grok -yGửi tin nhắn đã sao chép trong cuộc trò chuyện Grok Bot. Bot sẽ lấy tệp SKILL.md này từ GitHub và thêm vào bộ skill mà mọi Bot của bạn dùng chung.
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/eval_pipeline/03-align-humanTừ nhà phát hành
Measure whether your automatic judge agrees with human judgment, detect where and why they disagree, and build a roadmap to reduce human review over time.
You MUST create a task for each item and complete them in order:
Don't hand-write the calibration statistics — that is exactly where subtle bugs hide. Run
the bundled, tested script (scripts/calibration.py, standard library only, no OpenJudge
dependency):
python scripts/calibration.py --pairs pairs.jsonl # one paired file, OR
python scripts/calibration.py --verdicts verdicts.jsonl --labels labels.jsonl --stratum-key difficulty
Paired rows look like {"id","judge":"pass|fail","human":"pass|fail","stratum"?} (judge/human
may also be 1/0). It prints the confusion matrix, TPR/TNR/F1 with bootstrap 95% CIs, Cohen's
kappa, Gwet's AC1 (auto-flags the kappa paradox), directional bias, per-stratum TPR/TNR, and
the calibration gate verdict (calibrated / not_calibrated / insufficient_evidence;
exit code 0 only if calibrated). --json for machine output, --self-test to verify it.
Always report and interpret the actual numbers the script returns — TPR/TNR (with their 95% CIs), Cohen's kappa, Gwet's AC1, directional bias, per-stratum TPR/TNR, and the gate verdict — never just state that you ran it. If you don't yet have the paired verdicts/labels, say exactly what's missing (e.g. the judge verdicts file, or N more labels per class).
The Steps below explain what each number means and how to act on it — read them to interpret the script's output. The inline snippets are the reference behind the script; you normally just run the script rather than re-implementing it.
Human labels live in labels/<grader_name>.jsonl, one row per judged sample. Keep them
separate from the dataset so they can be re-paired with any judge run:
{"id": "sample_017", "label": "pass", # "pass"|"fail" (or 1/0); joins to a dataset row id
"annotator": "alice", "rationale": "Order number matches context.",
"timestamp": "2026-06-20T10:00:00Z", "schema_version": 1}
Match judge verdicts with human labels:
import json
# Load human labels and judge verdicts
labels = {item["id"]: item["label"] for item in json.load(open("labels.jsonl"))}
verdicts = json.load(open("runs/verdicts-dev.jsonl"))
# Pair them
paired = []
for v in verdicts:
if v["id"] in labels:
paired.append({
"id": v["id"],
"judge": v["verdict"], # "pass" or "fail"
"human": labels[v["id"]], # "pass" or "fail"
})
print(f"Paired: {len(paired)}, Unmatched: {len(verdicts) - len(paired)}")
# Warn if severe class imbalance
pass_rate = sum(1 for p in paired if p["human"] == "pass") / len(paired)
if pass_rate > 0.8 or pass_rate < 0.2:
print(f"WARNING: Human label pass rate is {pass_rate:.0%} — "
"kappa may be paradoxically low. Use Gwet's AC1 as complement.")
Build a confusion matrix and compute per-stratum metrics:
from openjudge.analyzer.validation import (
AccuracyAnalyzer, F1ScoreAnalyzer,
FalsePositiveAnalyzer, FalseNegativeAnalyzer,
)
# Convert to OpenJudge-compatible dataset with labels
analysis_dataset = [
{"query": p.get("query", ""), "response": p.get("response", ""),
"label": 1 if p["human"] == "pass" else 0}
for p in paired
]
# Binary grader results (1=pass, 0=fail)
grader_results = [
GraderScore(name="judge", score=1.0 if p["judge"] == "pass" else 0.0, reason="")
for p in paired
]
accuracy = AccuracyAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
f1 = F1ScoreAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
fpr = FalsePositiveAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
fnr = FalseNegativeAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
# TPR = 1 - FNR, TNR = 1 - FPR
tpr = 1 - fnr.false_negative_rate
tnr = 1 - fpr.false_positive_rate
print(f"TPR={tpr:.2f}, TNR={tnr:.2f}, F1={f1.f1_score:.2f}")
# Per-stratum breakdown if difficulty data exists
for stratum in ["easy", "boundary", "hard"]:
stratum_data = [d for d in analysis_dataset
if d.get("metadata", {}).get("difficulty") == stratum]
if len(stratum_data) >= 10:
# Compute TPR/TNR per stratum
...
Why per-stratum matters: A judge with TPR=0.9 overall but TPR=0.5 on boundary cases is unreliable exactly where judgment matters most. This is the "progress illusion" (EMNLP 2025) — aggregate metrics hide stratum-level failure.
Use bootstrap resampling to quantify uncertainty:
import numpy as np
def bootstrap_ci(samples, metric_fn, n_iter=1000, ci=95):
"""Compute bootstrap confidence interval for a metric."""
n = len(samples)
values = []
for _ in range(n_iter):
idx = np.random.choice(n, n, replace=True)
resampled = [samples[i] for i in idx]
values.append(metric_fn(resampled))
lower = np.percentile(values, (100 - ci) / 2)
upper = np.percentile(values, 100 - (100 - ci) / 2)
return np.mean(values), lower, upper
tpr_mean, tpr_low, tpr_high = bootstrap_ci(
paired, lambda s: sum(1 for p in s if p["judge"] == "fail" and p["human"] == "fail")
/ max(1, sum(1 for p in s if p["human"] == "fail"))
)
kappa = (p_o - p_e) / (1 - p_e)
= 0.8: substantial — judge is consistent with humans
When 90% of samples are "pass," kappa can be paradoxically low even with high agreement. Gwet's AC1 corrects for this. If kappa and AC1 differ by > 0.15, report both and note the class imbalance effect.
bias = P(judge=fail | human=pass) - P(judge=pass | human=fail)
For samples where judge and human disagree, cluster them to find root causes:
judge_prompt_ambiguous: pass/fail definitions not clear for this casehuman_inconsistent: multiple human annotators disagreed on this sampletask_inherently_subjective: the dimension is fundamentally subjectivelabel_error: human label appears wrong, judge's reasoning is more convincingjudge_too_strict: judge applies criteria more harshly than humans intendjudge_too_lenient: judge overlooks issues humans catchPresent the top 3 patterns with 3 exemplars each so the user can decide whether to refine the judge prompt or accept the disagreement as inherent noise.
Present the alignment dashboard:
Alignment Results for [grader_name]:
TPR: 0.91 TNR: 0.88 F1: 0.90
Kappa: 0.87 AC1: 0.89 Bias: +0.03 (none)
95% CI (accuracy): [0.84, 0.93]
Per-stratum:
easy: TPR=0.96 TNR=0.94
boundary: TPR=0.82 TNR=0.79 ← weakest, monitor
hard: TPR=0.85 TNR=0.83
Bias checks:
✓ position: no bias detected
✓ verbosity: r=0.12 (clean)
✓ self-enh: judge model != target model
✓ progress: boundary gap 0.09 (acceptable)
✓ label drift: KL=0.08 (stable)
Phase: 3 (Auto-gate) — judge is calibrated and aligned
Disagreement patterns:
- 12 samples: judge slightly stricter on multi-step queries
Root cause: judge_prompt_ambiguous
Recommendation: add a multi-step borderline example to few-shot
Recommendation: [calibrated + aligned | needs refinement | not ready]
After 03-align-human:
04-eval-report: Generate a comprehensive report with maturity assessment.02-metric-design: If you need to redesign graders based on bias findings.01-eval-design: If disagreement patterns reveal dataset coverage gaps.>
---name: align-humandescription: >---# skills/eval_pipeline/03-align-human/SKILL.mdCâu hỏi thường gặp
Tải skills/eval_pipeline/03-align-human/SKILL.md từ nhà phát hành, hoặc sao chép đường dẫn và mở kho mã. PlugMCP chỉ liệt kê skill và không lưu trữ tệp này.
Nhà phát hành duy trì nó tại https://github.com/agentscope-ai/OpenJudge.
Claude, Cursor, Codex và Grok Build có thể đọc tệp SKILL.md mà bạn thêm từ kho mã của nhà phát hành. Grok Bot cài đặt skill khi bạn gửi liên kết thư mục GitHub của skill trong cuộc trò chuyện.