- name
SKILL.md
Tệp skill vẫn nằm trong kho mã của nhà phát hành.
Dán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add agentscope-ai/OpenJudge --skill 06-prompt-regression -a cursor -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add agentscope-ai/OpenJudge --skill 06-prompt-regression -a claude-code -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add agentscope-ai/OpenJudge --skill 06-prompt-regression -a codex -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add agentscope-ai/OpenJudge --skill 06-prompt-regression -a grok -yGửi tin nhắn đã sao chép trong cuộc trò chuyện Grok Bot. Bot sẽ lấy tệp SKILL.md này từ GitHub và thêm vào bộ skill mà mọi Bot của bạn dùng chung.
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/eval_pipeline/06-prompt-regressionTừ nhà phát hành
Compare two prompts head-to-head and determine, with statistical rigor, whether the candidate is better, worse, or tied on each evaluation dimension.
You MUST create a task for each item and complete them in order:
Don't hand-write the win-rate + bootstrap math (the swap-aggregation and CI are easy to get
wrong). Run the bundled, tested script (scripts/pairwise.py, standard library only, no
OpenJudge dependency):
python scripts/pairwise.py --comparisons comparisons.jsonl --candidate candidate --baseline baseline
Each comparison row: {"id","model_a","model_b","score","dimension"?} where score >= 0.5
means model_a won. Emit two rows per query with A/B swapped to debias position. The
script reports per-dimension candidate/baseline/tie rates, bootstrap 95% CI, and a verdict
(BETTER / WORSE / TIED / INSUFFICIENT_EVIDENCE / INCONCLUSIVE; exit 0 only if
better). --self-test to verify it.
Steps below explain how to derive dimensions and produce the comparisons (with OpenJudge or any judge); the inline snippets are the reference behind the script.
Read the baseline and candidate prompts. Identify:
Based on the task type and what changed, derive 3-5 comparison dimensions.
Chatbot / Conversational:
RAG Generation:
Code Review / Generation:
Agent Instructions:
Each dimension gets:
id (slug)pairwise or judge or ruleDecision priority:
FunctionGrader or StringMatchGrader. Free, deterministic.
Example: output length, keyword presence, JSON validity.pairwise against reference.pairwise A/B comparison.judge (binary pass/fail per output).LLM judges have position bias — the first response shown wins 5-15% more often. Swap-aggregate eliminates this: run each comparison twice with swapped positions, keep only consistent wins:
from openjudge.graders.llm_grader import LLMGrader
from openjudge.graders.schema import GraderMode
from openjudge.runner.grading_runner import GradingRunner
from openjudge.analyzer.pairwise_analyzer import PairwiseAnalyzer
# Judge prompt for relevance comparison
relevance_judge = LLMGrader(
model=model,
name="relevance_compare",
mode=GraderMode.POINTWISE,
template="""
Compare Response A and Response B for the query below.
Which response better addresses the user's question?
Query: {query}
Response A: {response_a}
Response B: {response_b}
Score 1.0 if A is better, 0.0 if B is better, 0.5 if tied.
Respond in JSON: {{"score": <float>, "reason": "<explanation>"}}
""",
)
# Build pairwise dataset with position swap
dataset = []
for sample in test_samples:
# Original order
dataset.append({
"query": sample["query"],
"response_a": baseline_outputs[sample["id"]],
"response_b": candidate_outputs[sample["id"]],
"metadata": {"model_a": "baseline", "model_b": "candidate"},
})
# Swapped order — critical for debiasing
dataset.append({
"query": sample["query"],
"response_a": candidate_outputs[sample["id"]],
"response_b": baseline_outputs[sample["id"]],
"metadata": {"model_a": "candidate", "model_b": "baseline"},
})
runner = GradingRunner(
grader_configs={"relevance": relevance_judge},
max_concurrency=8,
)
results = await runner.arun(dataset)
# Analyze with PairwiseAnalyzer
analyzer = PairwiseAnalyzer(model_names=["baseline", "candidate"])
analysis = analyzer.analyze(dataset, results["relevance"])
print(f"Win rates: {analysis.win_rates}")
# → {'baseline': 0.35, 'candidate': 0.55} → candidate wins 55% of comparisons
print(f"Best model: {analysis.best_model}")
Why swap-aggregate? Without it, if the judge prefers the first response shown, and you always show baseline first, you'll systematically underrate the candidate.
For each dimension, report:
PairwiseAnalyzer.analyze interprets each comparison as score >= 0.5 → model_a wins,
using the row's metadata.model_a / metadata.model_b. So derive a per-comparison
winner list from dataset + results, then bootstrap over that list — never index the
PairwiseAnalysisResult object (it has no per-sample rows).
import numpy as np
from openjudge.graders.schema import GraderScore
def per_comparison_winners(dataset, grader_results):
"""One named winner per comparison row (handles swapped order via metadata)."""
winners = []
for sample, result in zip(dataset, grader_results):
if not isinstance(result, GraderScore):
continue # skip errors
meta = sample.get("metadata", {})
winners.append(meta["model_a"] if result.score >= 0.5 else meta["model_b"])
return winners
def bootstrap_win_rate(winners, target, n_iter=1000):
n = len(winners)
rates = []
for _ in range(n_iter):
idx = np.random.choice(n, n, replace=True)
rates.append(sum(1 for i in idx if winners[i] == target) / n)
return float(np.percentile(rates, 2.5)), float(np.percentile(rates, 97.5))
winners = per_comparison_winners(dataset, results["relevance"])
n = len(winners)
candidate_rate = sum(1 for w in winners if w == "candidate") / n
baseline_rate = sum(1 for w in winners if w == "baseline") / n
ci_low, ci_high = bootstrap_win_rate(winners, target="candidate")
if ci_low > 0.5:
verdict = "candidate BETTER"
elif ci_high < 0.5:
verdict = "candidate WORSE"
elif (ci_high - ci_low) < 0.3:
verdict = "TIED (CI brackets 0.5, narrow)"
else:
verdict = "INCONCLUSIVE (CI too wide — need more samples)"
print({"candidate_win_rate": candidate_rate, "baseline_win_rate": baseline_rate,
"ci_95": [ci_low, ci_high], "verdict": verdict})
Note: with swap-aggregate each query produces 2 comparison rows. Bootstrapping over rows (above) is the simple approach; for a tighter estimate, bootstrap over queries and average the 2 swapped rows per query so position pairs stay together.
Prompt Regression: v1 (baseline) vs v2 (candidate)
Task: Customer support chatbot
Samples: 50
Dimension Candidate Baseline Tie 95% CI Verdict
===========================================================================
Answer relevance 58% 32% 10% [51%, 65%] ✓ BETTER
Factual accuracy 48% 44% 8% [41%, 55%] = TIED
Tone appropriateness 38% 52% 10% [31%, 45%] ✗ WORSE
Conciseness 62% 28% 10% [55%, 69%] ✓ BETTER
Summary: v2 is significantly better on relevance and conciseness,
but worse on tone appropriateness. The tone regression likely comes
from the new "be direct" instruction — consider softening it.
Top 3 tone failures (candidate worse):
1. Query: "I'm really frustrated..." → v2 response too curt
2. Query: "This is my first time..." → v2 missing empathetic opening
3. Query: "Can you help me understand..." → v2 skipped explanation
After 06-prompt-regression:
03-align-human: Calibrate the pairwise judge against human preferences.02-metric-design: Turn validated dimensions into permanent graders.04-eval-report: Include prompt comparison results in a comprehensive report.>
---name: prompt-regressiondescription: >---# skills/eval_pipeline/06-prompt-regression/SKILL.mdCâu hỏi thường gặp
Tải skills/eval_pipeline/06-prompt-regression/SKILL.md từ nhà phát hành, hoặc sao chép đường dẫn và mở kho mã. PlugMCP chỉ liệt kê skill và không lưu trữ tệp này.
Nhà phát hành duy trì nó tại https://github.com/agentscope-ai/OpenJudge.
Claude, Cursor, Codex và Grok Build có thể đọc tệp SKILL.md mà bạn thêm từ kho mã của nhà phát hành. Grok Bot cài đặt skill khi bạn gửi liên kết thư mục GitHub của skill trong cuộc trò chuyện.