- name
SKILL.md
The skill file stays in the publisher's repository.
Paste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add agentscope-ai/OpenJudge --skill 06-prompt-regression -a cursor -yPaste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add agentscope-ai/OpenJudge --skill 06-prompt-regression -a claude-code -yPaste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add agentscope-ai/OpenJudge --skill 06-prompt-regression -a codex -yPaste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add agentscope-ai/OpenJudge --skill 06-prompt-regression -a grok -ySend the copied message in a Grok Bot chat. The Bot fetches this SKILL.md from GitHub and adds it to the skills all your Bots share.
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/eval_pipeline/06-prompt-regressionFrom the publisher
Compare two prompts head-to-head and determine, with statistical rigor, whether the candidate is better, worse, or tied on each evaluation dimension.
You MUST create a task for each item and complete them in order:
Don't hand-write the win-rate + bootstrap math (the swap-aggregation and CI are easy to get
wrong). Run the bundled, tested script (scripts/pairwise.py, standard library only, no
OpenJudge dependency):
python scripts/pairwise.py --comparisons comparisons.jsonl --candidate candidate --baseline baseline
Each comparison row: {"id","model_a","model_b","score","dimension"?} where score >= 0.5
means model_a won. Emit two rows per query with A/B swapped to debias position. The
script reports per-dimension candidate/baseline/tie rates, bootstrap 95% CI, and a verdict
(BETTER / WORSE / TIED / INSUFFICIENT_EVIDENCE / INCONCLUSIVE; exit 0 only if
better). --self-test to verify it.
Steps below explain how to derive dimensions and produce the comparisons (with OpenJudge or any judge); the inline snippets are the reference behind the script.
Read the baseline and candidate prompts. Identify:
Based on the task type and what changed, derive 3-5 comparison dimensions.
Chatbot / Conversational:
RAG Generation:
Code Review / Generation:
Agent Instructions:
Each dimension gets:
id (slug)pairwise or judge or ruleDecision priority:
FunctionGrader or StringMatchGrader. Free, deterministic.
Example: output length, keyword presence, JSON validity.pairwise against reference.pairwise A/B comparison.judge (binary pass/fail per output).LLM judges have position bias — the first response shown wins 5-15% more often. Swap-aggregate eliminates this: run each comparison twice with swapped positions, keep only consistent wins:
from openjudge.graders.llm_grader import LLMGrader
from openjudge.graders.schema import GraderMode
from openjudge.runner.grading_runner import GradingRunner
from openjudge.analyzer.pairwise_analyzer import PairwiseAnalyzer
# Judge prompt for relevance comparison
relevance_judge = LLMGrader(
model=model,
name="relevance_compare",
mode=GraderMode.POINTWISE,
template="""
Compare Response A and Response B for the query below.
Which response better addresses the user's question?
Query: {query}
Response A: {response_a}
Response B: {response_b}
Score 1.0 if A is better, 0.0 if B is better, 0.5 if tied.
Respond in JSON: {{"score": <float>, "reason": "<explanation>"}}
""",
)
# Build pairwise dataset with position swap
dataset = []
for sample in test_samples:
# Original order
dataset.append({
"query": sample["query"],
"response_a": baseline_outputs[sample["id"]],
"response_b": candidate_outputs[sample["id"]],
"metadata": {"model_a": "baseline", "model_b": "candidate"},
})
# Swapped order — critical for debiasing
dataset.append({
"query": sample["query"],
"response_a": candidate_outputs[sample["id"]],
"response_b": baseline_outputs[sample["id"]],
"metadata": {"model_a": "candidate", "model_b": "baseline"},
})
runner = GradingRunner(
grader_configs={"relevance": relevance_judge},
max_concurrency=8,
)
results = await runner.arun(dataset)
# Analyze with PairwiseAnalyzer
analyzer = PairwiseAnalyzer(model_names=["baseline", "candidate"])
analysis = analyzer.analyze(dataset, results["relevance"])
print(f"Win rates: {analysis.win_rates}")
# → {'baseline': 0.35, 'candidate': 0.55} → candidate wins 55% of comparisons
print(f"Best model: {analysis.best_model}")
Why swap-aggregate? Without it, if the judge prefers the first response shown, and you always show baseline first, you'll systematically underrate the candidate.
For each dimension, report:
PairwiseAnalyzer.analyze interprets each comparison as score >= 0.5 → model_a wins,
using the row's metadata.model_a / metadata.model_b. So derive a per-comparison
winner list from dataset + results, then bootstrap over that list — never index the
PairwiseAnalysisResult object (it has no per-sample rows).
import numpy as np
from openjudge.graders.schema import GraderScore
def per_comparison_winners(dataset, grader_results):
"""One named winner per comparison row (handles swapped order via metadata)."""
winners = []
for sample, result in zip(dataset, grader_results):
if not isinstance(result, GraderScore):
continue # skip errors
meta = sample.get("metadata", {})
winners.append(meta["model_a"] if result.score >= 0.5 else meta["model_b"])
return winners
def bootstrap_win_rate(winners, target, n_iter=1000):
n = len(winners)
rates = []
for _ in range(n_iter):
idx = np.random.choice(n, n, replace=True)
rates.append(sum(1 for i in idx if winners[i] == target) / n)
return float(np.percentile(rates, 2.5)), float(np.percentile(rates, 97.5))
winners = per_comparison_winners(dataset, results["relevance"])
n = len(winners)
candidate_rate = sum(1 for w in winners if w == "candidate") / n
baseline_rate = sum(1 for w in winners if w == "baseline") / n
ci_low, ci_high = bootstrap_win_rate(winners, target="candidate")
if ci_low > 0.5:
verdict = "candidate BETTER"
elif ci_high < 0.5:
verdict = "candidate WORSE"
elif (ci_high - ci_low) < 0.3:
verdict = "TIED (CI brackets 0.5, narrow)"
else:
verdict = "INCONCLUSIVE (CI too wide — need more samples)"
print({"candidate_win_rate": candidate_rate, "baseline_win_rate": baseline_rate,
"ci_95": [ci_low, ci_high], "verdict": verdict})
Note: with swap-aggregate each query produces 2 comparison rows. Bootstrapping over rows (above) is the simple approach; for a tighter estimate, bootstrap over queries and average the 2 swapped rows per query so position pairs stay together.
Prompt Regression: v1 (baseline) vs v2 (candidate)
Task: Customer support chatbot
Samples: 50
Dimension Candidate Baseline Tie 95% CI Verdict
===========================================================================
Answer relevance 58% 32% 10% [51%, 65%] ✓ BETTER
Factual accuracy 48% 44% 8% [41%, 55%] = TIED
Tone appropriateness 38% 52% 10% [31%, 45%] ✗ WORSE
Conciseness 62% 28% 10% [55%, 69%] ✓ BETTER
Summary: v2 is significantly better on relevance and conciseness,
but worse on tone appropriateness. The tone regression likely comes
from the new "be direct" instruction — consider softening it.
Top 3 tone failures (candidate worse):
1. Query: "I'm really frustrated..." → v2 response too curt
2. Query: "This is my first time..." → v2 missing empathetic opening
3. Query: "Can you help me understand..." → v2 skipped explanation
After 06-prompt-regression:
03-align-human: Calibrate the pairwise judge against human preferences.02-metric-design: Turn validated dimensions into permanent graders.04-eval-report: Include prompt comparison results in a comprehensive report.>
---name: prompt-regressiondescription: >---# skills/eval_pipeline/06-prompt-regression/SKILL.mdFAQ
Download skills/eval_pipeline/06-prompt-regression/SKILL.md from the publisher, or copy the path and open the repository. PlugMCP lists the skill and does not host the file.
The publisher keeps it at https://github.com/agentscope-ai/OpenJudge.
Claude, Cursor, Codex, and Grok Build can read a SKILL.md you add from the publisher's repository. Grok Bot installs it when you send the skill's GitHub folder link in a chat.