- name
SKILL.md
技能檔案保留在發布者的儲存庫中。
在終端機中貼上複製的指令。它會將這個 SKILL.md 安裝到對應的代理中。
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a cursor -y在終端機中貼上複製的指令。它會將這個 SKILL.md 安裝到對應的代理中。
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a claude-code -y在終端機中貼上複製的指令。它會將這個 SKILL.md 安裝到對應的代理中。
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a codex -y在終端機中貼上複製的指令。它會將這個 SKILL.md 安裝到對應的代理中。
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a grok -y在 Grok Bot 聊天中傳送複製的訊息。Bot 會從 GitHub 取得這個 SKILL.md,並加入你所有 Bot 共用的技能中。
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/arena-eval/01-auto-arena來自發布者
End-to-end automated model comparison using the OpenJudge AutoArenaPipeline:
# Install OpenJudge
pip install py-openjudge
# Extra dependency for auto_arena (chart generation)
pip install matplotlib
# Run evaluation
python -m cookbooks.auto_arena --config config.yaml --save
# Use pre-generated queries
python -m cookbooks.auto_arena --config config.yaml \
--queries_file queries.json --save
# Start fresh, ignore checkpoint
python -m cookbooks.auto_arena --config config.yaml --fresh --save
# Re-run only pairwise evaluation with new judge model
# (keeps queries, responses, and rubrics)
python -m cookbooks.auto_arena --config config.yaml --rerun-judge --save
import asyncio
from cookbooks.auto_arena.auto_arena_pipeline import AutoArenaPipeline
async def main():
pipeline = AutoArenaPipeline.from_config("config.yaml")
result = await pipeline.evaluate()
print(f"Best model: {result.best_pipeline}")
for rank, (model, win_rate) in enumerate(result.rankings, 1):
print(f"{rank}. {model}: {win_rate:.1%}")
asyncio.run(main())
import asyncio
from cookbooks.auto_arena.auto_arena_pipeline import AutoArenaPipeline
from cookbooks.auto_arena.schema import OpenAIEndpoint
async def main():
pipeline = AutoArenaPipeline(
task_description="Customer service chatbot for e-commerce",
target_endpoints={
"gpt4": OpenAIEndpoint(
base_url="https://api.openai.com/v1",
api_key="sk-...",
model="gpt-4",
),
"qwen": OpenAIEndpoint(
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
api_key="sk-...",
model="qwen-max",
),
},
judge_endpoint=OpenAIEndpoint(
base_url="https://api.openai.com/v1",
api_key="sk-...",
model="gpt-4",
),
num_queries=20,
)
result = await pipeline.evaluate()
print(f"Best: {result.best_pipeline}")
asyncio.run(main())
task:
description: "Academic GPT assistant for research and writing tasks"
target_endpoints:
model_v1:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-4"
model_v2:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-3.5-turbo"
judge_endpoint:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-4"
Same fields as target_endpoints.<name>. Use a strong model (e.g. gpt-4, qwen-max) with low temperature (~0.1) for consistent judgments.
Win rate: percentage of pairwise comparisons a model wins. Each pair is evaluated in both orders (original + swapped) to eliminate position bias.
Rankings example:
1. gpt4_baseline [################----] 80.0%
2. qwen_candidate [############--------] 60.0%
3. llama_finetuned [##########----------] 50.0%
Win matrix: win_matrix[A][B] = how often model A beats model B across all queries.
The pipeline saves progress after each step. Interrupted runs resume automatically:
--fresh — ignore checkpoint, start from scratch--rerun-judge — re-run only the pairwise evaluation step (useful when switching judge models); keeps queries, responses, and rubrics intactevaluation_results/
├── evaluation_results.json # Rankings, win rates, win matrix
├── evaluation_report.md # Detailed Markdown report (if enabled)
├── win_rate_chart.png # Win-rate bar chart (if enabled)
├── win_rate_matrix.png # Matrix heatmap (if matrix_enabled)
├── queries.json # Generated test queries
├── responses.json # All model responses
├── rubrics.json # Generated evaluation rubrics
├── comparison_details.json # Pairwise comparison details
└── checkpoint.json # Pipeline checkpoint
>
---name: 01-auto-arenadescription: >---# skills/arena-eval/01-auto-arena/SKILL.md常見問題
從發布者下載 skills/arena-eval/01-auto-arena/SKILL.md,或複製路徑後開啟儲存庫。PlugMCP 只收錄這項技能,不代管檔案。
發布者將它存放在 https://github.com/agentscope-ai/OpenJudge。
Claude、Cursor、Codex 和 Grok Build 可以讀取你從發布者儲存庫加入的 SKILL.md。在聊天中傳送技能的 GitHub 資料夾連結,Grok Bot 就會安裝它。