- name
SKILL.md
File skill tetap berada di repositori penerbit.
Tempel perintah yang disalin di terminal. Perintah itu memasang SKILL.md ini ke agen tersebut.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a cursor -yTempel perintah yang disalin di terminal. Perintah itu memasang SKILL.md ini ke agen tersebut.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a claude-code -yTempel perintah yang disalin di terminal. Perintah itu memasang SKILL.md ini ke agen tersebut.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a codex -yTempel perintah yang disalin di terminal. Perintah itu memasang SKILL.md ini ke agen tersebut.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a grok -yKirim pesan yang disalin di chat Grok Bot. Bot mengambil SKILL.md ini dari GitHub dan menambahkannya ke skill yang dipakai bersama oleh semua Bot Anda.
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/arena-eval/01-auto-arenaDari penerbit
End-to-end automated model comparison using the OpenJudge AutoArenaPipeline:
# Install OpenJudge
pip install py-openjudge
# Extra dependency for auto_arena (chart generation)
pip install matplotlib
# Run evaluation
python -m cookbooks.auto_arena --config config.yaml --save
# Use pre-generated queries
python -m cookbooks.auto_arena --config config.yaml \
--queries_file queries.json --save
# Start fresh, ignore checkpoint
python -m cookbooks.auto_arena --config config.yaml --fresh --save
# Re-run only pairwise evaluation with new judge model
# (keeps queries, responses, and rubrics)
python -m cookbooks.auto_arena --config config.yaml --rerun-judge --save
import asyncio
from cookbooks.auto_arena.auto_arena_pipeline import AutoArenaPipeline
async def main():
pipeline = AutoArenaPipeline.from_config("config.yaml")
result = await pipeline.evaluate()
print(f"Best model: {result.best_pipeline}")
for rank, (model, win_rate) in enumerate(result.rankings, 1):
print(f"{rank}. {model}: {win_rate:.1%}")
asyncio.run(main())
import asyncio
from cookbooks.auto_arena.auto_arena_pipeline import AutoArenaPipeline
from cookbooks.auto_arena.schema import OpenAIEndpoint
async def main():
pipeline = AutoArenaPipeline(
task_description="Customer service chatbot for e-commerce",
target_endpoints={
"gpt4": OpenAIEndpoint(
base_url="https://api.openai.com/v1",
api_key="sk-...",
model="gpt-4",
),
"qwen": OpenAIEndpoint(
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
api_key="sk-...",
model="qwen-max",
),
},
judge_endpoint=OpenAIEndpoint(
base_url="https://api.openai.com/v1",
api_key="sk-...",
model="gpt-4",
),
num_queries=20,
)
result = await pipeline.evaluate()
print(f"Best: {result.best_pipeline}")
asyncio.run(main())
task:
description: "Academic GPT assistant for research and writing tasks"
target_endpoints:
model_v1:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-4"
model_v2:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-3.5-turbo"
judge_endpoint:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-4"
Same fields as target_endpoints.<name>. Use a strong model (e.g. gpt-4, qwen-max) with low temperature (~0.1) for consistent judgments.
Win rate: percentage of pairwise comparisons a model wins. Each pair is evaluated in both orders (original + swapped) to eliminate position bias.
Rankings example:
1. gpt4_baseline [################----] 80.0%
2. qwen_candidate [############--------] 60.0%
3. llama_finetuned [##########----------] 50.0%
Win matrix: win_matrix[A][B] = how often model A beats model B across all queries.
The pipeline saves progress after each step. Interrupted runs resume automatically:
--fresh — ignore checkpoint, start from scratch--rerun-judge — re-run only the pairwise evaluation step (useful when switching judge models); keeps queries, responses, and rubrics intactevaluation_results/
├── evaluation_results.json # Rankings, win rates, win matrix
├── evaluation_report.md # Detailed Markdown report (if enabled)
├── win_rate_chart.png # Win-rate bar chart (if enabled)
├── win_rate_matrix.png # Matrix heatmap (if matrix_enabled)
├── queries.json # Generated test queries
├── responses.json # All model responses
├── rubrics.json # Generated evaluation rubrics
├── comparison_details.json # Pairwise comparison details
└── checkpoint.json # Pipeline checkpoint
>
---name: 01-auto-arenadescription: >---# skills/arena-eval/01-auto-arena/SKILL.mdFAQ
Unduh skills/arena-eval/01-auto-arena/SKILL.md dari penerbit, atau salin path-nya dan buka repositori. PlugMCP hanya mendaftar skill ini dan tidak meng-host filenya.
Penerbit menyimpannya di https://github.com/agentscope-ai/OpenJudge.
Claude, Cursor, Codex, dan Grok Build bisa membaca SKILL.md yang Anda tambahkan dari repositori penerbit. Grok Bot memasangnya saat Anda mengirim tautan folder GitHub skill tersebut di chat.