- name
SKILL.md
스킬 파일은 게시자의 저장소에 그대로 있습니다.
복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a cursor -y복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a claude-code -y복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a codex -y복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a grok -y복사한 메시지를 Grok Bot 채팅에 보냅니다. Bot이 GitHub에서 이 SKILL.md를 가져와 모든 Bot이 공유하는 스킬에 추가합니다.
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/arena-eval/01-auto-arena게시자 제공
End-to-end automated model comparison using the OpenJudge AutoArenaPipeline:
# Install OpenJudge
pip install py-openjudge
# Extra dependency for auto_arena (chart generation)
pip install matplotlib
# Run evaluation
python -m cookbooks.auto_arena --config config.yaml --save
# Use pre-generated queries
python -m cookbooks.auto_arena --config config.yaml \
--queries_file queries.json --save
# Start fresh, ignore checkpoint
python -m cookbooks.auto_arena --config config.yaml --fresh --save
# Re-run only pairwise evaluation with new judge model
# (keeps queries, responses, and rubrics)
python -m cookbooks.auto_arena --config config.yaml --rerun-judge --save
import asyncio
from cookbooks.auto_arena.auto_arena_pipeline import AutoArenaPipeline
async def main():
pipeline = AutoArenaPipeline.from_config("config.yaml")
result = await pipeline.evaluate()
print(f"Best model: {result.best_pipeline}")
for rank, (model, win_rate) in enumerate(result.rankings, 1):
print(f"{rank}. {model}: {win_rate:.1%}")
asyncio.run(main())
import asyncio
from cookbooks.auto_arena.auto_arena_pipeline import AutoArenaPipeline
from cookbooks.auto_arena.schema import OpenAIEndpoint
async def main():
pipeline = AutoArenaPipeline(
task_description="Customer service chatbot for e-commerce",
target_endpoints={
"gpt4": OpenAIEndpoint(
base_url="https://api.openai.com/v1",
api_key="sk-...",
model="gpt-4",
),
"qwen": OpenAIEndpoint(
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
api_key="sk-...",
model="qwen-max",
),
},
judge_endpoint=OpenAIEndpoint(
base_url="https://api.openai.com/v1",
api_key="sk-...",
model="gpt-4",
),
num_queries=20,
)
result = await pipeline.evaluate()
print(f"Best: {result.best_pipeline}")
asyncio.run(main())
task:
description: "Academic GPT assistant for research and writing tasks"
target_endpoints:
model_v1:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-4"
model_v2:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-3.5-turbo"
judge_endpoint:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-4"
Same fields as target_endpoints.<name>. Use a strong model (e.g. gpt-4, qwen-max) with low temperature (~0.1) for consistent judgments.
Win rate: percentage of pairwise comparisons a model wins. Each pair is evaluated in both orders (original + swapped) to eliminate position bias.
Rankings example:
1. gpt4_baseline [################----] 80.0%
2. qwen_candidate [############--------] 60.0%
3. llama_finetuned [##########----------] 50.0%
Win matrix: win_matrix[A][B] = how often model A beats model B across all queries.
The pipeline saves progress after each step. Interrupted runs resume automatically:
--fresh — ignore checkpoint, start from scratch--rerun-judge — re-run only the pairwise evaluation step (useful when switching judge models); keeps queries, responses, and rubrics intactevaluation_results/
├── evaluation_results.json # Rankings, win rates, win matrix
├── evaluation_report.md # Detailed Markdown report (if enabled)
├── win_rate_chart.png # Win-rate bar chart (if enabled)
├── win_rate_matrix.png # Matrix heatmap (if matrix_enabled)
├── queries.json # Generated test queries
├── responses.json # All model responses
├── rubrics.json # Generated evaluation rubrics
├── comparison_details.json # Pairwise comparison details
└── checkpoint.json # Pipeline checkpoint
>
---name: 01-auto-arenadescription: >---# skills/arena-eval/01-auto-arena/SKILL.mdFAQ
게시자에게서 skills/arena-eval/01-auto-arena/SKILL.md를 다운로드하거나 경로를 복사해 저장소를 엽니다. PlugMCP는 스킬을 목록에 올릴 뿐 파일을 호스팅하지 않습니다.
게시자가 https://github.com/agentscope-ai/OpenJudge에서 관리합니다.
Claude, Cursor, Codex, Grok Build는 게시자의 저장소에서 추가한 SKILL.md를 읽을 수 있습니다. Grok Bot은 채팅에서 스킬의 GitHub 폴더 링크를 보내면 설치합니다.