- name
SKILL.md
Le fichier du skill reste dans le dépôt de l'éditeur.
Collez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a cursor -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a claude-code -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a codex -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -a grok -yEnvoyez le message copié dans un chat Grok Bot. Le Bot récupère ce SKILL.md sur GitHub et l'ajoute aux skills que partagent tous vos Bots.
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/arena-eval/01-auto-arenaDe l'éditeur
End-to-end automated model comparison using the OpenJudge AutoArenaPipeline:
# Install OpenJudge
pip install py-openjudge
# Extra dependency for auto_arena (chart generation)
pip install matplotlib
# Run evaluation
python -m cookbooks.auto_arena --config config.yaml --save
# Use pre-generated queries
python -m cookbooks.auto_arena --config config.yaml \
--queries_file queries.json --save
# Start fresh, ignore checkpoint
python -m cookbooks.auto_arena --config config.yaml --fresh --save
# Re-run only pairwise evaluation with new judge model
# (keeps queries, responses, and rubrics)
python -m cookbooks.auto_arena --config config.yaml --rerun-judge --save
import asyncio
from cookbooks.auto_arena.auto_arena_pipeline import AutoArenaPipeline
async def main():
pipeline = AutoArenaPipeline.from_config("config.yaml")
result = await pipeline.evaluate()
print(f"Best model: {result.best_pipeline}")
for rank, (model, win_rate) in enumerate(result.rankings, 1):
print(f"{rank}. {model}: {win_rate:.1%}")
asyncio.run(main())
import asyncio
from cookbooks.auto_arena.auto_arena_pipeline import AutoArenaPipeline
from cookbooks.auto_arena.schema import OpenAIEndpoint
async def main():
pipeline = AutoArenaPipeline(
task_description="Customer service chatbot for e-commerce",
target_endpoints={
"gpt4": OpenAIEndpoint(
base_url="https://api.openai.com/v1",
api_key="sk-...",
model="gpt-4",
),
"qwen": OpenAIEndpoint(
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
api_key="sk-...",
model="qwen-max",
),
},
judge_endpoint=OpenAIEndpoint(
base_url="https://api.openai.com/v1",
api_key="sk-...",
model="gpt-4",
),
num_queries=20,
)
result = await pipeline.evaluate()
print(f"Best: {result.best_pipeline}")
asyncio.run(main())
task:
description: "Academic GPT assistant for research and writing tasks"
target_endpoints:
model_v1:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-4"
model_v2:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-3.5-turbo"
judge_endpoint:
base_url: "https://api.openai.com/v1"
api_key: "${OPENAI_API_KEY}"
model: "gpt-4"
Same fields as target_endpoints.<name>. Use a strong model (e.g. gpt-4, qwen-max) with low temperature (~0.1) for consistent judgments.
Win rate: percentage of pairwise comparisons a model wins. Each pair is evaluated in both orders (original + swapped) to eliminate position bias.
Rankings example:
1. gpt4_baseline [################----] 80.0%
2. qwen_candidate [############--------] 60.0%
3. llama_finetuned [##########----------] 50.0%
Win matrix: win_matrix[A][B] = how often model A beats model B across all queries.
The pipeline saves progress after each step. Interrupted runs resume automatically:
--fresh — ignore checkpoint, start from scratch--rerun-judge — re-run only the pairwise evaluation step (useful when switching judge models); keeps queries, responses, and rubrics intactevaluation_results/
├── evaluation_results.json # Rankings, win rates, win matrix
├── evaluation_report.md # Detailed Markdown report (if enabled)
├── win_rate_chart.png # Win-rate bar chart (if enabled)
├── win_rate_matrix.png # Matrix heatmap (if matrix_enabled)
├── queries.json # Generated test queries
├── responses.json # All model responses
├── rubrics.json # Generated evaluation rubrics
├── comparison_details.json # Pairwise comparison details
└── checkpoint.json # Pipeline checkpoint
>
---name: 01-auto-arenadescription: >---# skills/arena-eval/01-auto-arena/SKILL.mdFAQ
Téléchargez skills/arena-eval/01-auto-arena/SKILL.md auprès de l'éditeur, ou copiez le chemin et ouvrez le dépôt. PlugMCP répertorie le skill et n'héberge pas le fichier.
L'éditeur le maintient sur https://github.com/agentscope-ai/OpenJudge.
Claude, Cursor, Codex et Grok Build peuvent lire un SKILL.md que vous ajoutez depuis le dépôt de l'éditeur. Grok Bot l'installe lorsque vous envoyez dans un chat le lien du dossier GitHub du skill.