- name
SKILL.md
El archivo de la skill permanece en el repositorio del publicador.
Pega el comando copiado en una terminal. Instala este SKILL.md en ese agente.
npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -a cursor -yPega el comando copiado en una terminal. Instala este SKILL.md en ese agente.
npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -a claude-code -yPega el comando copiado en una terminal. Instala este SKILL.md en ese agente.
npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -a codex -yPega el comando copiado en una terminal. Instala este SKILL.md en ese agente.
npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -a grok -yEnvía el mensaje copiado en un chat de Grok Bot. El Bot obtiene este SKILL.md de GitHub y lo añade a las skills que comparten todos tus Bots.
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/openjudge-core/01-graders-and-pipelineDel publicador
Build evaluation pipelines for LLM applications using the openjudge library.
Read the relevant sub-document before writing any code.
pip install py-openjudge
Dataset (List[dict])
│
▼
GradingRunner ← orchestrates everything
│
├─► Grader A ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank
├─► Grader B ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank
└─► Grader C ...
│
├─► Aggregator (optional) ← combine multiple grader scores into one
│
└─► RunnerResult ← {grader_name: [GraderScore, ...]}
│
▼
Analyzer ← statistics, win rates, validation metrics
Evaluate responses for correctness using a built-in grader:
import asyncio
from openjudge.models.openai_chat_model import OpenAIChatModel
from openjudge.graders.common.correctness import CorrectnessGrader
from openjudge.runner.grading_runner import GradingRunner
# 1. Configure the judge model (OpenAI-compatible endpoint)
model = OpenAIChatModel(
model="qwen-plus",
api_key="sk-xxx",
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
)
# 2. Instantiate a grader
grader = CorrectnessGrader(model=model)
# 3. Prepare dataset
dataset = [
{
"query": "What is the capital of France?",
"response": "Paris is the capital of France.",
"reference_response": "Paris.",
},
{
"query": "What is 2 + 2?",
"response": "The answer is five.",
"reference_response": "4.",
},
]
# 4. Run evaluation
async def main():
runner = GradingRunner(
grader_configs={"correctness": grader},
max_concurrency=8,
)
results = await runner.arun(dataset)
for i, result in enumerate(results["correctness"]):
print(f"[{i}] score={result.score} reason={result.reason}")
asyncio.run(main())
Expected output:
[0] score=5 reason=The response accurately states Paris as capital...
[1] score=1 reason=The response gives the wrong answer (five vs 4)...
from openjudge.graders.schema import GraderScore, GraderRank, GraderError
for grader_name, grader_results in results.items():
for i, result in enumerate(grader_results):
if isinstance(result, GraderScore):
print(f"{grader_name}[{i}]: score={result.score}")
elif isinstance(result, GraderRank):
print(f"{grader_name}[{i}]: rank={result.rank}")
elif isinstance(result, GraderError):
print(f"{grader_name}[{i}]: ERROR — {result.error}")
All LLM-based graders accept either a BaseChatModel instance or a dict config:
# Option A: instance
from openjudge.models.openai_chat_model import OpenAIChatModel
model = OpenAIChatModel(model="gpt-4o", api_key="sk-...")
# Option B: dict (auto-creates OpenAIChatModel)
model_cfg = {"model": "gpt-4o", "api_key": "sk-..."}
grader = CorrectnessGrader(model=model_cfg)
# OpenAI-compatible endpoints (DashScope / local / etc.)
model = OpenAIChatModel(
model="qwen-plus",
api_key="sk-xxx",
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
)
>
---name: 01-graders-and-pipelinedescription: >---# skills/openjudge-core/01-graders-and-pipeline/SKILL.mdPreguntas frecuentes
Descarga skills/openjudge-core/01-graders-and-pipeline/SKILL.md desde el publicador, o copia la ruta y abre el repositorio. PlugMCP lista la skill y no aloja el archivo.
El publicador lo mantiene en https://github.com/agentscope-ai/OpenJudge.
Claude, Cursor, Codex y Grok Build pueden leer un SKILL.md que añadas desde el repositorio del publicador. Grok Bot lo instala cuando envías en un chat el enlace a la carpeta de la skill en GitHub.