- name
SKILL.md
스킬 파일은 게시자의 저장소에 그대로 있습니다.
복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -a cursor -y복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -a claude-code -y복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -a codex -y복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -a grok -y복사한 메시지를 Grok Bot 채팅에 보냅니다. Bot이 GitHub에서 이 SKILL.md를 가져와 모든 Bot이 공유하는 스킬에 추가합니다.
Install this skill: https://github.com/agentscope-ai/OpenJudge/tree/HEAD/skills/openjudge-core/01-graders-and-pipeline게시자 제공
Build evaluation pipelines for LLM applications using the openjudge library.
Read the relevant sub-document before writing any code.
pip install py-openjudge
Dataset (List[dict])
│
▼
GradingRunner ← orchestrates everything
│
├─► Grader A ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank
├─► Grader B ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank
└─► Grader C ...
│
├─► Aggregator (optional) ← combine multiple grader scores into one
│
└─► RunnerResult ← {grader_name: [GraderScore, ...]}
│
▼
Analyzer ← statistics, win rates, validation metrics
Evaluate responses for correctness using a built-in grader:
import asyncio
from openjudge.models.openai_chat_model import OpenAIChatModel
from openjudge.graders.common.correctness import CorrectnessGrader
from openjudge.runner.grading_runner import GradingRunner
# 1. Configure the judge model (OpenAI-compatible endpoint)
model = OpenAIChatModel(
model="qwen-plus",
api_key="sk-xxx",
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
)
# 2. Instantiate a grader
grader = CorrectnessGrader(model=model)
# 3. Prepare dataset
dataset = [
{
"query": "What is the capital of France?",
"response": "Paris is the capital of France.",
"reference_response": "Paris.",
},
{
"query": "What is 2 + 2?",
"response": "The answer is five.",
"reference_response": "4.",
},
]
# 4. Run evaluation
async def main():
runner = GradingRunner(
grader_configs={"correctness": grader},
max_concurrency=8,
)
results = await runner.arun(dataset)
for i, result in enumerate(results["correctness"]):
print(f"[{i}] score={result.score} reason={result.reason}")
asyncio.run(main())
Expected output:
[0] score=5 reason=The response accurately states Paris as capital...
[1] score=1 reason=The response gives the wrong answer (five vs 4)...
from openjudge.graders.schema import GraderScore, GraderRank, GraderError
for grader_name, grader_results in results.items():
for i, result in enumerate(grader_results):
if isinstance(result, GraderScore):
print(f"{grader_name}[{i}]: score={result.score}")
elif isinstance(result, GraderRank):
print(f"{grader_name}[{i}]: rank={result.rank}")
elif isinstance(result, GraderError):
print(f"{grader_name}[{i}]: ERROR — {result.error}")
All LLM-based graders accept either a BaseChatModel instance or a dict config:
# Option A: instance
from openjudge.models.openai_chat_model import OpenAIChatModel
model = OpenAIChatModel(model="gpt-4o", api_key="sk-...")
# Option B: dict (auto-creates OpenAIChatModel)
model_cfg = {"model": "gpt-4o", "api_key": "sk-..."}
grader = CorrectnessGrader(model=model_cfg)
# OpenAI-compatible endpoints (DashScope / local / etc.)
model = OpenAIChatModel(
model="qwen-plus",
api_key="sk-xxx",
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
)
>
---name: 01-graders-and-pipelinedescription: >---# skills/openjudge-core/01-graders-and-pipeline/SKILL.mdFAQ
게시자에게서 skills/openjudge-core/01-graders-and-pipeline/SKILL.md를 다운로드하거나 경로를 복사해 저장소를 엽니다. PlugMCP는 스킬을 목록에 올릴 뿐 파일을 호스팅하지 않습니다.
게시자가 https://github.com/agentscope-ai/OpenJudge에서 관리합니다.
Claude, Cursor, Codex, Grok Build는 게시자의 저장소에서 추가한 SKILL.md를 읽을 수 있습니다. Grok Bot은 채팅에서 스킬의 GitHub 폴더 링크를 보내면 설치합니다.