- name
SKILL.md
스킬 파일은 게시자의 저장소에 그대로 있습니다.
복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -a cursor -y복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -a claude-code -y복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -a codex -y복사한 명령을 터미널에 붙여넣습니다. 이 SKILL.md가 해당 에이전트에 설치됩니다.
npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -a grok -y복사한 메시지를 Grok Bot 채팅에 보냅니다. Bot이 GitHub에서 이 SKILL.md를 가져와 모든 Bot이 공유하는 스킬에 추가합니다.
Install this skill: https://github.com/NVIDIA/SkillEvaluator/tree/HEAD/src/skillevaluator/tier3/reference_skills/create-custom-grader게시자 제공
Convert team-owned benchmark definitions into runnable SkillEvaluator custom graders and, when needed, native Harbor tasks.
Help an agent author valid SkillEvaluator BYOG/BYOT files from a user's benchmark instead of leaving the user with empty grader templates.
Use this skill when the user wants to:
evals/grader.py or evals/grader.shtask.yaml, task.json, pytest checks, or shell
verifiers into BYOG or BYOTDo not use this skill for ordinary evals/evals.json authoring when no custom
grading logic is needed. Use the normal dataset authoring workflow for that.
evals/, benchmark prompts, fixtures, and any verifier code.default_plus_custom when custom metrics should complement default evaluator scoring.custom_only only when the user wants the custom grader to own pass/fail semantics.evals/grader.py or evals/grader.sh, then validate the Harbor contract.skillevaluator init-custom-grader <skill-dir> --language python --mode default_plus_custom
skillevaluator tier3 validate <skill-dir>
SKILL.md.skillevaluator.Choose one path before writing files:
Default to default_plus_custom unless the user explicitly wants the custom
grader to replace the default evaluator metrics.
Resolve the target skill and benchmark source.
Read the target SKILL.md, existing evals/, benchmark prompts, fixtures,
rubric, reference solution, tags, and any expected trigger/non-trigger
metadata.
Map benchmark fields into evaluator inputs.
Use benchmark prompts or prompt variants as question entries. Use the
target skill as expected_skill. Put each case's required starter files
under evals/files/<case-id>/, and declare
files: ["evals/files/<case-id>"] on every corresponding eval entry. Do not
omit files in a multi-case dataset, because omission intentionally stages
the entire shared directory for legacy compatibility. Preserve
benchmark-specific rubric text in the entry only when the grader needs to
read it.
Scaffold the evaluator contract. For generated tasks:
skillevaluator init-custom-grader <skill-dir> --language python --mode default_plus_custom
For shell checks:
skillevaluator init-custom-grader <skill-dir> --language shell --mode default_plus_custom
For native Harbor tasks:
skillevaluator init-harbor-task <skill-dir> --case-id <case-id> --with-config
Replace scaffold placeholders. The custom grader is real executable logic, not metadata. It must read available evidence, compute numeric scores, and write the evaluator reward contract.
Validate before running.
skillevaluator validate <skill-dir> --harbor-contract
Fix missing files, invalid Python, missing reward output, and native Harbor ID mismatches before evaluation.
Run the deepest practical proof. Prefer a real with-skill/baseline run. If services, credentials, GPU, or cost block full E2E, state exactly what was validated and what was not.
Python and shell graders run inside the Harbor verifier context. They may read:
/logs/agent/trajectory.json for agent actions and final answer evidence/tests/entry.json for the eval case metadata/workspace/input/ for the entry's declared committed fixtures from
evals/files//solution/ or other task outputs only when the task environment produces
themThey must write:
/logs/verifier/reward.json/logs/verifier/reward.txt with a numeric score from 0.0 to 1.0Use this reward shape:
{
"overall": 0.92,
"custom_metrics": {
"domain_repair": 1.0,
"domain_verification": 0.8
},
"details": {
"domain_repair": {
"score": 1.0,
"reason": "The solution repaired the required files."
}
}
}
In default_plus_custom, default evaluator scoring keeps its overall
authoritative and adds the grader's custom_metrics into reports. In
custom_only, the grader's overall is the pass/fail reward.
Never emit custom metric names that collide with reserved evaluator fields:
security, skill_execution, skill_efficiency, accuracy,
goal_accuracy, behavior_check, overall, details, metrics,
metric_set, or entry_id.
details.0.0 through 1.0.expected_skill,
expected_behavior, negative cases, or custom metrics that inspect
trajectory evidence.For a benchmark task with task.yaml, code/, prompt variants, coverage, and a
rubric:
code/ into evals/files/<case-id>/.evals/evals.json entries from the prompt variants, and
set files: ["evals/files/<case-id>"] on each corresponding entry.expected_skill to the benchmark's target skill.evals/grader.py to inspect the agent trajectory and changed
workspace files.rapids_diagnosis, rapids_requirements_repair,
rapids_repair_safety, and rapids_verification.init-custom-grader creates scaffolding only; the agent must replace the
placeholder scoring logic.When finished, report:
Use when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.
---name: create-custom-graderdescription: Use when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into…---# src/skillevaluator/tier3/reference_skills/create-custom-grader/SKILL.mdFAQ
게시자에게서 src/skillevaluator/tier3/reference_skills/create-custom-grader/SKILL.md를 다운로드하거나 경로를 복사해 저장소를 엽니다. PlugMCP는 스킬을 목록에 올릴 뿐 파일을 호스팅하지 않습니다.
게시자가 https://github.com/NVIDIA/SkillEvaluator에서 관리합니다.
Claude, Cursor, Codex, Grok Build는 게시자의 저장소에서 추가한 SKILL.md를 읽을 수 있습니다. Grok Bot은 채팅에서 스킬의 GitHub 폴더 링크를 보내면 설치합니다.