- name
SKILL.md
Файл навыка остаётся в репозитории издателя.
Вставьте скопированную команду в терминал. Она установит этот SKILL.md в выбранного агента.
npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -a cursor -yВставьте скопированную команду в терминал. Она установит этот SKILL.md в выбранного агента.
npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -a claude-code -yВставьте скопированную команду в терминал. Она установит этот SKILL.md в выбранного агента.
npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -a codex -yВставьте скопированную команду в терминал. Она установит этот SKILL.md в выбранного агента.
npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -a grok -yОтправьте скопированное сообщение в чат Grok Bot. Бот загрузит этот SKILL.md с GitHub и добавит его к навыкам, общим для всех ваших ботов.
Install this skill: https://github.com/NVIDIA/SkillEvaluator/tree/HEAD/src/skillevaluator/tier3/reference_skills/create-custom-graderОт издателя
Convert team-owned benchmark definitions into runnable SkillEvaluator custom graders and, when needed, native Harbor tasks.
Help an agent author valid SkillEvaluator BYOG/BYOT files from a user's benchmark instead of leaving the user with empty grader templates.
Use this skill when the user wants to:
evals/grader.py or evals/grader.shtask.yaml, task.json, pytest checks, or shell
verifiers into BYOG or BYOTDo not use this skill for ordinary evals/evals.json authoring when no custom
grading logic is needed. Use the normal dataset authoring workflow for that.
evals/, benchmark prompts, fixtures, and any verifier code.default_plus_custom when custom metrics should complement default evaluator scoring.custom_only only when the user wants the custom grader to own pass/fail semantics.evals/grader.py or evals/grader.sh, then validate the Harbor contract.skillevaluator init-custom-grader <skill-dir> --language python --mode default_plus_custom
skillevaluator tier3 validate <skill-dir>
SKILL.md.skillevaluator.Choose one path before writing files:
Default to default_plus_custom unless the user explicitly wants the custom
grader to replace the default evaluator metrics.
Resolve the target skill and benchmark source.
Read the target SKILL.md, existing evals/, benchmark prompts, fixtures,
rubric, reference solution, tags, and any expected trigger/non-trigger
metadata.
Map benchmark fields into evaluator inputs.
Use benchmark prompts or prompt variants as question entries. Use the
target skill as expected_skill. Put each case's required starter files
under evals/files/<case-id>/, and declare
files: ["evals/files/<case-id>"] on every corresponding eval entry. Do not
omit files in a multi-case dataset, because omission intentionally stages
the entire shared directory for legacy compatibility. Preserve
benchmark-specific rubric text in the entry only when the grader needs to
read it.
Scaffold the evaluator contract. For generated tasks:
skillevaluator init-custom-grader <skill-dir> --language python --mode default_plus_custom
For shell checks:
skillevaluator init-custom-grader <skill-dir> --language shell --mode default_plus_custom
For native Harbor tasks:
skillevaluator init-harbor-task <skill-dir> --case-id <case-id> --with-config
Replace scaffold placeholders. The custom grader is real executable logic, not metadata. It must read available evidence, compute numeric scores, and write the evaluator reward contract.
Validate before running.
skillevaluator validate <skill-dir> --harbor-contract
Fix missing files, invalid Python, missing reward output, and native Harbor ID mismatches before evaluation.
Run the deepest practical proof. Prefer a real with-skill/baseline run. If services, credentials, GPU, or cost block full E2E, state exactly what was validated and what was not.
Python and shell graders run inside the Harbor verifier context. They may read:
/logs/agent/trajectory.json for agent actions and final answer evidence/tests/entry.json for the eval case metadata/workspace/input/ for the entry's declared committed fixtures from
evals/files//solution/ or other task outputs only when the task environment produces
themThey must write:
/logs/verifier/reward.json/logs/verifier/reward.txt with a numeric score from 0.0 to 1.0Use this reward shape:
{
"overall": 0.92,
"custom_metrics": {
"domain_repair": 1.0,
"domain_verification": 0.8
},
"details": {
"domain_repair": {
"score": 1.0,
"reason": "The solution repaired the required files."
}
}
}
In default_plus_custom, default evaluator scoring keeps its overall
authoritative and adds the grader's custom_metrics into reports. In
custom_only, the grader's overall is the pass/fail reward.
Never emit custom metric names that collide with reserved evaluator fields:
security, skill_execution, skill_efficiency, accuracy,
goal_accuracy, behavior_check, overall, details, metrics,
metric_set, or entry_id.
details.0.0 through 1.0.expected_skill,
expected_behavior, negative cases, or custom metrics that inspect
trajectory evidence.For a benchmark task with task.yaml, code/, prompt variants, coverage, and a
rubric:
code/ into evals/files/<case-id>/.evals/evals.json entries from the prompt variants, and
set files: ["evals/files/<case-id>"] on each corresponding entry.expected_skill to the benchmark's target skill.evals/grader.py to inspect the agent trajectory and changed
workspace files.rapids_diagnosis, rapids_requirements_repair,
rapids_repair_safety, and rapids_verification.init-custom-grader creates scaffolding only; the agent must replace the
placeholder scoring logic.When finished, report:
Use when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.
---name: create-custom-graderdescription: Use when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into…---# src/skillevaluator/tier3/reference_skills/create-custom-grader/SKILL.mdFAQ
Скачайте src/skillevaluator/tier3/reference_skills/create-custom-grader/SKILL.md у издателя или скопируйте путь и откройте репозиторий. PlugMCP только показывает навык в каталоге и не хранит файл.
Издатель хранит его в https://github.com/NVIDIA/SkillEvaluator.
Claude, Cursor, Codex и Grok Build могут читать SKILL.md, который вы добавляете из репозитория издателя. Grok Bot устанавливает его, когда вы отправляете в чат ссылку на папку навыка на GitHub.