- name
SKILL.md
スキルファイルは公開元のリポジトリに置かれたままです。
コピーしたコマンドをターミナルに貼り付けてください。この SKILL.md がそのエージェントにインストールされます。
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a cursor -yコピーしたコマンドをターミナルに貼り付けてください。この SKILL.md がそのエージェントにインストールされます。
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a claude-code -yコピーしたコマンドをターミナルに貼り付けてください。この SKILL.md がそのエージェントにインストールされます。
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a codex -yコピーしたコマンドをターミナルに貼り付けてください。この SKILL.md がそのエージェントにインストールされます。
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a grok -yコピーしたメッセージを Grok Bot のチャットで送信してください。Bot がこの SKILL.md を GitHub から取得し、すべての Bot が共有するスキルに追加します。
Install this skill: https://github.com/Aperivue/medsci-skills/tree/HEAD/skills/design-ai-benchmarking公開元より
## AI-Benchmark Design Review
Evaluation question: ...
Arms / systems compared: ...
Reference (human-expert panel): ...
Unit of rating: (item / case / output)
### Rubric (decoupled dimensions)
- dimension -> construct -> anchors (1..k)
### Calibration probes (blinded, randomized)
- positive-control / known-bad / instability / mechanism-contradiction
### Reviewer panel
- n reviewers, metadata captured, per-reviewer randomized order
### Reliability plan
- IRR target on the anchor set of real items (ICC form + 95% CI) + control-item hit rate (a competence check, reported separately)
### Judge strategy
- human-as-judge / LLM-as-judge / both + adjudication rule
### Validity risks
1. ...
### Minimal fixes
- ...
### Decision
- Ready to collect / Needs rubric revision / Needs arm or judge redesign
Reviewer ratings, reference labels, probe outcomes and agreement statistics come only from collected
rating records. Never invent them: a reported ICC, kappa or score with no underlying rating record is
the failure this skill exists to prevent. Cite a reference only with a /search-lit-confirmed DOI or
PMID; mark any other [UNVERIFIED - NEEDS MANUAL CHECK]. Flag an unconfirmed clinical definition,
diagnostic criterion or guideline recommendation [VERIFY] and ask the user.
Pin down, in writing:
Gate: Present the reconstructed evaluation question, arms, and reference to the user and confirm before designing the rubric. A wrong reconstruction misdirects the entire benchmark.
${CLAUDE_SKILL_DIR}/references/elicitation_rubric_template.md (dimensions, anchors,
probe flavors).Plant a few deliberate control items, blinded and randomized across raters (record who received which
via a probe_arm flag), to anchor the scale, measure rater drift/fatigue, and audit the rubric and
pipeline. Four flavors:
Probes are planted or adjudicated, never fabricated to fit a hypothesis.
Gate: Present the panel composition, stratification, and randomization plan for user review before recruitment is finalized.
/analyze-stats).When the item pool exceeds what one reader can rate in a session, do not make every reader rate
every item (that caps the pool at the per-reader limit). Use anchor-and-rotate (an anchor set
plus rotating incomplete blocks — not a balanced incomplete block design, since reader pairs are
not co-rated equally often): all readers rate a shared anchor set of real items (which carries
the inter-rater reliability; the planted controls carry only the competence check) plus a
rotating unique block each. The binding
constraint is usually the number of available expert readers, so solve the reverse problem (largest
pool for R readers) to size the must-rate set. Pre-specify anchor membership, raters-per-item, and the
rotation seed before rating. Read ${CLAUDE_SKILL_DIR}/references/anchor_rotate_reader_allocation.md
for the formulas, trade-offs, and a stdlib implementation.
Write the machine-readable rating record as a JSON schema, starting from
${CLAUDE_SKILL_DIR}/references/benchmark_export_schema.json: per-item ratings on every rubric
dimension, free-text justifications, follow-up flags, the probe_arm flag, reviewer id and metadata,
item order, and timing.
Gate: Present the final rubric, probe set, panel plan, judge strategy, and export schema together; collect explicit user approval before any rating begins, because changes after collection starts compromise the comparison.
/analyze-stats for ICC (stated form) / Krippendorff's α / kappa (two raters) / DeLong, agreement sample size, and effect-size real-world
translation of the benchmark results/check-reporting for STARD-AI, CLAIM, or TRIPOD+AI item-level reporting once the design is locked/design-study when the broader study around the benchmark (cohort logic, analysis unit,
comparator) also needs review/peer-review or /self-review only after ratings exist and a manuscript is being assessedUse when designing a study that benchmarks AI systems against a human-expert panel, before data collection. Plans the arms, decoupled rubrics with anchors, planted calibration probes, reviewer panel,
---name: design-ai-benchmarkingdescription: Use when designing a study that benchmarks AI systems against a human-expert panel, before data collection…---# skills/design-ai-benchmarking/SKILL.md概要
Use when designing a study that benchmarks AI systems against a human-expert panel, before data collection. Plans the arms, decoupled rubrics with anchors, planted calibration probes, reviewer panel, inter-rater reliability targets and LLM-as-judge versus human adjudication.
よくある質問
公開元から skills/design-ai-benchmarking/SKILL.md をダウンロードするか、パスをコピーしてリポジトリを開いてください。PlugMCP はスキルを掲載するだけで、ファイルはホストしていません。
公開元は https://github.com/Aperivue/medsci-skills で管理しています。
Claude、Cursor、Codex、Grok Build は、公開元のリポジトリから追加した SKILL.md を読み込めます。Grok Bot は、チャットでスキルの GitHub フォルダーのリンクを送るとインストールします。