- name
SKILL.md
Le fichier du skill reste dans le dépôt de l'éditeur.
Collez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a cursor -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a claude-code -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a codex -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a grok -yEnvoyez le message copié dans un chat Grok Bot. Le Bot récupère ce SKILL.md sur GitHub et l'ajoute aux skills que partagent tous vos Bots.
Install this skill: https://github.com/Aperivue/medsci-skills/tree/HEAD/skills/design-ai-benchmarkingDe l'éditeur
## AI-Benchmark Design Review
Evaluation question: ...
Arms / systems compared: ...
Reference (human-expert panel): ...
Unit of rating: (item / case / output)
### Rubric (decoupled dimensions)
- dimension -> construct -> anchors (1..k)
### Calibration probes (blinded, randomized)
- positive-control / known-bad / instability / mechanism-contradiction
### Reviewer panel
- n reviewers, metadata captured, per-reviewer randomized order
### Reliability plan
- IRR target on the anchor set of real items (ICC form + 95% CI) + control-item hit rate (a competence check, reported separately)
### Judge strategy
- human-as-judge / LLM-as-judge / both + adjudication rule
### Validity risks
1. ...
### Minimal fixes
- ...
### Decision
- Ready to collect / Needs rubric revision / Needs arm or judge redesign
Reviewer ratings, reference labels, probe outcomes and agreement statistics come only from collected
rating records. Never invent them: a reported ICC, kappa or score with no underlying rating record is
the failure this skill exists to prevent. Cite a reference only with a /search-lit-confirmed DOI or
PMID; mark any other [UNVERIFIED - NEEDS MANUAL CHECK]. Flag an unconfirmed clinical definition,
diagnostic criterion or guideline recommendation [VERIFY] and ask the user.
Pin down, in writing:
Gate: Present the reconstructed evaluation question, arms, and reference to the user and confirm before designing the rubric. A wrong reconstruction misdirects the entire benchmark.
${CLAUDE_SKILL_DIR}/references/elicitation_rubric_template.md (dimensions, anchors,
probe flavors).Plant a few deliberate control items, blinded and randomized across raters (record who received which
via a probe_arm flag), to anchor the scale, measure rater drift/fatigue, and audit the rubric and
pipeline. Four flavors:
Probes are planted or adjudicated, never fabricated to fit a hypothesis.
Gate: Present the panel composition, stratification, and randomization plan for user review before recruitment is finalized.
/analyze-stats).When the item pool exceeds what one reader can rate in a session, do not make every reader rate
every item (that caps the pool at the per-reader limit). Use anchor-and-rotate (an anchor set
plus rotating incomplete blocks — not a balanced incomplete block design, since reader pairs are
not co-rated equally often): all readers rate a shared anchor set of real items (which carries
the inter-rater reliability; the planted controls carry only the competence check) plus a
rotating unique block each. The binding
constraint is usually the number of available expert readers, so solve the reverse problem (largest
pool for R readers) to size the must-rate set. Pre-specify anchor membership, raters-per-item, and the
rotation seed before rating. Read ${CLAUDE_SKILL_DIR}/references/anchor_rotate_reader_allocation.md
for the formulas, trade-offs, and a stdlib implementation.
Write the machine-readable rating record as a JSON schema, starting from
${CLAUDE_SKILL_DIR}/references/benchmark_export_schema.json: per-item ratings on every rubric
dimension, free-text justifications, follow-up flags, the probe_arm flag, reviewer id and metadata,
item order, and timing.
Gate: Present the final rubric, probe set, panel plan, judge strategy, and export schema together; collect explicit user approval before any rating begins, because changes after collection starts compromise the comparison.
/analyze-stats for ICC (stated form) / Krippendorff's α / kappa (two raters) / DeLong, agreement sample size, and effect-size real-world
translation of the benchmark results/check-reporting for STARD-AI, CLAIM, or TRIPOD+AI item-level reporting once the design is locked/design-study when the broader study around the benchmark (cohort logic, analysis unit,
comparator) also needs review/peer-review or /self-review only after ratings exist and a manuscript is being assessedUse when designing a study that benchmarks AI systems against a human-expert panel, before data collection. Plans the arms, decoupled rubrics with anchors, planted calibration probes, reviewer panel,
---name: design-ai-benchmarkingdescription: Use when designing a study that benchmarks AI systems against a human-expert panel, before data collection…---# skills/design-ai-benchmarking/SKILL.mdVue d'ensemble
Use when designing a study that benchmarks AI systems against a human-expert panel, before data collection. Plans the arms, decoupled rubrics with anchors, planted calibration probes, reviewer panel, inter-rater reliability targets and LLM-as-judge versus human adjudication.
FAQ
Téléchargez skills/design-ai-benchmarking/SKILL.md auprès de l'éditeur, ou copiez le chemin et ouvrez le dépôt. PlugMCP répertorie le skill et n'héberge pas le fichier.
L'éditeur le maintient sur https://github.com/Aperivue/medsci-skills.
Claude, Cursor, Codex et Grok Build peuvent lire un SKILL.md que vous ajoutez depuis le dépôt de l'éditeur. Grok Bot l'installe lorsque vous envoyez dans un chat le lien du dossier GitHub du skill.