- name
SKILL.md
स्किल फ़ाइल पब्लिशर की रिपॉज़िटरी में ही रहती है।
कॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a cursor -yकॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a claude-code -yकॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a codex -yकॉपी की गई कमांड टर्मिनल में पेस्ट करें। यह इस SKILL.md को उस एजेंट में इंस्टॉल करती है।
npx skills add Aperivue/medsci-skills --skill design-ai-benchmarking -a grok -yकॉपी किया गया संदेश Grok Bot चैट में भेजें। Bot यह SKILL.md GitHub से लाता है और उसे उन स्किल में जोड़ देता है जो आपके सभी Bot साझा करते हैं।
Install this skill: https://github.com/Aperivue/medsci-skills/tree/HEAD/skills/design-ai-benchmarkingपब्लिशर से
## AI-Benchmark Design Review
Evaluation question: ...
Arms / systems compared: ...
Reference (human-expert panel): ...
Unit of rating: (item / case / output)
### Rubric (decoupled dimensions)
- dimension -> construct -> anchors (1..k)
### Calibration probes (blinded, randomized)
- positive-control / known-bad / instability / mechanism-contradiction
### Reviewer panel
- n reviewers, metadata captured, per-reviewer randomized order
### Reliability plan
- IRR target on the anchor set of real items (ICC form + 95% CI) + control-item hit rate (a competence check, reported separately)
### Judge strategy
- human-as-judge / LLM-as-judge / both + adjudication rule
### Validity risks
1. ...
### Minimal fixes
- ...
### Decision
- Ready to collect / Needs rubric revision / Needs arm or judge redesign
Reviewer ratings, reference labels, probe outcomes and agreement statistics come only from collected
rating records. Never invent them: a reported ICC, kappa or score with no underlying rating record is
the failure this skill exists to prevent. Cite a reference only with a /search-lit-confirmed DOI or
PMID; mark any other [UNVERIFIED - NEEDS MANUAL CHECK]. Flag an unconfirmed clinical definition,
diagnostic criterion or guideline recommendation [VERIFY] and ask the user.
Pin down, in writing:
Gate: Present the reconstructed evaluation question, arms, and reference to the user and confirm before designing the rubric. A wrong reconstruction misdirects the entire benchmark.
${CLAUDE_SKILL_DIR}/references/elicitation_rubric_template.md (dimensions, anchors,
probe flavors).Plant a few deliberate control items, blinded and randomized across raters (record who received which
via a probe_arm flag), to anchor the scale, measure rater drift/fatigue, and audit the rubric and
pipeline. Four flavors:
Probes are planted or adjudicated, never fabricated to fit a hypothesis.
Gate: Present the panel composition, stratification, and randomization plan for user review before recruitment is finalized.
/analyze-stats).When the item pool exceeds what one reader can rate in a session, do not make every reader rate
every item (that caps the pool at the per-reader limit). Use anchor-and-rotate (an anchor set
plus rotating incomplete blocks — not a balanced incomplete block design, since reader pairs are
not co-rated equally often): all readers rate a shared anchor set of real items (which carries
the inter-rater reliability; the planted controls carry only the competence check) plus a
rotating unique block each. The binding
constraint is usually the number of available expert readers, so solve the reverse problem (largest
pool for R readers) to size the must-rate set. Pre-specify anchor membership, raters-per-item, and the
rotation seed before rating. Read ${CLAUDE_SKILL_DIR}/references/anchor_rotate_reader_allocation.md
for the formulas, trade-offs, and a stdlib implementation.
Write the machine-readable rating record as a JSON schema, starting from
${CLAUDE_SKILL_DIR}/references/benchmark_export_schema.json: per-item ratings on every rubric
dimension, free-text justifications, follow-up flags, the probe_arm flag, reviewer id and metadata,
item order, and timing.
Gate: Present the final rubric, probe set, panel plan, judge strategy, and export schema together; collect explicit user approval before any rating begins, because changes after collection starts compromise the comparison.
/analyze-stats for ICC (stated form) / Krippendorff's α / kappa (two raters) / DeLong, agreement sample size, and effect-size real-world
translation of the benchmark results/check-reporting for STARD-AI, CLAIM, or TRIPOD+AI item-level reporting once the design is locked/design-study when the broader study around the benchmark (cohort logic, analysis unit,
comparator) also needs review/peer-review or /self-review only after ratings exist and a manuscript is being assessedUse when designing a study that benchmarks AI systems against a human-expert panel, before data collection. Plans the arms, decoupled rubrics with anchors, planted calibration probes, reviewer panel,
---name: design-ai-benchmarkingdescription: Use when designing a study that benchmarks AI systems against a human-expert panel, before data collection…---# skills/design-ai-benchmarking/SKILL.mdओवरव्यू
Use when designing a study that benchmarks AI systems against a human-expert panel, before data collection. Plans the arms, decoupled rubrics with anchors, planted calibration probes, reviewer panel, inter-rater reliability targets and LLM-as-judge versus human adjudication.
FAQ
पब्लिशर से skills/design-ai-benchmarking/SKILL.md डाउनलोड करें, या पाथ कॉपी करके रिपॉज़िटरी खोलें। PlugMCP स्किल को सूचीबद्ध करता है, फ़ाइल होस्ट नहीं करता।
पब्लिशर इसे https://github.com/Aperivue/medsci-skills पर रखता है।
Claude, Cursor, Codex और Grok Build वह SKILL.md पढ़ सकते हैं जिसे आप पब्लिशर की रिपॉज़िटरी से जोड़ते हैं। जब आप चैट में स्किल के GitHub फ़ोल्डर का लिंक भेजते हैं, तो Grok Bot उसे इंस्टॉल कर देता है।