Download skills/05-kthorn-research-superpower/research/evaluating-paper-relevance/SKILL.md from the publisher, or copy the path and open the repository. PlugMCP lists the skill and does not host the file.
Where does Evaluating Paper Relevance live?
The publisher keeps it at https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills.
Which agents can use Evaluating Paper Relevance?
Claude, Cursor, and Codex can read a SKILL.md you add from the publisher's repository.
Two-stage paper screening - abstract scoring then deep dive for specific data extraction
name: Evaluating Paper Relevance
description: Two-stage paper screening - abstract scoring then deep dive for specific data extraction
when_to_use: After literature search returns results. When need to determine if paper contains specific data. When screening papers for relevance. When extracting methods, results, data from papers.
version: 1.0.0
Evaluating Paper Relevance
Overview
Two-stage screening process: quick abstract scoring followed by deep dive into promising papers.
Core principle: Precision over breadth. Find papers that actually contain the specific data/methods user needs, not just topically related papers.
When to Use
Use this skill when:
Have list of papers from search
Need to determine which papers have relevant data
User asks for specific information (measurements, protocols, datasets, etc.)
Screening papers one-by-one
Any research domain (medicinal chemistry, genomics, ecology, computational methods, etc.)
Report to user: "✓ ChEMBL: CHEMBL3870308 (45 data points)"
Structured SAR data available without PDF parsing
Continue to full text fetch for context, methods, discussion.
2. Fetch Full Text
Try in order:
A. PubMed Central (free full text):
# Check if available in PMC
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pmc&term=PMID[PMID]&retmode=json"
# If found, fetch full text XML via API
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=PMCID&rettype=full&retmode=xml"
# Or fetch HTML directly (note: use pmc.ncbi.nlm.nih.gov, not www.ncbi.nlm.nih.gov/pmc)
curl "https://pmc.ncbi.nlm.nih.gov/articles/PMCID/"
B. DOI resolution:
# Try publisher link
curl -L "https://doi.org/10.1234/example.2023"
# May hit paywall - check response
C. Unpaywall (MANDATORY if paywalled):CRITICAL: If step B hits a paywall, you MUST immediately try Unpaywall before giving up.
Use skills/research/finding-open-access-papers to find free OA version:
curl "https://api.unpaywall.org/v2/DOI?email=USER_EMAIL"
# Often finds versions in repositories, preprint servers, author copies
# IMPORTANT: Ask user for their email if not already provided - do NOT use [email protected]
Report to user:
⚠️ Paper behind paywall, checking Unpaywall...
✓ Found open access version at [repository/preprint server]
or
⚠️ Paper behind paywall, checking Unpaywall...
✗ No open access version available - continuing with abstract only
Makes papers easy to access directly from SUMMARY.md
Progress Reporting
CRITICAL: Report to user as you work - never work silently!
For every paper, report:
Start screening:📄 [N/Total] Screening: "Title..."
Abstract score:Abstract score: X/10
Decision: What you're doing next (fetching full text / skipping / etc)
For relevant papers, report findings immediately (adapt to domain):
Medicinal chemistry example:
📄 [15/127] Screening: "Selective BTK inhibitors..."
Abstract score: 8 → Fetching full text...
✓ Found IC50 data for 8 compounds (Table 2)
✓ Selectivity data vs 50 kinases (Figure 3)
→ Added to SUMMARY.md
Genomics example:
📄 [23/89] Screening: "Gene expression in liver disease..."
Abstract score: 9 → Fetching full text...
✓ RNA-seq data available (GEO: GSE12345)
✓ 123 DEGs identified (Table 1, FDR < 0.05)
→ Added to SUMMARY.md
Computational methods example:
📄 [7/45] Screening: "Novel phylogenetic algorithm..."
Abstract score: 8 → Fetching full text...
✓ Code available (github.com/user/tool)
✓ Benchmark results (10x faster, Table 2)
→ Added to SUMMARY.md
Update user every 5-10 papers with summary:
📊 Progress: Reviewed 30/127 papers
- Highly relevant: 3
- Relevant: 5
- Currently screening paper 31...
Why this matters: User needs to see work happening and provide feedback/corrections early
Integration with Other Skills
For medicinal chemistry papers:
Use skills/research/checking-chembl to find curated SAR data
Check BEFORE attempting to parse activity tables from PDFs
~30-40% of medicinal chemistry papers have ChEMBL data
During full text fetching:
If paywalled: MANDATORY to use skills/research/finding-open-access-papers (Unpaywall)
Do NOT skip this step - Unpaywall finds ~50% of paywalled papers for free
After finding relevant paper:
Check ChEMBL (if medicinal chemistry)
Extract findings to SUMMARY.md
Download files to papers/ folder
Call traversing-citations skill to find related papers
Update papers-reviewed.json to avoid re-processing
Scoring Rubric
Score
Meaning
Action
0-4
Not relevant
Skip, brief note in summary
5-6
Possibly relevant
Note for later, skip deep dive for now
7-8
Relevant
Deep dive, extract data, add to summary
9-10
Highly relevant
Deep dive, extract data, follow citations, highlight in summary
Helper Scripts (Optional)
When screening many papers (>20), consider creating a helper script:
Fetch abstracts - PubMed efetch with error handling
Score abstracts - Implement scoring rubric (0-10)
Rate limiting - 500ms delay between API calls (or longer if running parallel subagents)
Save results - JSON with scored papers categorized by relevance
Progress reporting - Print status as it runs
Progressive Enhancement Pattern (Recommended for 50+ papers)
For large-scale screening, use two-script pattern:
Script 1: Abstract Screening (screen_papers.py)
Batch fetch abstracts
Score using rubric (0-10)
Categorize by relevance
Output: evaluated-papers.json with basic metadata
Script 2: Deep Dive (deep_dive_papers.py)
Read Script 1 output
Fetch full text for highly relevant papers (score ≥8)
Extract domain-specific data (measurements, protocols, datasets, etc.)
Update same JSON file with enhanced metadata
Benefits:
Can run steps independently - Score abstracts once, re-run deep dive multiple times
Resume if interrupted - No need to re-fetch abstracts if deep dive fails
Re-run deep dive without re-scoring abstracts - Adjust extraction logic, keep scores
Consistent and reproducible - Same scoring logic applied to all papers
Save API calls - Abstract screening happens once, deep dive only on relevant papers
Script design:
Parameterize keywords and data types for specific query
Progressive enhancement - add detail to same JSON file
Include rate limiting (500ms between API calls for single script, longer if parallel)
Keep scripts with research session for reproducibility
When NOT to create helper script:
Few papers (<20)
One-off quick searches
Manual screening is faster
Common Mistakes
Not tracking all papers: Only adding relevant papers to papers-reviewed.json → Add EVERY paper regardless of score to prevent re-review
Skipping Unpaywall: Hitting paywall and giving up → ALWAYS check Unpaywall first, many papers have free versions
Creating unnecessary files for small searches: For <50 papers, use ONLY papers-reviewed.json and SUMMARY.md. For large searches (>100 papers), structured evaluated-papers.json and auxiliary files (README.md, TOP_PRIORITY_PAPERS.md) add significant value and should be used.
Too strict: Skipping papers that mention data indirectly → Re-read abstract carefully
Too lenient: Deep diving into tangentially related papers → Focus on specific data user needs
Missing supplementary data: Many papers hide key data in SI → Always check for supplementary files
Silent screening: User can't see progress → Report EVERY paper as you screen it
No periodic summaries: User loses big picture → Update every 5-10 papers
Non-clickable DOIs/PMIDs: Plain text identifiers → Always use markdown links
Re-reviewing papers: Wastes time → Always check papers-reviewed.json first
Not using helper scripts: Manually screening 100+ papers → Consider batch script
Quick Reference
Task
Action
Check if reviewed
Look up DOI in papers-reviewed.json
Score abstract
Keywords (0-3) + Data type (0-4) + Specificity (0-3)
Get full text
Try PMC → DOI → Unpaywall → Preprints
Find data
Grep for terms, focus on Methods/Results/Tables
Download PDF
curl -L -o papers/FILE.pdf URL
Update tracking
Add to papers-reviewed.json + SUMMARY.md
Next Steps
After evaluating paper:
If score ≥ 7: Call skills/research/traversing-citations
Continue to next paper in search results
Check if reached 50 papers or 5 minutes → ask user to continue or stop
Auxiliary Files (for large searches >100 papers)
README.md Template
Use this structure for research projects with 100+ papers:
Project Overview
Query description
Target molecules/topics
Date completed
Quick Start Guide
Where to start reading
Priority lists
File Inventory
Description of each file
What each is used for
Key Findings Summary
Statistics
Top findings
Coverage by category
Methodology
Scoring rubric
Decision rules
Data sources
Next Steps
Recommended actions
Priority order
TOP_PRIORITY_PAPERS.md Template
For datasets with >50 relevant papers, create curated priority list:
Organized by tier (Tier 1: Must-read, Tier 2: High-value, etc.)
Download skills/05-kthorn-research-superpower/research/evaluating-paper-relevance/SKILL.md from the publisher, or copy the path and open the repository. PlugMCP lists the skill and does not host the file.
Where does Evaluating Paper Relevance live?
The publisher keeps it at https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills.
Which agents can use Evaluating Paper Relevance?
Claude, Cursor, and Codex can read a SKILL.md you add from the publisher's repository.