- name
SKILL.md
Le fichier du skill reste dans le dépôt de l'éditeur.
Collez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add Light0305/Light-Science --skill light-crawl -a cursor -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add Light0305/Light-Science --skill light-crawl -a claude-code -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add Light0305/Light-Science --skill light-crawl -a codex -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add Light0305/Light-Science --skill light-crawl -a grok -yEnvoyez le message copié dans un chat Grok Bot. Le Bot récupère ce SKILL.md sur GitHub et l'ajoute aux skills que partagent tous vos Bots.
Install this skill: https://github.com/Light0305/Light-Science/tree/HEAD/light-crawlDe l'éditeur
需要落盘或继续已有项目时,先读 项目目录约定,按其中的本阶段位置、版本与关联规则保存;用户指定路径和旧项目优先。包内资源相对本 skill,研究产物相对真实项目,二者不要混用。只做局部讨论时无需初始化目录。
A routing-first crawler skill. The agent's job is to compile a target into a deterministic scraper, not to click through every page in a chat loop.
For research, first identify the caller's actual data gap and prefer an existing dataset, lawful download or documented API when sufficient. Read research-data.md when collecting research inputs or handing data to Literature, Design, Experiment or Analysis. Acquisition is an on-demand service in the research network, not a mandatory stage or evidence that a hypothesis is true.
scope.md. The user's request for identified
public data or an existing authorization can establish scope; record its
actual basis without asking again. Clarify a material access/scope gap before
traffic. --authorized records that decision; it does not prove legal rights.light-crawl session acquire <profile> --owner <case-id>, release it when
done, and export state/HAR instead of sharing a profile between tools.evidence.jsonl.Use it when the user wants to:
Do not use it for:
curl already answers, unless the user asks for a repeatable
extractor.Available progress labels are:
INTAKE → PROBED → ROUTED → CAPTURED → UNDERSTOOD → REPRODUCED → BUILT → VERIFIED → OPERATING → LEARNED
Use stages relevant to the task; a simple HTTP extraction needs no reverse
engineering. L2/L3 require their capture/reproduction evidence. A phase label
alone never proves execution: advance VERIFIED cannot create a passed check.
Record an explicitly selected trusted check with verify --run-check; default
verify only checks retained files. learn requires a current executed check
and a valid case. Neither certifies live coverage or scientific suitability.
L5_SCALE is a delivery overlay, not an acquisition lane. A case can be
L0_API + spider delivery.
Fetcher for HTTP,
StealthySession for browser rendering and Cloudflare, Spider for scale.For a real project, use the absolute launcher path and a project case directory:
python "<skill-root>/run_light_crawl.py" doctor
python "<skill-root>/run_light_crawl.py" new-case case --case-root "<project>/Acquisition/ACQ-0001_采集目标" --target https://example.com --authorized
python "<skill-root>/run_light_crawl.py" probe "<project>/Acquisition/ACQ-0001_采集目标/case"
python "<skill-root>/run_light_crawl.py" route "<project>/Acquisition/ACQ-0001_采集目标/case"
python "<skill-root>/run_light_crawl.py" report "<project>/Acquisition/ACQ-0001_采集目标/case"
If installed with install.ps1, use the user-level runtime:
& "$env:USERPROFILE\.light-crawl\src\light-crawl\scripts\light-crawl.ps1" doctor
From a local checkout on Windows:
& "<skill-root>/scripts/light-crawl.ps1" doctor
& "<skill-root>/scripts/light-crawl.ps1" demo --out "<temporary-demo-directory>" --force
After pip install -e ., the same absolute project case paths work with the
light-crawl entrypoint. The examples below also illustrate compatibility with
the existing cases/my-case layout; do not create real project data in the skill
installation directory.
light-crawl doctor
light-crawl new-case my-case --target https://example.com --authorized
light-crawl probe my-case
light-crawl route my-case
light-crawl capture my-case
light-crawl capture my-case --engine scrapling --wait-selector main
light-crawl verify my-case
light-crawl report my-case
light-crawl metrics record my-case --items 100 --requests 120 --success-rate 0.95
light-crawl metrics show my-case
light-crawl session acquire js-reverse-default --owner my-case
light-crawl session status js-reverse-default
light-crawl session release js-reverse-default --owner my-case
light-crawl benchmark
light-crawl benchmark --source-cases cases
light-crawl site list --stale-days 30
light-crawl site show examples/site-packs/books
light-crawl site check examples/site-packs/books --write
light-crawl site migrate --all
light-crawl site run examples/site-packs/quotes
For an offline contract demo:
python run_light_crawl.py demo --out runs/demo --force
Every case keeps its working memory on disk, not in the chat:
cases/<slug>/
scope.md # authorization, rate, PII, stop conditions
target.json # URL/kind/volume/auth/geo
probe.json # cheap HTTP classification
route.json # lane, delivery, confidence, next actions
evidence.jsonl # append-only observation ledger
artifacts/ # hashed files referenced by evidence
har/ # network captures
fixtures/ # redacted request/response samples
captures/CAP-*/ # versioned acquisition snapshots; locally review before sharing
repro/ # sign.py, replay.py, local rebuild
spider/ # Scrapling spider/config
reports/report.md # rendered status
reports/checks/ # immutable selected-check receipts and stdout/stderr
reports/verification.json # pointer to the current selected check
reports/dataset-v01.json # optional explicit research-data handoff
The four tools exchange files, not huge tool outputs:
fixtures/request.json, har/session.har, repro/caller.jsrepro/sign.py, fixtures/*.json, evidence.jsonlevidence.jsonl, endpoint contract, fixtures/spider/, reports/, item counts, error taxonomyCapture evidence points to the versioned snapshot, not mutable top-level fixture working copies. Repeated probes/routes retain hashed observations and can change lanes without moving progress backwards. Preserve previous datasets and raw failures; changes to already-used data require a new export/version and notification to affected downstream tasks.
L0_STATIC.StealthySession and
record_requests=True; the JSON API usually makes L0_API possible.L2_SIGNED when the API candidate and signature hints survive a
second look. A random token string is not proof of a signature.L3_BLOCKED when the response is actually a challenge or the
target returns repeated 403/429.L4_APP when the web surface has no usable endpoint, or the app
contains the protocol you need.For reusable/production delivery, check the relevant items below. A bounded diagnostic task can finish with partial observations and explicit limitations; do not invent a live sample or initialize unrelated stages.
scope.md is explicit and authorization is confirmed.probe.json and route.json exist and agree.evidence.jsonl contains the observation that justified the lane.fixtures/ contains at least one redacted request/response pair for L2/L3.repro/ reproduces the signature or request outside the browser.light-crawl verify <case> passes.verify <case> --run-check repro/verify.py; inspect its receipt and logs.light-crawl metrics record ... has actual run metrics, not only budgets.light-crawl advance <case> VERIFIED --note "..." was recorded.reports/report.md was regenerated.For research inputs, additionally document selection units, fields/units,
missingness, duplicates and coverage in a dataset spec. Export/check the actual
file handoff with dataset export / dataset check; a 200 response, item count,
hash or passed fixture is not a scientific quality verdict.
references/routing.md — lane details, failure taxonomy, escalation.references/acquisition.md — Scrapling patterns and spider design.references/runtime-reverse.md — js-reverse + reverse-skill L2 playbook.references/deep-targets.md — REA for Electron/APK/native/HAR.references/evidence.md — evidence schema and report contract.references/research-data.md — research scope, dataset spec and native downstream handoffs.references/learning.md — site packs, fixtures, benchmarks.references/safety.md — scope, prompt injection, PII, rate policy.docs/CLI.md — stable command reference and exit codes.docs/STABILITY.md — CLI/schema freeze and deprecation policy.docs/SCHEDULING.md — Windows drift task registration.templates/ — generic spider, replay, sign lab, verifier, case docs.Build, debug, and operate reproducible web/app data acquisition, including research dataset handoffs. Use for repeated extraction, pagination, JS-rendered or blocked sites, signed APIs, and verifiable
---name: light-crawldescription: Build, debug, and operate reproducible web/app data acquisition, including research dataset handoffs. Use for…---# light-crawl/SKILL.mdVue d'ensemble
Build, debug, and operate reproducible web/app data acquisition, including research dataset handoffs. Use for repeated extraction, pagination, JS-rendered or blocked sites, signed APIs, and verifiable or resumable crawlers. Route among HTTP/API, Scrapling, runtime inspection and packaged-app analysis as evidence requires; ordinary single-page lookups do not need a crawler.
FAQ
Téléchargez light-crawl/SKILL.md auprès de l'éditeur, ou copiez le chemin et ouvrez le dépôt. PlugMCP répertorie le skill et n'héberge pas le fichier.
L'éditeur le maintient sur https://github.com/Light0305/Light-Science.
Claude, Cursor, Codex et Grok Build peuvent lire un SKILL.md que vous ajoutez depuis le dépôt de l'éditeur. Grok Bot l'installe lorsque vous envoyez dans un chat le lien du dossier GitHub du skill.