- name
SKILL.md
O arquivo da skill permanece no repositório do publicador.
Cole o comando copiado em um terminal. Ele instala este SKILL.md nesse agente.
npx skills add Light0305/Light-Science --skill light-crawl -a cursor -yCole o comando copiado em um terminal. Ele instala este SKILL.md nesse agente.
npx skills add Light0305/Light-Science --skill light-crawl -a claude-code -yCole o comando copiado em um terminal. Ele instala este SKILL.md nesse agente.
npx skills add Light0305/Light-Science --skill light-crawl -a codex -yCole o comando copiado em um terminal. Ele instala este SKILL.md nesse agente.
npx skills add Light0305/Light-Science --skill light-crawl -a grok -yEnvie a mensagem copiada em um chat do Grok Bot. O Bot busca este SKILL.md no GitHub e o adiciona às skills que todos os seus Bots compartilham.
Install this skill: https://github.com/Light0305/Light-Science/tree/HEAD/light-crawlDo publicador
需要落盘或继续已有项目时,先读 项目目录约定,按其中的本阶段位置、版本与关联规则保存;用户指定路径和旧项目优先。包内资源相对本 skill,研究产物相对真实项目,二者不要混用。只做局部讨论时无需初始化目录。
A routing-first crawler skill. The agent's job is to compile a target into a deterministic scraper, not to click through every page in a chat loop.
For research, first identify the caller's actual data gap and prefer an existing dataset, lawful download or documented API when sufficient. Read research-data.md when collecting research inputs or handing data to Literature, Design, Experiment or Analysis. Acquisition is an on-demand service in the research network, not a mandatory stage or evidence that a hypothesis is true.
scope.md. The user's request for identified
public data or an existing authorization can establish scope; record its
actual basis without asking again. Clarify a material access/scope gap before
traffic. --authorized records that decision; it does not prove legal rights.light-crawl session acquire <profile> --owner <case-id>, release it when
done, and export state/HAR instead of sharing a profile between tools.evidence.jsonl.Use it when the user wants to:
Do not use it for:
curl already answers, unless the user asks for a repeatable
extractor.Available progress labels are:
INTAKE → PROBED → ROUTED → CAPTURED → UNDERSTOOD → REPRODUCED → BUILT → VERIFIED → OPERATING → LEARNED
Use stages relevant to the task; a simple HTTP extraction needs no reverse
engineering. L2/L3 require their capture/reproduction evidence. A phase label
alone never proves execution: advance VERIFIED cannot create a passed check.
Record an explicitly selected trusted check with verify --run-check; default
verify only checks retained files. learn requires a current executed check
and a valid case. Neither certifies live coverage or scientific suitability.
L5_SCALE is a delivery overlay, not an acquisition lane. A case can be
L0_API + spider delivery.
Fetcher for HTTP,
StealthySession for browser rendering and Cloudflare, Spider for scale.For a real project, use the absolute launcher path and a project case directory:
python "<skill-root>/run_light_crawl.py" doctor
python "<skill-root>/run_light_crawl.py" new-case case --case-root "<project>/Acquisition/ACQ-0001_采集目标" --target https://example.com --authorized
python "<skill-root>/run_light_crawl.py" probe "<project>/Acquisition/ACQ-0001_采集目标/case"
python "<skill-root>/run_light_crawl.py" route "<project>/Acquisition/ACQ-0001_采集目标/case"
python "<skill-root>/run_light_crawl.py" report "<project>/Acquisition/ACQ-0001_采集目标/case"
If installed with install.ps1, use the user-level runtime:
& "$env:USERPROFILE\.light-crawl\src\light-crawl\scripts\light-crawl.ps1" doctor
From a local checkout on Windows:
& "<skill-root>/scripts/light-crawl.ps1" doctor
& "<skill-root>/scripts/light-crawl.ps1" demo --out "<temporary-demo-directory>" --force
After pip install -e ., the same absolute project case paths work with the
light-crawl entrypoint. The examples below also illustrate compatibility with
the existing cases/my-case layout; do not create real project data in the skill
installation directory.
light-crawl doctor
light-crawl new-case my-case --target https://example.com --authorized
light-crawl probe my-case
light-crawl route my-case
light-crawl capture my-case
light-crawl capture my-case --engine scrapling --wait-selector main
light-crawl verify my-case
light-crawl report my-case
light-crawl metrics record my-case --items 100 --requests 120 --success-rate 0.95
light-crawl metrics show my-case
light-crawl session acquire js-reverse-default --owner my-case
light-crawl session status js-reverse-default
light-crawl session release js-reverse-default --owner my-case
light-crawl benchmark
light-crawl benchmark --source-cases cases
light-crawl site list --stale-days 30
light-crawl site show examples/site-packs/books
light-crawl site check examples/site-packs/books --write
light-crawl site migrate --all
light-crawl site run examples/site-packs/quotes
For an offline contract demo:
python run_light_crawl.py demo --out runs/demo --force
Every case keeps its working memory on disk, not in the chat:
cases/<slug>/
scope.md # authorization, rate, PII, stop conditions
target.json # URL/kind/volume/auth/geo
probe.json # cheap HTTP classification
route.json # lane, delivery, confidence, next actions
evidence.jsonl # append-only observation ledger
artifacts/ # hashed files referenced by evidence
har/ # network captures
fixtures/ # redacted request/response samples
captures/CAP-*/ # versioned acquisition snapshots; locally review before sharing
repro/ # sign.py, replay.py, local rebuild
spider/ # Scrapling spider/config
reports/report.md # rendered status
reports/checks/ # immutable selected-check receipts and stdout/stderr
reports/verification.json # pointer to the current selected check
reports/dataset-v01.json # optional explicit research-data handoff
The four tools exchange files, not huge tool outputs:
fixtures/request.json, har/session.har, repro/caller.jsrepro/sign.py, fixtures/*.json, evidence.jsonlevidence.jsonl, endpoint contract, fixtures/spider/, reports/, item counts, error taxonomyCapture evidence points to the versioned snapshot, not mutable top-level fixture working copies. Repeated probes/routes retain hashed observations and can change lanes without moving progress backwards. Preserve previous datasets and raw failures; changes to already-used data require a new export/version and notification to affected downstream tasks.
L0_STATIC.StealthySession and
record_requests=True; the JSON API usually makes L0_API possible.L2_SIGNED when the API candidate and signature hints survive a
second look. A random token string is not proof of a signature.L3_BLOCKED when the response is actually a challenge or the
target returns repeated 403/429.L4_APP when the web surface has no usable endpoint, or the app
contains the protocol you need.For reusable/production delivery, check the relevant items below. A bounded diagnostic task can finish with partial observations and explicit limitations; do not invent a live sample or initialize unrelated stages.
scope.md is explicit and authorization is confirmed.probe.json and route.json exist and agree.evidence.jsonl contains the observation that justified the lane.fixtures/ contains at least one redacted request/response pair for L2/L3.repro/ reproduces the signature or request outside the browser.light-crawl verify <case> passes.verify <case> --run-check repro/verify.py; inspect its receipt and logs.light-crawl metrics record ... has actual run metrics, not only budgets.light-crawl advance <case> VERIFIED --note "..." was recorded.reports/report.md was regenerated.For research inputs, additionally document selection units, fields/units,
missingness, duplicates and coverage in a dataset spec. Export/check the actual
file handoff with dataset export / dataset check; a 200 response, item count,
hash or passed fixture is not a scientific quality verdict.
references/routing.md — lane details, failure taxonomy, escalation.references/acquisition.md — Scrapling patterns and spider design.references/runtime-reverse.md — js-reverse + reverse-skill L2 playbook.references/deep-targets.md — REA for Electron/APK/native/HAR.references/evidence.md — evidence schema and report contract.references/research-data.md — research scope, dataset spec and native downstream handoffs.references/learning.md — site packs, fixtures, benchmarks.references/safety.md — scope, prompt injection, PII, rate policy.docs/CLI.md — stable command reference and exit codes.docs/STABILITY.md — CLI/schema freeze and deprecation policy.docs/SCHEDULING.md — Windows drift task registration.templates/ — generic spider, replay, sign lab, verifier, case docs.Build, debug, and operate reproducible web/app data acquisition, including research dataset handoffs. Use for repeated extraction, pagination, JS-rendered or blocked sites, signed APIs, and verifiable
---name: light-crawldescription: Build, debug, and operate reproducible web/app data acquisition, including research dataset handoffs. Use for…---# light-crawl/SKILL.mdVisão geral
Build, debug, and operate reproducible web/app data acquisition, including research dataset handoffs. Use for repeated extraction, pagination, JS-rendered or blocked sites, signed APIs, and verifiable or resumable crawlers. Route among HTTP/API, Scrapling, runtime inspection and packaged-app analysis as evidence requires; ordinary single-page lookups do not need a crawler.
Perguntas frequentes
Baixe light-crawl/SKILL.md do publicador ou copie o caminho e abra o repositório. O PlugMCP lista a skill e não hospeda o arquivo.
O publicador o mantém em https://github.com/Light0305/Light-Science.
Claude, Cursor, Codex e Grok Build conseguem ler um SKILL.md que você adiciona a partir do repositório do publicador. O Grok Bot o instala quando você envia em um chat o link da pasta da skill no GitHub.