- name
SKILL.md
Tệp skill vẫn nằm trong kho mã của nhà phát hành.
Dán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add fcakyon/labrat --skill debug -a cursor -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add fcakyon/labrat --skill debug -a claude-code -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add fcakyon/labrat --skill debug -a codex -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add fcakyon/labrat --skill debug -a grok -yGửi tin nhắn đã sao chép trong cuộc trò chuyện Grok Bot. Bot sẽ lấy tệp SKILL.md này từ GitHub và thêm vào bộ skill mà mọi Bot của bạn dùng chung.
Install this skill: https://github.com/fcakyon/labrat/tree/HEAD/plugin/skills/debugTừ nhà phát hành
The most expensive class of mistake in ML debugging is asserting a cause based on plausibility, then attempting a "fix" that masks the real problem. This skill enforces the discipline of probe → hypothesis → smoke → controls → claim, in that order.
The agentic Stop hook routes here from reason when an assistant claims a cause without backing tool output.
The user just said any of:
Before forming any hypothesis, gather the cheap evidence. None of these cost more than a few seconds:
Process state:
ps aux | grep -E '(python|train|torchrun|accelerate)' | grep -v grep
Is the process still running? Zombie? Defunct? Multiple instances?
Kernel / system events:
dmesg | tail -100 # OOM kills, hardware errors, NFS errors
journalctl -xe --since "1 hour ago" | tail -50
GPU state:
nvidia-smi
nvidia-smi --query-gpu=utilization.gpu,memory.used,temperature.gpu --format=csv
Is the GPU even being used? Idle GPU during "training" means the process is blocked on data loading or has died.
Disk / filesystem:
df -h /path/to/run-dir
du -sh /path/to/run-dir/*
Out of disk? Checkpoints not being written?
Log scrollback: Read the last few hundred lines of the training log. Don't trust the user's summary, they may have skimmed. Look for:
Checkpoint state:
ls -la /path/to/run-dir/checkpoints/
When was the last checkpoint written? What does its size suggest? An empty .pt is different from a 2GB one cut short.
After the probe, state what might be happening, explicitly framed as a hypothesis:
"Hypothesis: the run is OOMing because dmesg shows
oom-kill3 minutes ago and the process is gone. Alternative hypotheses I haven't ruled out: (a) NFS write timeout, (b) explicit kill from a sibling process."
Never skip to "the cause is X." The hypothesis labels what you don't yet know.
The cheapest way to confirm or refute a hypothesis is to reproduce the failure shape under a controlled condition:
batch_size=1 for 1 step. If it survives, OOM is confirmed; if it fails the same way, OOM is wrong.eval() mode. Loss finite? Outputs sane?lr=0. If the loss still explodes, the loss itself is broken (not the optimizer).A 30-second smoke beats a 30-minute restart-and-pray.
If the smoke is ambiguous, run a control: change exactly one variable from the failing config and rerun the smoke. The differences narrow what mechanism is responsible.
Common control axes (change one at a time):
Only after evidence stacks up, probe, smoke, control, do you assert a cause. The claim should cite the specific tool output that proves it:
"Root cause: NFS write timeout. Evidence: dmesg shows
nfs server X not respondingat 14:23 (the same minute the last checkpoint was written), and the smoke withbatch=1reproduces the timeout. Recommended fix: bind-mount a local scratch dir for checkpoints and rsync to NFS at end of epoch."
If the evidence isn't stacking up, do not promote a hypothesis to a cause. Say "I don't yet know" and propose the next probe.
A concise diagnostic report: (1) what the probes showed, (2) the hypothesis, (3) the smoke outcome, (4) the cause-or-uncertain verdict, (5) the recommended next action. Each claim cites the tool output that backs it.
Evidence-before-action diagnosis of failing ML experiments. Probes the system before guessing causes, process list, dmesg, GPU stats, log scrollback, checkpoint state, then states a hypothesis as a hy
---name: debugdescription: Evidence-before-action diagnosis of failing ML experiments. Probes the system before guessing causes, process…---# plugin/skills/debug/SKILL.mdTổng quan
Evidence-before-action diagnosis of failing ML experiments. Probes the system before guessing causes, process list, dmesg, GPU stats, log scrollback, checkpoint state, then states a hypothesis as a hypothesis and runs a smoke before claiming a root cause. Use when the user asks why a run is failing, diverging, OOMing, hanging, slow, producing weird metrics, has crashed, or asks to debug, diagnose, troubleshoot, or investigate a training issue.
Câu hỏi thường gặp
Tải plugin/skills/debug/SKILL.md từ nhà phát hành, hoặc sao chép đường dẫn và mở kho mã. PlugMCP chỉ liệt kê skill và không lưu trữ tệp này.
Nhà phát hành duy trì nó tại https://github.com/fcakyon/labrat.
Claude, Cursor, Codex và Grok Build có thể đọc tệp SKILL.md mà bạn thêm từ kho mã của nhà phát hành. Grok Bot cài đặt skill khi bạn gửi liên kết thư mục GitHub của skill trong cuộc trò chuyện.