- name
SKILL.md
Le fichier du skill reste dans le dépôt de l'éditeur.
Collez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add fcakyon/labrat --skill compare -a cursor -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add fcakyon/labrat --skill compare -a claude-code -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add fcakyon/labrat --skill compare -a codex -yCollez la commande copiée dans un terminal. Elle installe ce SKILL.md dans cet agent.
npx skills add fcakyon/labrat --skill compare -a grok -yEnvoyez le message copié dans un chat Grok Bot. Le Bot récupère ce SKILL.md sur GitHub et l'ajoute aux skills que partagent tous vos Bots.
Install this skill: https://github.com/fcakyon/labrat/tree/HEAD/plugin/skills/compareDe l'éditeur
The most common comparison error is reporting "run A is 4 percentage points behind baseline" when run A is at epoch 11 of 100 and the baseline number is from epoch 100. The student is still training; the comparison is meaningless. This skill enforces same-epoch alignment.
The agentic Stop hook routes here from reason when an assistant reports a delta without aligning the runs.
If the question is "why did X fail / diverge / underperform", go to /labrat:debug first.
Check in this order:
WANDB_API_KEY env var set, or wandb imports or a *.yaml config with wandb keys in the project → wandbNEPTUNE_API_TOKEN env var set, or a *.yaml config with neptune keys → neptuneMLFLOW_TRACKING_URI env var set, or mlruns/ dir present → mlflowruns/ or lightning_logs/ dir present → tensorboard*results*.json / *meta*.json / eval*.csv files in run dirs → local file format: read one sample file and write the analysis code for that format, not a hardcoded scriptIf none, ask the user where metrics live before guessing.
Get full names (no shortcodes). If the user says "run-b vs the baseline", clarify:
run-b run (project + entity + run-id)You need the full curve, not the last reported value. Final-value-only comparisons hide convergence dynamics.
For wandb:
import wandb
api = wandb.Api()
run = api.run("entity/project/run-id")
history = run.history(samples=10000) # full history, not just summary
For tensorboard, parse the event files (tensorboard.backend.event_processing.event_accumulator.EventAccumulator).
For neptune / mlflow, use their respective APIs.
The student is the run still in progress (or the one being evaluated). Get its current epoch / step from the latest history row.
This is the critical step. The baseline went all the way to (say) epoch 100. The student is at epoch 11. Pull the baseline's metrics at epoch 11, not at epoch 100.
student_step = student_history['epoch'].max()
baseline_at_same_step = baseline_history[baseline_history['epoch'] == student_step]
If the baseline doesn't have an exactly-matching step, interpolate or pick the nearest. State which.
Most ML pipelines have a proxy metric (cheap, computed during training, kNN accuracy on features, loss, perplexity) and a target downstream metric (expensive, computed periodically or only at the end, finetuned linear probe accuracy, downstream task F1).
The proxy is for tracking convergence; the target is what the project is actually optimizing. Reporting only the proxy can mislead, a run that lags on kNN may close the gap on downstream finetune. Report both, separately:
| student (ep 11) | baseline (ep 11) | delta |
| proxy (kNN top-1) | 36.4% | 38.9% | -2.5 |
| downstream (linear) | not yet | 42.1% | n/a |
If the user only has proxy data, say so explicitly. Never declare a winner from proxy alone.
In every line of the report, use full run names. Never run-a vs run-b; always imagenet-resnet50-baseline-bs256 vs imagenet-resnet50-distill-from-vit-l. Future-you reading this will not remember the shortcode.
Compact comparison table per metric pair (proxy + downstream). Each row aligned at the student's current step. Each cell traceable to a specific tracker run-id and step. End with one or two sentences interpreting the comparison, student is on track to catch up at step N, projected from current slope is a useful framing; student is winning / losing is rarely warranted before convergence.
Compare runs across more than the primary metric:
Put them in the Results Table of the experiment report below, and identify:
/labrat:rebuttalWhen the user asks for a report or summary:
## Experiment Report
### Overview
- Total runs: N
- Date range: start to end
- Best result: [metric] = [value] (run [name])
### Results Table
| Run | Step | Config Diff | Proxy | Downstream | GPU Hours | Status |
|-----|------|-------------|-------|------------|-----------|--------|
### Key Findings
1. [Finding with evidence]
2. [Finding with evidence]
### Recommendations
1. [Next experiment to try, with justification]
Align every Results Table row at the same step and keep proxy and downstream apart, as in Output above.
Set up periodic monitoring:
Create a CronCreate job on */10 * * * * that pipes a {"title", "message"} JSON object to the plugin notifier, which forwards it to whatever /labrat:setup configured (ntfy, Slack or email):
echo "{\"title\": \"Training check\", \"message\": \"GPU util: $(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader)\"}" | ${CLAUDE_PLUGIN_ROOT}/scripts/notify.sh
Same-epoch comparison and analysis of training runs across wandb, neptune, tensorboard, mlflow, or local result files. Aligns runs at the student's current step (never current-vs-final-of-baseline) an
---name: comparedescription: Same-epoch comparison and analysis of training runs across wandb, neptune, tensorboard, mlflow, or local…---# plugin/skills/compare/SKILL.mdVue d'ensemble
Same-epoch comparison and analysis of training runs across wandb, neptune, tensorboard, mlflow, or local result files. Aligns runs at the student's current step (never current-vs-final-of-baseline) and separates proxy metrics from downstream targets. Also writes run summaries and experiment reports, analyzes ablations, and monitors training. Use when the user asks to compare runs, check if a run is improving, track lag against a baseline, rank experiments, evaluate run-vs-run performance, analyze results, write a training report or experiment summary, summarize training results, monitor training, or asks which run is best.
FAQ
Téléchargez plugin/skills/compare/SKILL.md auprès de l'éditeur, ou copiez le chemin et ouvrez le dépôt. PlugMCP répertorie le skill et n'héberge pas le fichier.
L'éditeur le maintient sur https://github.com/fcakyon/labrat.
Claude, Cursor, Codex et Grok Build peuvent lire un SKILL.md que vous ajoutez depuis le dépôt de l'éditeur. Grok Bot l'installe lorsque vous envoyez dans un chat le lien du dossier GitHub du skill.