-
Pre-flight: ensure container + system probe.
"$SKILL_DIR/scripts/kermt_container.sh" check_system | python -c "
import json, sys; d = json.load(sys.stdin)
if not d['ok']:
print('System check failed:', d['gaps']); sys.exit(1)
print(f'OK: {len(d[\"gpus\"])} GPU(s); CUDA via container toolkit')
"
Refuse to proceed if ok: false.
-
Compute run directory.
RUN_DIR=$KERMT_REPO/runs/finetune_$(date -u +%Y-%m-%dT%H-%M-%SZ)
-
Resolve & validate the checkpoint.
Resolve — only if --ckpt was omitted. Default to the released
pretrained hybrid model nvidia/NV-KERMT-70M-v2:
Validate the resolved (or user-provided) ckpt:
"$SKILL_DIR/scripts/kermt_container.sh" run --ckpt <user-ckpt> -- \
"python /skill/scripts/check_checkpoint.py --mode finetune_init --ckpt /ckpt"
Parse the JSON. Abort on ok: false. The validator rejects already-
finetuned ckpts (has_task_ffn: true) with a redirect to kermt-infer.
-
Validate the data.
"$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> -- \
"python /skill/scripts/check_data.py --mode finetune --csv /data/<basename> [--targets COL1 COL2 ...]"
If --targets was not given by the user, surface auto_detected_targets
from the JSON and ask the user to confirm before continuing. Abort on
ok: false.
-
Prepare the data (skip if --from-prepare given).
Pre-flight: check for sibling val.csv / test.csv. Before invoking
prepare_data, inspect the parent directory of <user-csv>. If a
canonical-looking sibling val.csv (or val_*.csv — common variants
include val_T.csv, val_clean.csv) AND a matching test.csv /
test_*.csv exist next to the train CSV, the dataset ships its own
pre-defined split. In that case set --split-type index_predetermined
AND pass --val-csv / --test-csv — otherwise the configured
split_type (default scaffold_balanced) will re-split the train CSV
from scratch and silently discard the user's val/test files. When in
doubt — or when the sibling files use non-canonical suffixes (_T,
_v2, etc.) — surface the situation to the user and ask which they
want.
Quoting target names. If any of the --targets column names
contain shell metacharacters (>, &, |, (, ), $, etc.),
single-quote each one when passing on the CLI to keep the shell from
eating part of the name. Example: --targets 'Log_Caco2_Papp_A>B' 'logD'. The CSV header itself is read directly by the downstream
trainer and is unaffected, but the prepare_data.json manifest's
targets[] field captures whatever the shell delivers — unquoted
metacharacters get truncated there.
Mount note: kermt_container.sh --data <host-csv> mounts the
parent directory of <host-csv> at /data. --val-csv and
--test-csv must therefore reference files in that same parent
directory. If val/test live in a separate directory (e.g. a sibling
splits/ folder), mount the parent of all three using --data <dir>
on a directory rather than a file.
"$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> --run-dir $RUN_DIR -- \
"python /skill/scripts/prepare_data.py --mode finetune \\
--csv /data/<basename> --out /runs/data \\
--split-type <split_type> \\
[--val-csv /data/<val-basename> --test-csv /data/<test-basename>] \\
[--val-frac 0.1 --test-frac 0.1 --seed 0] \\
--targets <COL1> [COL2 ...]"
Outputs land at $RUN_DIR/data/prepare_data.json. For scaffold_balanced
and , prep emits a single + ;
the runner passes them through to which calls
internally with the user-supplied seed.
-
Estimate runtime + echo applied defaults.
- Finetune wall time is typically minutes-to-hours on 1 GPU.
- Surface a summary of every flag that was filled from the defaults
vs user-supplied, so the user knows what was assumed. The runner
records this in
args_applied.
- Sample message:
"Filling from defaults_finetune.json: epochs=30, batch_size=32, split_type=scaffold_balanced. Override any of these with --<flag>."
-
Targets confirmation gate (hard requirement). Before launching the
runner, regardless of how the targets list was determined (CLI --targets,
auto-detection in step 4, or a user natural-language request like
"finetune on Caco2 and HLM"), echo the final targets list to the user with
an explicit count:
"Will finetune on N target(s): COL1, COL2, ...". If the user's request
specified a subset that doesn't match this list (e.g., they asked for 2
tasks via natural language but the list still has 4), treat it as a
discrepancy and re-prompt with the diff — never silently proceed on the
wrong target set. Wait for explicit confirmation before launching unless
--yes was given.
-
Launch the runner detached. (Consistent with the pretrain skills.)
"$SKILL_DIR/scripts/kermt_container.sh" run_detached \\
--name kermt-finetune-<ts> \\
--ckpt <user-ckpt> --run-dir $RUN_DIR -- \\
"python /skill/scripts/run_finetune_local.py \\
--ckpt /ckpt \\
--prepare-manifest /runs/data/prepare_data.json \\
--dataset-type <type> \\
--out /runs \\
[--gpus 0] \\
[--num-gpus N] \\
[--epochs N --batch-size N --init-lr F ...] \\
[--ffn-num-task-specific-layers N --ffn-task-specific-hidden-size H]"
Returns the container name + id + log file path.
-
Report to the user. Output a short summary:
- Container name + id
$RUN_DIR/run.json (manifest with cmd_replay + image digest)- Log file:
$RUN_DIR/logs/finetune.log
- TensorBoard:
$RUN_DIR/logs/tb (open with tensorboard --logdir $RUN_DIR/logs/tb)
- Final checkpoints land at
$RUN_DIR/ckpt/fold_0/model_0/model.pt
(best-val) and last_checkpoint.pt (sibling, auto-resume target).
Held-out test predictions + metrics land at
$RUN_DIR/ckpt/fold_0/test_result.csv. Paths vary with --num-folds
/ --ensemble-size.
- To follow progress:
kermt-monitor <RUN_DIR> (one-shot) or
docker logs -f <container-name> (streaming).
- To block until the run finishes (useful for short test runs):
docker wait <container-name> — prints the exit code on completion.