Checkpoints¶
Each training mode writes checkpoints, evaluation outputs, and plots beneath
the selected profile's run_root/<mode>/ directory. Profile-independent
full_training writes beneath full_training_artifacts.run_root/.
| Mode | Checkpoint role | Primary downstream use |
|---|---|---|
cross_validation |
Trial checkpoint | Compare specifications and epochs |
training |
Held-out checkpoint | Out-of-sample scoring and evaluation |
full_training |
Final checkpoint | Full-sample match-quality scores |
Keep the checkpoint together with its run configuration and evaluation
artifacts. Scoring uses the checkpoint selected in the active profile or
full_training_artifacts; update that selection when choosing a different
checkpoint.
Cross-validation writes artifacts under
training_profiles.<profile>.run_root/cross_validation/trials/<trial-name>/
and comparison summaries under
training_profiles.<profile>.run_root/cross_validation/summary/.
Resuming training¶
For training and full_training, regular checkpoints are written to the
resolved checkpoint directory as model_iter<iteration>.pt. Completed-epoch
checkpoints are written to the sibling epoch_checkpoints/ directory as
model_epoch<epoch>_iter<iteration>.pt. They retain model, optimizer, scaler,
iteration, and RNG state for continuation after a Slurm wall-time limit.
Resume automatically with training.train --resume-latest-epoch or
slurm/submit.sh ... --resume-latest-epoch. It selects the checkpoint with the
largest saved iteration from the selected mode's
epoch_checkpoints/ directory. --resume-latest-iteration selects the latest
regular checkpoint instead; it is valid but may restart the loader stream for
that epoch. --resume-from-checkpoint <path> remains available for an explicit
choice. Full commands and the required checkpoint settings are in Final
training.