Skip to content

Checkpoints

Each training mode writes checkpoints, evaluation outputs, and plots beneath the selected profile's run_root/<mode>/ directory. Profile-independent full_training writes beneath full_training_artifacts.run_root/.

Mode Checkpoint role Primary downstream use
cross_validation Trial checkpoint Compare specifications and epochs
training Held-out checkpoint Out-of-sample scoring and evaluation
full_training Final checkpoint Full-sample match-quality scores

Keep the checkpoint together with its run configuration and evaluation artifacts. Scoring uses the checkpoint selected in the active profile or full_training_artifacts; update that selection when choosing a different checkpoint.

Cross-validation writes artifacts under training_profiles.<profile>.run_root/cross_validation/trials/<trial-name>/ and comparison summaries under training_profiles.<profile>.run_root/cross_validation/summary/.

Resuming training

For training and full_training, regular checkpoints are written to the resolved checkpoint directory as model_iter<iteration>.pt. Completed-epoch checkpoints are written to the sibling epoch_checkpoints/ directory as model_epoch<epoch>_iter<iteration>.pt. They retain model, optimizer, scaler, iteration, and RNG state for continuation after a Slurm wall-time limit.

Resume automatically with training.train --resume-latest-epoch or slurm/submit.sh ... --resume-latest-epoch. It selects the checkpoint with the largest saved iteration from the selected mode's epoch_checkpoints/ directory. --resume-latest-iteration selects the latest regular checkpoint instead; it is valid but may restart the loader stream for that epoch. --resume-from-checkpoint <path> remains available for an explicit choice. Full commands and the required checkpoint settings are in Final training.