Final training¶
After model selection, run the two fitting modes needed for full-sample work:
trainingfits ontrain + validand evaluates once ontest. Use this checkpoint for out-of-sample scores and predictive evidence.full_trainingfits on all configured splits. Use this checkpoint for the final match-quality measure and downstream descriptive analyses.
Held-out training runs require --profile worker or --profile time; their
artifacts are kept separately under profile_worker/ and profile_time/.
full_training does not take a profile and writes to the shared full_training/
root.
The Training workflow provides the direct and Slurm commands. Avoid a large full-data hyperparameter search: choose the specification on one-percent data, then run a limited number of full-data trajectories.
Configure the selected run¶
Before submitting the full run, copy the chosen cross-validation values into
the coverage config, normally model/config/config_full.yaml:
| Config section | Set here |
|---|---|
model |
Architecture, representation dimensions, dropout, match-score form, and relationship-head/loss settings |
training |
Learning rates, weight decay, batch and loader settings, iterations, checkpoints, precision, and runtime settings |
evaluation |
Evaluation schedule, metric groups, sample sizes, and destination/pair diagnostics |
training_profiles / full_training_artifacts |
Split definitions, artifact roots, and selected score checkpoints |
training_modes.training / training_modes.full_training |
Which splits each fitting mode uses and evaluates |
Do not leave a selected cross-validation override only under
cross_validation_search: that section is applied exclusively to CV trial
copies. The base model and training settings are what final training reads.
Resume after a wall-time limit¶
Training checkpoints contain the model, hazard baseline, optimizer, mixed- precision scaler, iteration, and random-number-generator states. A run stopped by Slurm's wall-time limit can therefore continue without starting its fitting trajectory again.
Before submitting a long run, ensure its coverage config retains periodic checkpoints. Choose an interval that leaves enough time for at least one checkpoint to be written before the wall-time limit.
For a profile-aware run, the trainer writes regular checkpoints under
training_profiles.<profile>.run_root/<mode>/checkpoints/; full-sample training
uses full_training_artifacts.run_root/checkpoints/. It also writes a
completed-epoch checkpoint at every epoch boundary in the sibling epoch_checkpoints/
directory. The trainer can select the newest checkpoint automatically, using
the configured directory for the selected mode. Prefer the newest completed-
epoch checkpoint: the resumed data loader continues with the following epoch.
Resuming from a regular checkpoint saved in the middle of an epoch is
supported, but the loader restarts that epoch's stream.
For example, resume a stopped full-data full_training run automatically from
its newest completed-epoch checkpoint:
The three resume options are mutually exclusive.
Resume from the latest regular iteration checkpoint¶
Use --resume-latest-iteration only when you intentionally want the newest
regular checkpoint; it may restart the current epoch's data stream.
Resume from a specific checkpoint¶
Use --resume-from-checkpoint when you deliberately want a particular saved
state, for example to reproduce an earlier epoch or to resume a checkpoint
that is not the newest one. Specify the full in-container artifact path:
Replace the illustrative filename with the checkpoint you want to continue
from. Prefer an epoch_checkpoints/model_epoch... file unless you intentionally
want a regular checkpoints/model_iter... mid-epoch save.
Use the same mode and compatible model, data, split, batching, and optimizer
configuration as the checkpointed run. The current training.num_iterations
is the final target, so it may stay unchanged when resubmitting after a
wall-time limit. The Slurm log reports the automatically selected path, then
confirms resumption with resumed from epoch checkpoint or resumed from
mid-epoch checkpoint.