Skip to content

Final training

After model selection, run the two fitting modes needed for full-sample work:

  1. training fits on train + valid and evaluates once on test. Use this checkpoint for out-of-sample scores and predictive evidence.
  2. full_training fits on all configured splits. Use this checkpoint for the final match-quality measure and downstream descriptive analyses.

Held-out training runs require --profile worker or --profile time; their artifacts are kept separately under profile_worker/ and profile_time/. full_training does not take a profile and writes to the shared full_training/ root.

The Training workflow provides the direct and Slurm commands. Avoid a large full-data hyperparameter search: choose the specification on one-percent data, then run a limited number of full-data trajectories.

Configure the selected run

Before submitting the full run, copy the chosen cross-validation values into the coverage config, normally model/config/config_full.yaml:

Config section Set here
model Architecture, representation dimensions, dropout, match-score form, and relationship-head/loss settings
training Learning rates, weight decay, batch and loader settings, iterations, checkpoints, precision, and runtime settings
evaluation Evaluation schedule, metric groups, sample sizes, and destination/pair diagnostics
training_profiles / full_training_artifacts Split definitions, artifact roots, and selected score checkpoints
training_modes.training / training_modes.full_training Which splits each fitting mode uses and evaluates

Do not leave a selected cross-validation override only under cross_validation_search: that section is applied exclusively to CV trial copies. The base model and training settings are what final training reads.

Resume after a wall-time limit

Training checkpoints contain the model, hazard baseline, optimizer, mixed- precision scaler, iteration, and random-number-generator states. A run stopped by Slurm's wall-time limit can therefore continue without starting its fitting trajectory again.

Before submitting a long run, ensure its coverage config retains periodic checkpoints. Choose an interval that leaves enough time for at least one checkpoint to be written before the wall-time limit.

training:
  save_checkpoints: true
  checkpoint_every_iterations: 10000

For a profile-aware run, the trainer writes regular checkpoints under training_profiles.<profile>.run_root/<mode>/checkpoints/; full-sample training uses full_training_artifacts.run_root/checkpoints/. It also writes a completed-epoch checkpoint at every epoch boundary in the sibling epoch_checkpoints/ directory. The trainer can select the newest checkpoint automatically, using the configured directory for the selected mode. Prefer the newest completed- epoch checkpoint: the resumed data loader continues with the following epoch. Resuming from a regular checkpoint saved in the middle of an epoch is supported, but the loader restarts that epoch's stream.

For example, resume a stopped full-data full_training run automatically from its newest completed-epoch checkpoint:

python -m training.train --config config/config_full.yaml --mode full_training \
  --resume-latest-epoch
cd /hct-tower
slurm/submit.sh --task train --mode full_training --config config_full.yaml \
  --resume-latest-epoch

The three resume options are mutually exclusive.

Resume from the latest regular iteration checkpoint

Use --resume-latest-iteration only when you intentionally want the newest regular checkpoint; it may restart the current epoch's data stream.

python -m training.train --config config/config_full.yaml --mode full_training \
  --resume-latest-iteration
cd /hct-tower
slurm/submit.sh --task train --mode full_training --config config_full.yaml \
  --resume-latest-iteration

Resume from a specific checkpoint

Use --resume-from-checkpoint when you deliberately want a particular saved state, for example to reproduce an earlier epoch or to resume a checkpoint that is not the newest one. Specify the full in-container artifact path:

python -m training.train --config config/config_full.yaml --mode full_training \
  --resume-from-checkpoint \
  /artifacts/full/full_training/epoch_checkpoints/model_epoch42_iter420000.pt
cd /hct-tower
slurm/submit.sh --task train --mode full_training --config config_full.yaml \
  --resume-from-checkpoint /artifacts/full/full_training/epoch_checkpoints/model_epoch42_iter420000.pt

Replace the illustrative filename with the checkpoint you want to continue from. Prefer an epoch_checkpoints/model_epoch... file unless you intentionally want a regular checkpoints/model_iter... mid-epoch save.

Use the same mode and compatible model, data, split, batching, and optimizer configuration as the checkpointed run. The current training.num_iterations is the final target, so it may stay unchanged when resubmitting after a wall-time limit. The Slurm log reports the automatically selected path, then confirms resumption with resumed from epoch checkpoint or resumed from mid-epoch checkpoint.