Model build workflow¶
Run this after the employment workflow has produced validated model
relationship periods and company embeddings. Direct commands run from
/hct-tower/model in the model CUDA image (hct-tower-model-cuda-…) with:
The Slurm tasks use the corresponding model CUDA Apptainer image. Use the
samples coverage for an optional smoke test before this workflow when a code
or configuration change needs a cheap check.
-
Configure and run cross-validation for the coverage config you want to examine. The runner uses
training_modes.cross_validation: it fits ontrainand evaluates onvalid; it never chooses a winning trial automatically. Useone_percentfor the main search,samplesfor a cheap smoke test, andfullonly for targeted confirmation of a selected specification.In the chosen
model/config/config_<coverage>.yaml, set the base model inmodel:andtraining:and define one-factor alternatives undercross_validation_search.parameter_groups. The runner always creates the implicitbasetrial, then creates one trial per configured value as<group>__<label>.training_modes: cross_validation: train_splits: [train] eval_splits: [valid] eval_during_training: true cross_validation_search: parameter_groups: learning_rate: "3e-5": training.lr: 3e-5 training.sparse_company_id_lr: 3e-5 capacity_dim: "256": model.capacity_dim: 256 model: transformer_dropout: 0.2 capacity_dim: 128 training: lr: 1e-5 num_iterations: 150000modelandtrainingdefine the base trial. Every alternative must use existing dotted config paths and changes only its listed values. Configure evaluation frequency and comparison metrics intraining:andevaluation:. See Cross-validation for the full configuration explanation.Rerun the desired scope in sequence:
All configured trials
Base trial only
One named alternative
Compare the summary CSV and grouped metric plots, choose both the specification and a checkpoint/iteration, then copy the chosen values into the target coverage's base config.
Produces
Dataset / artifact File / dataset directory Folder Cross-validation trial checkpoints model_*.pt/artifacts/<coverage>/cross_validation/trials/<trial-name>/checkpoints/Trial metrics and plots */artifacts/<coverage>/cross_validation/trials/<trial-name>/{eval,plots}/Trial comparison summary */artifacts/<coverage>/cross_validation/summary/ -
Configure and run final training for the intended coverage. Copy the selected
model.*andtraining.*values from the chosen CV trial into the coverage config (normallymodel/config/config_full.yaml). Set shared optimizer, batching, iteration, precision, and loader settings intraining:; set architecture and loss weights inmodel:. Keep split use and output directories intraining_modes:.Run both fitting modes:
trainingfitstrain + validand evaluates once ontest; use this checkpoint for out-of-sample scores and predictive evidence. Its outputs are scoped underprofile_worker/orprofile_time/.full_trainingfits all splits; use this checkpoint for final match quality and downstream descriptive analyses. It is profile-independent and runs once underfull_training/.
Long-running fitting jobs are resumable from their saved checkpoints. Set
training.save_checkpointsandtraining.checkpoint_every_iterationsbefore submission. For example, to resume a stopped full-datafull_trainingjob from the newest completed epoch automatically:Use
--resume-latest-iterationto resume from the newest regular, potentially mid-epoch checkpoint, or--resume-from-checkpoint <path>to select a specific save. See Final training for both direct and Slurm forms of every resume option.Produces
Dataset / artifact File / dataset directory Folder Worker-profile checkpoints model_*.pt/artifacts/full/profile_worker/training/checkpoints/Worker-profile evaluation and plots */artifacts/full/profile_worker/training/{eval,plots}/Time-profile checkpoints model_*.pt/artifacts/full/profile_time/training/checkpoints/Time-profile evaluation and plots */artifacts/full/profile_time/training/{eval,plots}/Final model checkpoints model_*.pt/artifacts/full/full_training/checkpoints/Final model evaluation and plots */artifacts/full/full_training/{eval,plots}/ -
Preserve the selected checkpoints and configuration metadata. The scorer reads the
trainingcheckpoint for out-of-sample products and thefull_trainingcheckpoint for final MQ products. Do not score or construct analysis data until the required checkpoint has completed.