Skip to content

Model build workflow

Run this after the employment workflow has produced validated model relationship periods and company embeddings. Direct commands run from /hct-tower/model in the model CUDA image (hct-tower-model-cuda-…) with:

export PYTHONPATH=/hct-tower:/hct-tower/model

The Slurm tasks use the corresponding model CUDA Apptainer image. Use the samples coverage for an optional smoke test before this workflow when a code or configuration change needs a cheap check.

  1. Configure and run cross-validation for the coverage config you want to examine. The runner uses training_modes.cross_validation: it fits on train and evaluates on valid; it never chooses a winning trial automatically. Use one_percent for the main search, samples for a cheap smoke test, and full only for targeted confirmation of a selected specification.

    In the chosen model/config/config_<coverage>.yaml, set the base model in model: and training: and define one-factor alternatives under cross_validation_search.parameter_groups. The runner always creates the implicit base trial, then creates one trial per configured value as <group>__<label>.

    training_modes:
      cross_validation:
        train_splits: [train]
        eval_splits: [valid]
        eval_during_training: true
    
    cross_validation_search:
      parameter_groups:
        learning_rate:
          "3e-5":
            training.lr: 3e-5
            training.sparse_company_id_lr: 3e-5
        capacity_dim:
          "256":
            model.capacity_dim: 256
    
    model:
      transformer_dropout: 0.2
      capacity_dim: 128
    training:
      lr: 1e-5
      num_iterations: 150000
    

    model and training define the base trial. Every alternative must use existing dotted config paths and changes only its listed values. Configure evaluation frequency and comparison metrics in training: and evaluation:. See Cross-validation for the full configuration explanation.

    Rerun the desired scope in sequence:

    All configured trials

    python -m training.cross_validate --config config/config_one_percent.yaml --profile worker
    
    cd /hct-tower
    slurm/submit.sh --task cross-validate --profile worker --config config_one_percent.yaml
    

    Base trial only

    python -m training.cross_validate --config config/config_one_percent.yaml --profile worker --trial base
    
    cd /hct-tower
    CV_TRIAL=base slurm/submit.sh --task cross-validate --profile worker --config config_one_percent.yaml
    

    One named alternative

    python -m training.cross_validate --config config/config_one_percent.yaml --profile worker --trial learning_rate__3e-5
    
    cd /hct-tower
    CV_TRIAL=learning_rate__3e-5 slurm/submit.sh --task cross-validate --profile worker --config config_one_percent.yaml
    

    Compare the summary CSV and grouped metric plots, choose both the specification and a checkpoint/iteration, then copy the chosen values into the target coverage's base config.

    Produces
    Dataset / artifact File / dataset directory Folder
    Cross-validation trial checkpoints model_*.pt /artifacts/<coverage>/cross_validation/trials/<trial-name>/checkpoints/
    Trial metrics and plots * /artifacts/<coverage>/cross_validation/trials/<trial-name>/{eval,plots}/
    Trial comparison summary * /artifacts/<coverage>/cross_validation/summary/
  2. Configure and run final training for the intended coverage. Copy the selected model.* and training.* values from the chosen CV trial into the coverage config (normally model/config/config_full.yaml). Set shared optimizer, batching, iteration, precision, and loader settings in training:; set architecture and loss weights in model:. Keep split use and output directories in training_modes:.

    Run both fitting modes:

    • training fits train + valid and evaluates once on test; use this checkpoint for out-of-sample scores and predictive evidence. Its outputs are scoped under profile_worker/ or profile_time/.
    • full_training fits all splits; use this checkpoint for final match quality and downstream descriptive analyses. It is profile-independent and runs once under full_training/.
    python -m training.train --config config/config_full.yaml --profile worker --mode training
    python -m training.train --config config/config_full.yaml --mode full_training
    
    cd /hct-tower
    slurm/submit.sh --task train --profile worker --mode training --config config_full.yaml
    slurm/submit.sh --task train --mode full_training --config config_full.yaml
    

    Long-running fitting jobs are resumable from their saved checkpoints. Set training.save_checkpoints and training.checkpoint_every_iterations before submission. For example, to resume a stopped full-data full_training job from the newest completed epoch automatically:

    python -m training.train --config config/config_full.yaml --mode full_training \
      --resume-latest-epoch
    
    cd /hct-tower
    slurm/submit.sh --task train --mode full_training --config config_full.yaml \
      --resume-latest-epoch
    

    Use --resume-latest-iteration to resume from the newest regular, potentially mid-epoch checkpoint, or --resume-from-checkpoint <path> to select a specific save. See Final training for both direct and Slurm forms of every resume option.

    Produces
    Dataset / artifact File / dataset directory Folder
    Worker-profile checkpoints model_*.pt /artifacts/full/profile_worker/training/checkpoints/
    Worker-profile evaluation and plots * /artifacts/full/profile_worker/training/{eval,plots}/
    Time-profile checkpoints model_*.pt /artifacts/full/profile_time/training/checkpoints/
    Time-profile evaluation and plots * /artifacts/full/profile_time/training/{eval,plots}/
    Final model checkpoints model_*.pt /artifacts/full/full_training/checkpoints/
    Final model evaluation and plots * /artifacts/full/full_training/{eval,plots}/
  3. Preserve the selected checkpoints and configuration metadata. The scorer reads the training checkpoint for out-of-sample products and the full_training checkpoint for final MQ products. Do not score or construct analysis data until the required checkpoint has completed.