Skip to content

Cross-validation

Cross-validation works for every coverage config through training_modes.cross_validation, fitting on train and evaluating on valid. It is a manual comparison tool: it writes trial evidence but does not select a specification or checkpoint. Use one_percent for the main search, samples for a smoke test, and full only to confirm a selected specification.

cross_validation_search defines an implicit base trial and one-factor trials named <parameter_group>__<value_label>. Each trial writes checkpoints, metrics, and plots beneath:

Artifact Location
Trial outputs training_profiles.<profile>.run_root/cross_validation/trials/<trial-name>/
Comparison summaries training_profiles.<profile>.run_root/cross_validation/summary/

Set the unchanged specification in model: and training:. Then define only the alternatives to compare under cross_validation_search.parameter_groups:

cross_validation_search:
  parameter_groups:
    learning_rate:
      "3e-5":
        training.lr: 3e-5
        training.sparse_company_id_lr: 3e-5
    capacity_dim:
      "256":
        model.capacity_dim: 256

Each group/value pair must contain existing dotted config paths. The example produces base, learning_rate__3e-5, and capacity_dim__256; each alternative changes only its listed paths relative to base.

Configure split assignment in training_profiles.<profile>.splits and the train/valid use in training_modes.cross_validation. Trial and summary directories are derived from the selected profile's run_root. Configure evaluation frequency and selected metrics in training: and evaluation:; those metrics are the basis for trial comparison.

Run and select

Run every configured trial by omitting --trial, run base with --trial base, or run one named alternative with --trial <group>__<label>. In Slurm, set the equivalent CV_TRIAL environment variable. The model build workflow gives direct and Slurm commands for all three choices.

Review the summary CSV and grouped plots, select the specification and a checkpoint/iteration, then copy the selected model.* and training.* values into the base config for held-out and full training.