Cross-validation¶
Cross-validation works for every coverage config through
training_modes.cross_validation, fitting on train and evaluating on
valid. It is a manual comparison tool: it writes trial evidence but does not
select a specification or checkpoint. Use one_percent for the main search,
samples for a smoke test, and full only to confirm a selected
specification.
cross_validation_search defines an implicit base trial and one-factor
trials named <parameter_group>__<value_label>. Each trial writes checkpoints,
metrics, and plots beneath:
| Artifact | Location |
|---|---|
| Trial outputs | training_profiles.<profile>.run_root/cross_validation/trials/<trial-name>/ |
| Comparison summaries | training_profiles.<profile>.run_root/cross_validation/summary/ |
Configure a search¶
Set the unchanged specification in model: and training:. Then define only
the alternatives to compare under cross_validation_search.parameter_groups:
cross_validation_search:
parameter_groups:
learning_rate:
"3e-5":
training.lr: 3e-5
training.sparse_company_id_lr: 3e-5
capacity_dim:
"256":
model.capacity_dim: 256
Each group/value pair must contain existing dotted config paths. The example
produces base, learning_rate__3e-5, and capacity_dim__256; each
alternative changes only its listed paths relative to base.
Configure split assignment in training_profiles.<profile>.splits and the
train/valid use in training_modes.cross_validation. Trial and summary
directories are derived from the selected profile's run_root. Configure
evaluation frequency and selected metrics in training: and evaluation:;
those metrics are the basis for trial comparison.
Run and select¶
Run every configured trial by omitting --trial, run base with --trial
base, or run one named alternative with --trial <group>__<label>. In Slurm,
set the equivalent CV_TRIAL environment variable. The model build
workflow gives direct and Slurm commands for all three choices.
Review the summary CSV and grouped plots, select the specification and a
checkpoint/iteration, then copy the selected model.* and training.* values
into the base config for held-out and full training.