Model Pipeline

This document is the high-level map of the model pipeline. Step-specific details live in separate files so different people can own training, validation, evaluation, and scoring without crowding one document.

Model Stages

1. Training.
2. Cross-validation.
3. Evaluation.
4. Scoring observed matches.
5. Full-data training.

Coverage Strategy

The same model pipeline should work for:

1. samples
2. one_percent
3. full

Development should happen in that order. If a model step is complete, it should run across all three scales using the relevant config and artifacts.

Inputs

The model pipeline consumes the canonical processed employment data and its sampled equivalents:

• employment_events_processed/
• employees_processed.parquet
• companies_processed.parquet
• about_emb.parquet, if company about embeddings are used.

Before training, these inputs are expanded into the sharded model relationship-period dataset.

Split Semantics

Model splits are derived at runtime from the configured splits.strategy.

For splits.strategy: time, splits use the example date and config cutoffs:

• train: period_start < splits.train_before
• valid: splits.train_before <= period_start < splits.valid_before
• test: period_start >= splits.valid_before

For splits.strategy: worker, every example for a user_id is assigned to the same split using a stable seeded hash of worker_seed:user_id. worker_valid_fraction and worker_test_fraction set the holdout shares, with the remaining workers assigned to train.

The shared data builders do not create these splits. The model runtime applies them when it builds training examples, validation histories, and evaluation targets.

Training Modes

Each model config defines training_modes:

Mode	Train splits	Eval split	Typical use
cross_validation	[train]	valid	hyperparameter and epoch search
training	[train, valid]	test	final out-of-sample test run
full_training	[train, valid, test]	none	final model for downstream scoring/analysis

The base training block contains shared training parameters such as learning rate, batch size, epochs, and runtime settings. The selected training mode controls which splits are used and where artifacts are written.

Recommended Workflow

1. Use samples for smoke tests and broad, cheap checks.
2. Use one_percent cross-validation for the main hyperparameter search.
3. Manually copy the chosen hyperparameters into the base config.
4. Use training mode to train on train + valid and evaluate on test.
5. Use full_training only when the final model should consume all available data before scoring or downstream analysis.

For expensive full-data work, avoid full hyperparameter grids. Prefer a single full-data training trajectory with frequent validation/checkpoints when the goal is choosing the epoch.

Cross-validation

Grouped cross-validation is configured by cross_validation_search. It creates an implicit base trial plus one-factor deviations named <parameter_group>__<value_label>.

The cross-validation runner always uses training_modes.cross_validation. It writes per-trial artifacts under cross_validation_search.trial_root and manual-review outputs under cross_validation_search.summary_output_dir.

Evaluation And Scoring

Standalone evaluation reads a checkpoint from evaluation.ckpt_path and writes metrics/recommendations under evaluation.output_dir.

Observed-match scoring reads a checkpoint from scoring.<mode>.ckpt_path and writes the selected analysis dataset as sharded parquet parts plus metadata under scoring.<mode>.datasets.<dataset>.output_path. The scoring command streams processed relationship/event shards, preserves the relationship shard filenames in the output, and flushes bounded row groups controlled by scoring.<mode>.datasets.<dataset>.rows_per_write. Use --mode training for outputs from the train+valid checkpoint, and --mode full_training when the goal is a final model trained on all available splits.

The resulting products are documented as worker-firm-period scores and destination ranks.