Skip to content

Model relationship periods

This dataset expands worker-firm relationships into annual rows used by model training, evaluation, scoring, and several descriptive analyses.

flowchart TB
    events[(Employment events)]
    relationships[(Worker-firm relationships)]
    workers[(Workers)]
    firms[(Firms)]
    build[Create annual risk-set rows<br/>and model features]
    periods[(Model relationship periods)]

    events --> build
    relationships --> build
    workers --> build
    firms --> build
    build --> periods

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#e2f2e7,stroke:#3b7a4b
    class events,relationships,workers,firms source
    class build process
    class periods data
Inputs
Dataset Location
Employment events /data/<coverage>/processed/employment_events_processed/
Worker-firm relationships /data/<coverage>/processed/employment_relationships_processed/
Workers /data/<coverage>/processed/employees_processed.parquet
Processed firms /data/<coverage>/processed/companies_processed.parquet
Build
Component Location
Builder model/data/build_relationship_periods.py
Runtime hct-tower-model-cuda-… (CPU execution)
Entry point python -m data.build_model_datasets / build-model-periods
Output
Dataset File / location Grain Key and variables

| Model relationship-period shards and companions | Source-named Parquet shards in /data/<coverage>/processed/model_relationship_periods/ | Annual worker-firm risk-set period | Key: user_id, relationship_number, company_id, period_start
Variables include relationship_start_date, tenure_year, calendar_year, current_seniority, hazard_label, log_wage, wage_growth, destination_company_id, split_label_worker, split_label_time |

Rows contain worker, relationship, and firm keys; period date; tenure and calendar year; current job and wage fields; encoded worker features; employment history; observed destination fields; and hazard_label. The configured split strategy assigns train, valid, or test; under the current worker strategy, all rows for a worker share one split.

The model config's processed.observation_start_date left-truncates this artifact. Historical spells before that date are retained as inputs for tenure and worker history, but no pre-window rows are written. A spell active at the boundary receives a synthetic period beginning on the boundary and then resumes its real anniversary periods, preserving separations early in the first year.

At the configured observation end date, a relationship with no observed relationship-level end date contributes its final, partially observed period with hazard_label = 0. This means no relationship end was recorded by the cutoff; it does not establish that the relationship remained active through the entire anniversary year. A period containing an observed relationship end is labeled 1.

Both split assignments are written once alongside every period row: the seeded worker-level split (split_label_worker) and the time-window split (split_label_time). The historical split_label field remains as an alias for the worker split during downstream migration. Selecting a profile at training time reads its explicit label column; periods, histories, and features remain shared.

Companion outputs under the same processed directory are:

  • model_relationship_metadata.yaml, with source, schema, and split counts;
  • model_relationship_company_counts.parquet, with company example counts by profile and split label.
Command reference
Option Required / default Source-defined effect
--config PATH Required Model YAML config that supplies processed inputs, output paths, split settings, and feature contract.
--only all\|relationship_periods all Restricts data.build_model_datasets to the relationship-period artifact group.
--dry-run Off Validates the config and source paths and prints the plan without writing output.
--no-progress Off Disables progress bars.

The underlying build_relationship_periods.py builder exposes the same --config, --dry-run, and --no-progress options when invoked directly.

Option Required / default Source-defined effect
--task build-model-periods Required Selects slurm/model/build_model_periods.slurm.
--config CONFIG.yaml Required Resolves the model config passed to the builder.
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run Wrapper defaults Standard overrides; default request is CPU-only, 8 CPUs, 128 GB, and two days.
--repo-root, --data-root, --artifact-root, --container-root, --sif Wrapper defaults Override host paths or the Apptainer image.
python -m data.build_model_datasets --config config/config_one_percent.yaml
slurm/submit.sh --task build-model-periods --config config_one_percent.yaml

Output shard names match the processed relationship shards. Scoring and the disaggregated-data builder rely on that alignment.