Model relationship periods¶
This dataset expands worker-firm relationships into annual rows used by model training, evaluation, scoring, and several descriptive analyses.
flowchart TB
events[(Employment events)]
relationships[(Worker-firm relationships)]
workers[(Workers)]
firms[(Firms)]
build[Create annual risk-set rows<br/>and model features]
periods[(Model relationship periods)]
events --> build
relationships --> build
workers --> build
firms --> build
build --> periods
classDef source fill:#e8eef7,stroke:#52739e
classDef process fill:#fff0cc,stroke:#a66b00
classDef data fill:#e2f2e7,stroke:#3b7a4b
class events,relationships,workers,firms source
class build process
class periods data
Inputs
| Dataset | Location |
|---|---|
| Employment events | /data/<coverage>/processed/employment_events_processed/ |
| Worker-firm relationships | /data/<coverage>/processed/employment_relationships_processed/ |
| Workers | /data/<coverage>/processed/employees_processed.parquet |
| Processed firms | /data/<coverage>/processed/companies_processed.parquet |
Build
| Component | Location |
|---|---|
| Builder | model/data/build_relationship_periods.py |
| Runtime | hct-tower-model-cuda-… (CPU execution) |
| Entry point | python -m data.build_model_datasets / build-model-periods |
Output
| Dataset | File / location | Grain | Key and variables |
|---|---|---|---|
| Model relationship-period shards and companions | Source-named Parquet shards in /data/<coverage>/processed/model_relationship_periods/ | Annual worker-firm risk-set period | Key: user_id, relationship_number, company_id, period_start
Variables include relationship_start_date, tenure_year, calendar_year, current_seniority, hazard_label, log_wage, wage_growth, destination_company_id, split_label_worker, split_label_time |
Rows contain worker, relationship, and firm keys; period date; tenure and
calendar year; current job and wage fields; encoded worker features; employment
history; observed destination fields; and hazard_label. The configured split
strategy assigns train, valid, or test; under the current worker strategy,
all rows for a worker share one split.
The model config's processed.observation_start_date left-truncates this
artifact. Historical spells before that date are retained as inputs for tenure
and worker history, but no pre-window rows are written. A spell active at the
boundary receives a synthetic period beginning on the boundary and then resumes
its real anniversary periods, preserving separations early in the first year.
At the configured observation end date, a relationship with no observed
relationship-level end date contributes its final, partially observed period
with hazard_label = 0. This means no relationship end was recorded by the
cutoff; it does not establish that the relationship remained active through
the entire anniversary year. A period containing an observed relationship end
is labeled 1.
Both split assignments are written once alongside every period row: the seeded
worker-level split (split_label_worker) and the time-window split
(split_label_time). The historical split_label field remains as an alias
for the worker split during downstream migration. Selecting a profile at
training time reads its explicit label column; periods, histories, and features
remain shared.
Companion outputs under the same processed directory are:
model_relationship_metadata.yaml, with source, schema, and split counts;model_relationship_company_counts.parquet, with company example counts by profile and split label.
Command reference
| Option | Required / default | Source-defined effect |
|---|---|---|
--config PATH |
Required | Model YAML config that supplies processed inputs, output paths, split settings, and feature contract. |
--only all\|relationship_periods |
all |
Restricts data.build_model_datasets to the relationship-period artifact group. |
--dry-run |
Off | Validates the config and source paths and prints the plan without writing output. |
--no-progress |
Off | Disables progress bars. |
The underlying build_relationship_periods.py builder exposes the same
--config, --dry-run, and --no-progress options when invoked directly.
| Option | Required / default | Source-defined effect |
|---|---|---|
--task build-model-periods |
Required | Selects slurm/model/build_model_periods.slurm. |
--config CONFIG.yaml |
Required | Resolves the model config passed to the builder. |
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run |
Wrapper defaults | Standard overrides; default request is CPU-only, 8 CPUs, 128 GB, and two days. |
--repo-root, --data-root, --artifact-root, --container-root, --sif |
Wrapper defaults | Override host paths or the Apptainer image. |
Output shard names match the processed relationship shards. Scoring and the disaggregated-data builder rely on that alignment.