Skip to content

Disaggregated regression data

This dataset materializes the compact, regression-ready subset of the scored period data used by the hazard, predictive-performance, and wage analyses. It is not a copy of the full worker-firm-period score output.

flowchart TB
    scores[(Worker-firm-period scores)]
    periods[(Model relationship periods)]
    firms[(Firms)]
    build[Validate aligned shards<br/>and select analysis columns]
    output[(Disaggregated regression data)]

    scores --> build
    periods --> build
    firms --> build
    build --> output

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#eee5f5,stroke:#76528c
    class scores,periods,firms source
    class build process
    class output data
Inputs
Dataset Location
Worker-firm-period scores Training: /data/<coverage>/analyses/profile_<worker|time>/training/worker_firm_period/; full training: /data/<coverage>/analyses/full_training/worker_firm_period/
Model relationship periods /data/<coverage>/processed/model_relationship_periods/
Processed firms /data/<coverage>/processed/companies_processed.parquet
Build
Component Location
Builder analyses/modules/build_disaggregated_regression_data.R
Runtime hct-tower-analyses-cpu
Entry point analyses/pipelines/build_disaggregated_regression_data.R / build-disaggregated-regression-data
Output
Dataset File / location Grain Key and variables
Disaggregated regression data Score-named Parquet shards in /data/<coverage>/analyses/profile_<worker|time>/training/disaggregated_regression_data/, or /data/<coverage>/analyses/full_training/disaggregated_regression_data/ One row per scored period, aligned by row order with the matching score and model-period shard No unique period key is retained. Identifiers: user_id, company_idx
Outcomes, neural predictions, and sample flags: match_score, hazard_label, hazard_baseline_prob, hazard_prob, log_wage, wage_interpolated, is_train_split, is_valid_split, is_test_split, is_in_sample, is_out_of_sample
Period/job controls: calendar_year, calendar_year_idx, tenure_year, current_seniority, current_job_category
Worker controls: employee_cat__highest_degree, employee_cat__sex_predicted, employee_cat__ethnicity_predicted, employee_cat__role_k150, employee_cat__user_location
Firm controls: industry2, rics_k50, rics_k400, hq_metro_area, hq_country, observed_size_bucket

The builder selects the regression outcomes and neural hazard predictions from the score shards (match_score, hazard_label, hazard_baseline_prob, hazard_prob, and log_wage), preserves sample flags and calendar/tenure fields, and adds job, worker, and firm controls from the aligned model-period and processed-firm inputs. It deliberately excludes the remaining score schema, including destination fields, relationship_number, company_id, and period_start.

All worker and firm controls are categorical integer codes rather than their source labels. The worker-control columns follow the configured training_features.employee.categorical list; the fields shown above are the current coverage-config contract. company_idx is the retained firm identifier used for firm fixed effects. The builder validates source keys and row order before writing; output shards retain the source filenames and include _metadata.yaml.

For mode = training, select exactly one profile (worker or time). The builder reads that profile's scored shards and writes to its matching analysis root; the selected profile is recorded in _metadata.yaml. full_training remains profile-independent and does not accept --profile.

Command reference
Option Required / default Source-defined effect
--config PATH config/config_one_percent.yaml Analyses YAML config.
--mode training\|full_training full_training Selects profile-trained scores or the shared full-training scores.
--profile worker\|time Required with training; omit with full_training Selects the matching profile-specific score input and output directory.
--overwrite Off Rebuilds the selected output.
Option Required / default Source-defined effect
--task build-disaggregated-regression-data Required CPU task.
--config CONFIG.yaml, --mode, --profile, --overwrite Config required; mode full_training; profile required with training; overwrite off Passed to the R pipeline.
Rscript --vanilla pipelines/build_disaggregated_regression_data.R \
  --config config/config_full.yaml --mode training --profile worker --overwrite
Rscript --vanilla pipelines/build_disaggregated_regression_data.R \
  --config config/config_full.yaml --mode training --profile time --overwrite
Rscript --vanilla pipelines/build_disaggregated_regression_data.R \
  --config config/config_full.yaml --mode full_training --overwrite
slurm/submit.sh --task build-disaggregated-regression-data --mode training \
  --profile worker --config config_full.yaml --overwrite
slurm/submit.sh --task build-disaggregated-regression-data --mode training \
  --profile time --config config_full.yaml --overwrite
slurm/submit.sh --task build-disaggregated-regression-data --mode full_training \
  --config config_full.yaml --overwrite

The primary consumers are hazard estimates, predictive performance, and wage regressions.