Disaggregated regression data¶
This dataset materializes the compact, regression-ready subset of the scored period data used by the hazard, predictive-performance, and wage analyses. It is not a copy of the full worker-firm-period score output.
flowchart TB
scores[(Worker-firm-period scores)]
periods[(Model relationship periods)]
firms[(Firms)]
build[Validate aligned shards<br/>and select analysis columns]
output[(Disaggregated regression data)]
scores --> build
periods --> build
firms --> build
build --> output
classDef source fill:#e8eef7,stroke:#52739e
classDef process fill:#fff0cc,stroke:#a66b00
classDef data fill:#eee5f5,stroke:#76528c
class scores,periods,firms source
class build process
class output data
Inputs
| Dataset | Location |
|---|---|
| Worker-firm-period scores | Training: /data/<coverage>/analyses/profile_<worker|time>/training/worker_firm_period/; full training: /data/<coverage>/analyses/full_training/worker_firm_period/ |
| Model relationship periods | /data/<coverage>/processed/model_relationship_periods/ |
| Processed firms | /data/<coverage>/processed/companies_processed.parquet |
Build
| Component | Location |
|---|---|
| Builder | analyses/modules/build_disaggregated_regression_data.R |
| Runtime | hct-tower-analyses-cpu |
| Entry point | analyses/pipelines/build_disaggregated_regression_data.R / build-disaggregated-regression-data |
Output
| Dataset | File / location | Grain | Key and variables |
|---|---|---|---|
| Disaggregated regression data | Score-named Parquet shards in /data/<coverage>/analyses/profile_<worker|time>/training/disaggregated_regression_data/, or /data/<coverage>/analyses/full_training/disaggregated_regression_data/ |
One row per scored period, aligned by row order with the matching score and model-period shard | No unique period key is retained. Identifiers: user_id, company_idxOutcomes, neural predictions, and sample flags: match_score, hazard_label, hazard_baseline_prob, hazard_prob, log_wage, wage_interpolated, is_train_split, is_valid_split, is_test_split, is_in_sample, is_out_of_samplePeriod/job controls: calendar_year, calendar_year_idx, tenure_year, current_seniority, current_job_categoryWorker controls: employee_cat__highest_degree, employee_cat__sex_predicted, employee_cat__ethnicity_predicted, employee_cat__role_k150, employee_cat__user_locationFirm controls: industry2, rics_k50, rics_k400, hq_metro_area, hq_country, observed_size_bucket |
The builder selects the regression outcomes and neural hazard predictions from
the score shards (match_score, hazard_label, hazard_baseline_prob,
hazard_prob, and log_wage), preserves sample flags and calendar/tenure
fields, and adds job, worker, and firm controls from the aligned model-period
and processed-firm inputs. It deliberately excludes the remaining score
schema, including destination fields, relationship_number, company_id,
and period_start.
All worker and firm controls are categorical integer codes rather than their
source labels. The worker-control columns follow the configured
training_features.employee.categorical list; the fields shown above are the
current coverage-config contract. company_idx is the retained firm identifier
used for firm fixed effects. The builder validates source keys and row order
before writing; output shards retain the source filenames and include
_metadata.yaml.
For mode = training, select exactly one profile (worker or time). The
builder reads that profile's scored shards and writes to its matching analysis
root; the selected profile is recorded in _metadata.yaml. full_training
remains profile-independent and does not accept --profile.
Command reference
| Option | Required / default | Source-defined effect |
|---|---|---|
--config PATH |
config/config_one_percent.yaml |
Analyses YAML config. |
--mode training\|full_training |
full_training |
Selects profile-trained scores or the shared full-training scores. |
--profile worker\|time |
Required with training; omit with full_training |
Selects the matching profile-specific score input and output directory. |
--overwrite |
Off | Rebuilds the selected output. |
| Option | Required / default | Source-defined effect |
|---|---|---|
--task build-disaggregated-regression-data |
Required | CPU task. |
--config CONFIG.yaml, --mode, --profile, --overwrite |
Config required; mode full_training; profile required with training; overwrite off |
Passed to the R pipeline. |
Rscript --vanilla pipelines/build_disaggregated_regression_data.R \
--config config/config_full.yaml --mode training --profile worker --overwrite
Rscript --vanilla pipelines/build_disaggregated_regression_data.R \
--config config/config_full.yaml --mode training --profile time --overwrite
Rscript --vanilla pipelines/build_disaggregated_regression_data.R \
--config config/config_full.yaml --mode full_training --overwrite
slurm/submit.sh --task build-disaggregated-regression-data --mode training \
--profile worker --config config_full.yaml --overwrite
slurm/submit.sh --task build-disaggregated-regression-data --mode training \
--profile time --config config_full.yaml --overwrite
slurm/submit.sh --task build-disaggregated-regression-data --mode full_training \
--config config_full.yaml --overwrite
The primary consumers are hazard estimates, predictive performance, and wage regressions.