Skip to content

Wage regressions

This analysis relates log wages to MQ across model samples, fixed-effect specifications, and relationship-tenure groups.

flowchart LR
    data[(Available disaggregated data<br/>for five cohorts)]
    fit[[Fit five specifications<br/>in three tenure contexts per cohort]]
    models[(Compact fits, two-way covariance,<br/>and metrics)]
    csv[(wage_regressions_metrics.csv)]
    latex[/wage_regressions.tex/]

    data --> fit --> models
    models --> csv --> latex

    classDef input fill:#eee5f5,stroke:#76528c,stroke-width:2px
    classDef code fill:#f3f3f3,stroke:#666,stroke-width:2px
    classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
    classDef tableOutput fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
    class data input
    class fit code
    class models,csv support
    class latex tableOutput
Inputs
Dataset Location
Disaggregated regression data Full-training: /data/<coverage>/analyses/full_training/disaggregated_regression_data/; worker profile: /data/<coverage>/analyses/profile_worker/training/disaggregated_regression_data/; time profile: /data/<coverage>/analyses/profile_time/training/disaggregated_regression_data/
Build
Component Location
Builder analyses/modules/wage_regressions.R
Runtime hct-tower-analyses-cpu
Entry point analyses/pipelines/wage_regressions.R / wage-regressions
Output
Dataset File / location Grain and variables
Wage-regression results Metrics, compact fits, and wage_regressions.tex in /artifacts/<coverage>/analyses/wage_regression/<cohort>/ Cohorts are full_training_in_sample, profile_worker_in_sample, profile_worker_out_of_sample, profile_time_in_sample, and profile_time_out_of_sample; unavailable inputs are skipped

With no selectors, one invocation attempts all five tables:

  • full-training in-sample;
  • worker-profile in-sample and out-of-sample;
  • time-profile in-sample and out-of-sample.

The pipeline logs and skips a cohort if its configured data directory or Parquet shards are unavailable, then continues with the remaining cohorts. If --shards requests a shard that is absent for a cohort, that cohort is also skipped. Model or data errors after a cohort's inputs are found still stop the run.

Each table contains panels for all relationships, tenure 0–4, and tenure 5+. Within each panel, five columns add observable controls, firm fixed effects, worker fixed effects, and worker-plus-firm fixed effects to the MQ-only model. Standard errors are clustered two ways by worker and firm. Saved metrics also record the wage interpretation of a +0.1 MQ change.

All available calendar years are included, including 2024.

Command reference
Option Required / default Source-defined effect
--config PATH config/config_one_percent.yaml Analyses YAML config.
--mode training\|full_training Optional Restricts processing to the selected mode. full_training permits only in_sample.
--profile worker\|time Optional Restricts training-profile processing to one profile. Omit for both profiles.
--sample in_sample\|out_of_sample Optional Restricts processing to one sample status. Omit to process the available statuses.
--shards ID[,ID,…] Unset Restricts to user shards and writes isolated shard-scoped outputs.
--rerun Off Refits every saved model instead of reusing valid persisted fits.
Option Required / default Source-defined effect
--task wage-regressions Required Selects the CPU analyses task.
--config CONFIG.yaml, --mode, --profile, --sample, --shards, --rerun Config required; selectors optional Passed to the R pipeline. Without selectors the pipeline attempts all five cohorts and skips unavailable inputs.
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run Wrapper defaults Standard CPU overrides; default is 8 CPUs, 800 GB, and two days.
cd /hct-tower/analyses
Rscript --vanilla pipelines/wage_regressions.R \
  --config config/config_full.yaml
cd /hct-tower
slurm/submit.sh --task wage-regressions --config config_full.yaml

A normal rerun reuses valid model and metric bundles and can format the table without reopening source data. --rerun refits every specification.

Sharded runs

Use --shards with comma-separated user-shard identifiers for a scoped, exploratory result. Outputs are isolated under shards/user_<ids>/ beneath each cohort. You can restrict the run, for example, to the time-profile out-of-sample table with --mode training --profile time --sample out_of_sample.

cd /hct-tower/analyses
Rscript --vanilla pipelines/wage_regressions.R \
  --config config/config_full.yaml --mode training --profile time \
  --sample out_of_sample --shards 00,01
cd /hct-tower
slurm/submit.sh --task wage-regressions --config config_full.yaml \
  --mode training --profile time --sample out_of_sample --shards 00,01