Worker-firm regression data¶
Purpose¶
This builder collapses period-level MQ scores to one row per worker-firm pair
and sample status for the pairs_variations analysis. Training builds must
select the worker or time profile.
flowchart LR
scores[(Worker-firm-period scores)] --> aggregate[Average MQ by<br/>worker and firm]
aggregate --> output[(Worker-firm regression data)]
classDef source fill:#e8eef7,stroke:#52739e
classDef process fill:#fff0cc,stroke:#a66b00
classDef data fill:#eee5f5,stroke:#76528c
class scores source
class aggregate process
class output data
Inputs
| Dataset | Location |
|---|---|
| Worker-firm-period scores | Training: /data/<coverage>/analyses/profile_<profile>/training/worker_firm_period/; full training: /data/<coverage>/analyses/full_training/worker_firm_period/ |
Build
| Component | Location |
|---|---|
| Builder | analyses/modules/build_worker_firm_regression_data.R |
| Runtime | hct-tower-analyses-cpu |
| Entry point | analyses/pipelines/build_worker_firm_regression_data.R / build-worker-firm-regression-data |
Output
| Dataset | File / location | Grain | Key and variables |
|---|---|---|---|
| Worker-firm regression data | Score-named Parquet shards in the profile-specific or full-training directory | Worker-firm pair and sample status | Key: user_id, company_id, sample_statusVariables include pair_mq, period_count, in/out-of-sample flags, and mover |
Input and transformation¶
For training, the builder reads the selected profile's
worker_firm_period/ shards. It emits separate in-sample and out-of-sample
aggregates, including when a time-split worker-firm pair has scored periods in
both periods. Each shard is aggregated with DuckDB:
pair_mqis meanmatch_scoreacross non-missing periods in that status;period_countrecords the contributing periods for that status;sample_status, flags, and status-specific period counts identify the row;moveridentifies workers with at least two distinct firms in that status.
For full_training, every scored period is in-sample and each worker-firm pair
produces one row.
Score shards are worker-bucketed, so all pairs for a worker and the mover flag can be computed independently within each shard.
Output¶
/data/<coverage>/analyses/profile_<profile>/training/worker_firm_regression_data/
<score-shard-name>.parquet
_metadata.yaml
/data/<coverage>/analyses/full_training/worker_firm_regression_data/
<score-shard-name>.parquet
_metadata.yaml
The training grain is one user_id, company_id, and sample_status row;
the full-training data has one row per user_id and company_id.
Run¶
Command reference
| Option | Required / default | Source-defined effect |
|---|---|---|
--config PATH |
config/config_one_percent.yaml |
Analyses YAML config. |
--mode training\|full_training |
full_training |
Selects training-profile or full-training scores. |
--profile worker\|time |
Required with --mode training |
Selects the training split profile. Omit for full_training. |
--overwrite |
Off | Rebuilds the selected output. |
| Option | Required / default | Source-defined effect |
|---|---|---|
--task build-worker-firm-regression-data |
Required | CPU task. |
--config CONFIG.yaml, --mode, --profile, --overwrite |
Config required; mode full_training; profile required for training; overwrite off |
Passed to the R pipeline. |
Rscript --vanilla pipelines/build_worker_firm_regression_data.R \
--config config/config_full.yaml --mode training --profile time --overwrite
Rscript --vanilla pipelines/build_worker_firm_regression_data.R \
--config config/config_full.yaml --mode training --profile worker --overwrite
Rscript --vanilla pipelines/build_worker_firm_regression_data.R \
--config config/config_one_percent.yaml --mode full_training --overwrite
slurm/submit.sh --task build-worker-firm-regression-data --mode training --profile time \
--config config_full.yaml --overwrite
slurm/submit.sh --task build-worker-firm-regression-data --mode training --profile worker \
--config config_full.yaml --overwrite
slurm/submit.sh --task build-worker-firm-regression-data --mode full_training \
--config config_one_percent.yaml --overwrite
The builder logs the start, completion, elapsed time, and pair count for every shard. Match-quality variation uses the split flags to select the desired observations.