Skip to content

Worker-firm regression data

Purpose

This builder collapses period-level MQ scores to one row per worker-firm pair and sample status for the pairs_variations analysis. Training builds must select the worker or time profile.

flowchart LR
    scores[(Worker-firm-period scores)] --> aggregate[Average MQ by<br/>worker and firm]
    aggregate --> output[(Worker-firm regression data)]

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#eee5f5,stroke:#76528c
    class scores source
    class aggregate process
    class output data
Inputs
Dataset Location
Worker-firm-period scores Training: /data/<coverage>/analyses/profile_<profile>/training/worker_firm_period/; full training: /data/<coverage>/analyses/full_training/worker_firm_period/
Build
Component Location
Builder analyses/modules/build_worker_firm_regression_data.R
Runtime hct-tower-analyses-cpu
Entry point analyses/pipelines/build_worker_firm_regression_data.R / build-worker-firm-regression-data
Output
Dataset File / location Grain Key and variables
Worker-firm regression data Score-named Parquet shards in the profile-specific or full-training directory Worker-firm pair and sample status Key: user_id, company_id, sample_status
Variables include pair_mq, period_count, in/out-of-sample flags, and mover

Input and transformation

For training, the builder reads the selected profile's worker_firm_period/ shards. It emits separate in-sample and out-of-sample aggregates, including when a time-split worker-firm pair has scored periods in both periods. Each shard is aggregated with DuckDB:

  • pair_mq is mean match_score across non-missing periods in that status;
  • period_count records the contributing periods for that status;
  • sample_status, flags, and status-specific period counts identify the row;
  • mover identifies workers with at least two distinct firms in that status.

For full_training, every scored period is in-sample and each worker-firm pair produces one row.

Score shards are worker-bucketed, so all pairs for a worker and the mover flag can be computed independently within each shard.

Output

/data/<coverage>/analyses/profile_<profile>/training/worker_firm_regression_data/
  <score-shard-name>.parquet
  _metadata.yaml

/data/<coverage>/analyses/full_training/worker_firm_regression_data/
  <score-shard-name>.parquet
  _metadata.yaml

The training grain is one user_id, company_id, and sample_status row; the full-training data has one row per user_id and company_id.

Run

Command reference
Option Required / default Source-defined effect
--config PATH config/config_one_percent.yaml Analyses YAML config.
--mode training\|full_training full_training Selects training-profile or full-training scores.
--profile worker\|time Required with --mode training Selects the training split profile. Omit for full_training.
--overwrite Off Rebuilds the selected output.
Option Required / default Source-defined effect
--task build-worker-firm-regression-data Required CPU task.
--config CONFIG.yaml, --mode, --profile, --overwrite Config required; mode full_training; profile required for training; overwrite off Passed to the R pipeline.
Rscript --vanilla pipelines/build_worker_firm_regression_data.R \
  --config config/config_full.yaml --mode training --profile time --overwrite
Rscript --vanilla pipelines/build_worker_firm_regression_data.R \
  --config config/config_full.yaml --mode training --profile worker --overwrite
Rscript --vanilla pipelines/build_worker_firm_regression_data.R \
  --config config/config_one_percent.yaml --mode full_training --overwrite
slurm/submit.sh --task build-worker-firm-regression-data --mode training --profile time \
  --config config_full.yaml --overwrite
slurm/submit.sh --task build-worker-firm-regression-data --mode training --profile worker \
  --config config_full.yaml --overwrite
slurm/submit.sh --task build-worker-firm-regression-data --mode full_training \
  --config config_one_percent.yaml --overwrite

The builder logs the start, completion, elapsed time, and pair count for every shard. Match-quality variation uses the split flags to select the desired observations.