Skip to content

Worker-firm pair data

Purpose

This builder collapses period-level MQ scores to one row per worker-firm pair for the pairs_variations analysis.

flowchart LR
    scores[(Worker-firm-period scores)] --> aggregate[Average MQ by<br/>worker and firm]
    aggregate --> output[(Worker-firm pair data)]

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#eee5f5,stroke:#76528c
    class scores source
    class aggregate process
    class output data

Owner: analyses/
Runtime: analyses container
Module: analyses/modules/build_worker_firm_data.R
Entry point: analyses/pipelines/build_worker_firm_data.R

Input and transformation

For training or full_training, the builder reads the matching worker_firm_period_scores/ shards. Each shard is aggregated with DuckDB:

  • pair_mq is mean match_score across non-missing scored periods;
  • period_count records the number of contributing periods;
  • in-sample and out-of-sample period counts and flags are retained;
  • mover identifies workers with at least two distinct worker-firm pairs.

Score shards are worker-bucketed, so all pairs for a worker and the mover flag can be computed independently within each shard.

Output

/data/<coverage>/analyses/<mode>/worker_firm_regression_table/
  <score-shard-name>.parquet
  _metadata.yaml

The grain is one user_id and company_id pair.

Run

Rscript pipelines/build_worker_firm_data.R \
  --config config/config_one_percent.yaml --mode full_training --overwrite

On Slurm:

slurm/submit.sh --task build-worker-firm-data --mode full_training \
  --config config_one_percent.yaml --overwrite

The builder logs the start, completion, elapsed time, and pair count for every shard. Match-quality variation currently consumes the full-training in-sample rows.