Skip to content

Match-quality variation

This analysis measures how much pair-level MQ variation is explained by additive worker and firm components.

flowchart LR
    worker[(Worker-profile worker-firm regression data)]
    time[(Time-profile worker-firm regression data)]
    fit[[Estimate worker, firm,<br/>and worker + firm FE models]]
    metrics[(R2 and residual-SD metrics)]
    latex[/pairs_variations.tex/]

    worker --> fit
    time --> fit --> metrics --> latex

    classDef input fill:#eee5f5,stroke:#76528c,stroke-width:2px
    classDef code fill:#f3f3f3,stroke:#666,stroke-width:2px
    classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
    classDef tableOutput fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
    class worker input
    class time input
    class fit code
    class metrics support
    class latex tableOutput
Inputs
Dataset Location
Worker-firm regression data (training_profiles.worker and .time) /data/<coverage>/analyses/profile_worker/training/worker_firm_regression_data/ and /data/<coverage>/analyses/profile_time/training/worker_firm_regression_data/
Build
Component Location
Builder analyses/modules/pairs_variations.R
Runtime hct-tower-analyses-cpu
Entry point analyses/pipelines/pairs_variations.R / pairs-variations
Output
Dataset File / location Grain and variables
Pair-variation results Profile-specific pairs_variations_metrics.csv, pairs_variations.tex, metadata, and pairs_variations_cache.rds under /artifacts/<coverage>/analyses/pairs_variations/profile_worker/ and profile_time/ Each table compares in-sample and out-of-sample pairs for that profile; caches are independent by profile

The pipeline creates one table for each training profile. Each table uses the profile's own worker-firm regression data and compares in-sample with out-of-sample pairs. It reports the no-FE MQ standard deviation and the $R^2$ and residual standard deviation from worker FE, firm FE, and additive worker-plus-firm FE models, plus pair, worker, and firm counts. The input is checked to ensure a pair is not also flagged for the other split. In a time split, a worker-firm pair may contribute distinct in-sample and out-of-sample pair averages based on the periods assigned to each split.

The calculation includes all available calendar years, including 2024. It does not apply a temporary year cutoff.

The additive worker-plus-firm specification deliberately does not include a worker-firm fixed effect, which would absorb the pair component being measured.

Command reference
Option Required / default Source-defined effect
--config PATH config/config_one_percent.yaml Analyses YAML config.
--shards ID[,ID,…] Unset Restricts the user-shard subset processed by the pipeline.
--rerun Off Refit the pair fixed-effects models and replace the metrics cache. Without it, compatible cached metrics are reused to regenerate the table. If older results exist without a compatible cache, the command stops and asks for --rerun.
Option Required / default Source-defined effect
--task pairs-variations Required Selects the CPU analyses task.
--config CONFIG.yaml, --shards ID[,ID,…], --rerun Config required; shards unset; rerun off Passed to the R pipeline. Use --rerun to refit and replace cached pair-variation metrics.
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run Wrapper defaults Standard CPU overrides; default is 8 CPUs, 256 GB, and 12 hours.
cd /hct-tower/analyses
Rscript --vanilla pipelines/pairs_variations.R \
  --config config/config_one_percent.yaml
cd /hct-tower
slurm/submit.sh --task pairs-variations \
  --config config_full.yaml

Sharded runs

Use --shards with comma-separated user-shard identifiers for a scoped, exploratory result. Outputs are isolated under shards/user_<ids>/. The first run fits and caches the models; later runs reuse the cache to rewrite the table. Use --rerun if the inputs changed or the cached regressions need rebuilding. Older outputs without a compatible cache stop with an instruction to use --rerun, rather than silently refitting.

cd /hct-tower/analyses
Rscript --vanilla pipelines/pairs_variations.R \
  --config config/config_full.yaml --shards 00,01
cd /hct-tower
slurm/submit.sh --task pairs-variations --config config_full.yaml \
  --shards 00,01