Match-quality variation¶
This analysis measures how much pair-level MQ variation is explained by additive worker and firm components.
flowchart LR
worker[(Worker-profile worker-firm regression data)]
time[(Time-profile worker-firm regression data)]
fit[[Estimate worker, firm,<br/>and worker + firm FE models]]
metrics[(R2 and residual-SD metrics)]
latex[/pairs_variations.tex/]
worker --> fit
time --> fit --> metrics --> latex
classDef input fill:#eee5f5,stroke:#76528c,stroke-width:2px
classDef code fill:#f3f3f3,stroke:#666,stroke-width:2px
classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
classDef tableOutput fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
class worker input
class time input
class fit code
class metrics support
class latex tableOutput
Inputs
| Dataset | Location |
|---|---|
Worker-firm regression data (training_profiles.worker and .time) |
/data/<coverage>/analyses/profile_worker/training/worker_firm_regression_data/ and /data/<coverage>/analyses/profile_time/training/worker_firm_regression_data/ |
Build
| Component | Location |
|---|---|
| Builder | analyses/modules/pairs_variations.R |
| Runtime | hct-tower-analyses-cpu |
| Entry point | analyses/pipelines/pairs_variations.R / pairs-variations |
Output
| Dataset | File / location | Grain and variables |
|---|---|---|
| Pair-variation results | Profile-specific pairs_variations_metrics.csv, pairs_variations.tex, metadata, and pairs_variations_cache.rds under /artifacts/<coverage>/analyses/pairs_variations/profile_worker/ and profile_time/ |
Each table compares in-sample and out-of-sample pairs for that profile; caches are independent by profile |
The pipeline creates one table for each training profile. Each table uses the profile's own worker-firm regression data and compares in-sample with out-of-sample pairs. It reports the no-FE MQ standard deviation and the $R^2$ and residual standard deviation from worker FE, firm FE, and additive worker-plus-firm FE models, plus pair, worker, and firm counts. The input is checked to ensure a pair is not also flagged for the other split. In a time split, a worker-firm pair may contribute distinct in-sample and out-of-sample pair averages based on the periods assigned to each split.
The calculation includes all available calendar years, including 2024. It does not apply a temporary year cutoff.
The additive worker-plus-firm specification deliberately does not include a worker-firm fixed effect, which would absorb the pair component being measured.
Command reference
| Option | Required / default | Source-defined effect |
|---|---|---|
--config PATH |
config/config_one_percent.yaml |
Analyses YAML config. |
--shards ID[,ID,…] |
Unset | Restricts the user-shard subset processed by the pipeline. |
--rerun |
Off | Refit the pair fixed-effects models and replace the metrics cache. Without it, compatible cached metrics are reused to regenerate the table. If older results exist without a compatible cache, the command stops and asks for --rerun. |
| Option | Required / default | Source-defined effect |
|---|---|---|
--task pairs-variations |
Required | Selects the CPU analyses task. |
--config CONFIG.yaml, --shards ID[,ID,…], --rerun |
Config required; shards unset; rerun off | Passed to the R pipeline. Use --rerun to refit and replace cached pair-variation metrics. |
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run |
Wrapper defaults | Standard CPU overrides; default is 8 CPUs, 256 GB, and 12 hours. |
Sharded runs¶
Use --shards with comma-separated user-shard identifiers for a scoped,
exploratory result. Outputs are isolated under shards/user_<ids>/. The first
run fits and caches the models; later runs reuse the cache to rewrite the table.
Use --rerun if the inputs changed or the cached regressions need rebuilding.
Older outputs without a compatible cache stop with an instruction to use
--rerun, rather than silently refitting.