Skip to content

RCID-occupation-year match quality

This reusable analysis dataset aggregates full-training scored worker-firm relationship periods to the Revelio employer, occupation, and calendar-year level. Occupation is represented by the job_category field already present in the worker-firm-period score files. It retains every scored employer and occupation, including private and unmatched firms, and does not join Compustat, CRSP, or any firm mapping.

flowchart LR
    scores[(Worker-firm-period scores)]
    build[Aggregate MQ and<br/>relationship counts<br/>by job category]
    output[(RCID-occupation-year MQ)]

    scores --> build --> output

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#e2f2e7,stroke:#3b7a4b
    class scores source
    class build process
    class output data
Inputs
Dataset Location
Full-training worker-firm-period scores /data/<coverage>/analyses/full_training/worker_firm_period_scores/
Build
Component Location
Builder analyses/modules/build_rcid_occupation_year_mq.R
Runtime hct-tower-analyses-cpu
Entry point analyses/pipelines/build_rcid_occupation_year_mq.R
Output
Dataset File / location Grain Key and variables
RCID-occupation-year match quality rcid_occupation_year_mq.parquet in /data/<coverage>/analyses/full_training/ Revelio employer-occupation-year Key: rcid, job_category, year
MQ statistics: mq_n, mq_sum, mq_sum_sq, mq_mean, mq_variance, mq_sd, mq_min, mq_max
Employment counts: n_relationships, new_relationships

mq_n counts finite model score rows within each RCID-occupation-year. n_relationships counts distinct (user_id, relationship_number) pairs within each group; relationship numbers are worker-specific and must not be counted alone. new_relationships uses the same relationship key and counts a relationship only when its relationship_start_date falls in the output year.

Null or blank job_category values are excluded because they do not identify an occupation group. The dataset is created only from full_training scores. The scoring in-sample/out-of-sample flags are worker-level evaluation labels, not an RCID-occupation-year characteristic; they are intentionally neither retained nor used as an aggregation dimension.

The builder reads the existing worker-firm-period score shards directly. It does not rescore the model or require reprocessing of the underlying employment data.

Command reference
Option Required / default Source-defined effect
--config PATH config/config_one_percent.yaml Analyses YAML config.
--mode full_training full_training; only accepted value Selects the final score mode.
--overwrite Off Rebuilds an existing output.
Option Required / default Source-defined effect
--task build-rcid-occupation-year-mq Required CPU task; mode defaults to and is restricted to full_training.
--config CONFIG.yaml, --overwrite Config required; overwrite off Passed to the pipeline.
--partition, --time, --mem, --cpus, --job-name, --dependency, --exclude, --dry-run Wrapper defaults Standard Slurm overrides. The full-sample default requests 8 CPUs, 512 GB, and 24 hours because the distinct relationship aggregation can be memory-intensive.
cd /hct-tower/analyses
Rscript --vanilla pipelines/build_rcid_occupation_year_mq.R \
  --config config/config_full.yaml --mode full_training
cd /hct-tower
slurm/submit.sh --task build-rcid-occupation-year-mq \
  --config config_full.yaml --mode full_training

Pass --overwrite only to intentionally replace an existing output. The builder writes rcid_occupation_year_mq_metadata.yaml beside the Parquet file with the source score manifest, aggregation definitions, filters, and row counts.