RCID-occupation-year match quality¶
This reusable analysis dataset aggregates full-training scored worker-firm
relationship periods to the Revelio employer, occupation, and calendar-year
level. Occupation is represented by the job_category field already present in
the worker-firm-period score files. It retains every scored employer and
occupation, including private and unmatched firms, and does not join
Compustat, CRSP, or any firm mapping.
flowchart LR
scores[(Worker-firm-period scores)]
build[Aggregate MQ and<br/>relationship counts<br/>by job category]
output[(RCID-occupation-year MQ)]
scores --> build --> output
classDef source fill:#e8eef7,stroke:#52739e
classDef process fill:#fff0cc,stroke:#a66b00
classDef data fill:#e2f2e7,stroke:#3b7a4b
class scores source
class build process
class output data
Inputs
| Dataset | Location |
|---|---|
| Full-training worker-firm-period scores | /data/<coverage>/analyses/full_training/worker_firm_period_scores/ |
Build
| Component | Location |
|---|---|
| Builder | analyses/modules/build_rcid_occupation_year_mq.R |
| Runtime | hct-tower-analyses-cpu |
| Entry point | analyses/pipelines/build_rcid_occupation_year_mq.R |
Output
| Dataset | File / location | Grain | Key and variables |
|---|---|---|---|
| RCID-occupation-year match quality | rcid_occupation_year_mq.parquet in /data/<coverage>/analyses/full_training/ |
Revelio employer-occupation-year | Key: rcid, job_category, yearMQ statistics: mq_n, mq_sum, mq_sum_sq, mq_mean, mq_variance, mq_sd, mq_min, mq_maxEmployment counts: n_relationships, new_relationships |
mq_n counts finite model score rows within each RCID-occupation-year.
n_relationships counts distinct (user_id, relationship_number) pairs within
each group; relationship numbers are worker-specific and must not be counted
alone. new_relationships uses the same relationship key and counts a
relationship only when its relationship_start_date falls in the output year.
Null or blank job_category values are excluded because they do not identify an
occupation group. The dataset is created only from full_training scores. The
scoring in-sample/out-of-sample flags are worker-level evaluation labels, not an
RCID-occupation-year characteristic; they are intentionally neither retained
nor used as an aggregation dimension.
The builder reads the existing worker-firm-period score shards directly. It does not rescore the model or require reprocessing of the underlying employment data.
Command reference
| Option | Required / default | Source-defined effect |
|---|---|---|
--config PATH |
config/config_one_percent.yaml |
Analyses YAML config. |
--mode full_training |
full_training; only accepted value |
Selects the final score mode. |
--overwrite |
Off | Rebuilds an existing output. |
| Option | Required / default | Source-defined effect |
|---|---|---|
--task build-rcid-occupation-year-mq |
Required | CPU task; mode defaults to and is restricted to full_training. |
--config CONFIG.yaml, --overwrite |
Config required; overwrite off | Passed to the pipeline. |
--partition, --time, --mem, --cpus, --job-name, --dependency, --exclude, --dry-run |
Wrapper defaults | Standard Slurm overrides. The full-sample default requests 8 CPUs, 512 GB, and 24 hours because the distinct relationship aggregation can be memory-intensive. |
Pass --overwrite only to intentionally replace an existing output. The
builder writes rcid_occupation_year_mq_metadata.yaml beside the Parquet file
with the source score manifest, aggregation definitions, filters, and row
counts.