Skip to content

RCID-year match quality

This reusable analysis dataset aggregates scored worker-firm relationship periods to the Revelio employer-year level. It retains every scored employer, including private and unmatched firms, and deliberately does not join Compustat, CRSP, or any firm mapping. Training-profile builds keep in-sample and out-of-sample periods separate, yielding one row per RCID-year-status.

flowchart LR
    scores[(Profile-specific worker-firm-period scores)]
    build[Aggregate MQ and<br/>relationship counts by sample status]
    output[(One RCID-year MQ dataset<br/>profile rows include sample_status)]

    scores --> build --> output

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#e2f2e7,stroke:#3b7a4b
    class scores source
    class build process
    class output data
Inputs
Dataset Location
Worker-firm-period scores Training profile: /data/<coverage>/analyses/profile_<profile>/training/worker_firm_period/; full training: /data/<coverage>/analyses/full_training/worker_firm_period/
Build
Component Location
Builder analyses/modules/build_rcid_year_mq.R
Runtime hct-tower-analyses-cpu
Entry point analyses/pipelines/build_rcid_year_mq.R
Output
Dataset File / location Grain Key and variables
RCID-year match quality (training, worker profile) rcid_year_mq.parquet in /data/<coverage>/analyses/profile_worker/training/ Revelio employer-year and sample status Key: rcid, year, sample_status
MQ statistics: mq_n, mq_sum, mq_sum_sq, mq_mean, mq_variance, mq_sd, mq_min, mq_max
Counts: n_relationships, new_relationships
RCID-year match quality (training, time profile) rcid_year_mq.parquet in /data/<coverage>/analyses/profile_time/training/ Revelio employer-year and sample status Same measures; in-sample and out-of-sample periods are separate rows
RCID-year match quality (full_training) rcid_year_mq.parquet in /data/<coverage>/analyses/full_training/ Revelio employer-year Key: rcid, year
Same MQ statistics and counts; no sample-status column

mq_n counts finite model score rows. n_relationships counts distinct (user_id, relationship_number) pairs within each RCID-year; relationship numbers are worker-specific and must not be counted alone. new_relationships uses the same relationship key and counts a relationship only when its relationship_start_date falls in the output year. In training mode, both relationship counts and MQ statistics are calculated within sample_status in the same output dataset; a firm-year with periods in both samples has two rows, one for each status.

For the worker profile, worker-level assignments generally mean each worker's periods remain on one side of the split, while a firm's workforce can contain workers from both sides. For the time profile, the same firm-year can also have periods on both sides of the time cutoff. In both cases, sample_status keeps the two score populations distinct. The full_training build remains a single profile-independent RCID-year aggregate and has no sample_status column.

The current full-training artifact covers calendar years 2008--2024. The builder does not impose a separate year filter: 2008 is the first scored source year, while 2024 follows the shared configured observation end date of 2024-12-31. Consumers that construct forward outcomes should form those outcomes before restricting their regression sample to this MQ window.

The additive MQ moments (mq_n, mq_sum, and mq_sum_sq) allow a later public-firm builder to aggregate several RCIDs into a GVKEY-year statistic without re-reading score shards. GVKEY, PERMNO, Compustat, and CRSP fields are intentionally absent so mapping changes never require this dataset to be rebuilt.

The builder processes one score shard at a time and writes a temporary partial aggregate before merging the partials into the final dataset. Score shards are user-bucketed, so each worker's relationship counts occur in one shard and can be summed at merge time. MQ sample variance and standard deviation are combined from each shard's count, mean, and variance. Temporary partials and DuckDB spill files are removed when the job exits.

The builder defaults DuckDB's memory limit to 128GB for both direct and Slurm runs. The Slurm entry point requests 128G. DuckDB can spill intermediate work to the configured temporary directory. Set HCT_RCID_YEAR_MEMORY_LIMIT to override the DuckDB cap. HCT_RCID_YEAR_THREADS can override the thread count; otherwise the builder uses SLURM_CPUS_PER_TASK, capped at eight for direct runs.

Command reference
Option Required / default Source-defined effect
--config PATH config/config_one_percent.yaml Analyses YAML config.
--mode training\|full_training full_training Selects profile-specific training or shared full-training scores.
--profile worker\|time Required with --mode training; omit for full_training Selects the training split and output path.
--overwrite Off Rebuilds an existing output.
Option Required / default Source-defined effect
--task build-rcid-year-mq Required CPU task.
--config CONFIG.yaml, --mode, --profile, --overwrite Config required; mode full_training; profile required for training; overwrite off Passed to the pipeline.
--partition, --time, --mem, --cpus, --job-name, --dependency, --exclude, --dry-run Wrapper defaults Standard Slurm overrides. The default requests 8 CPUs, 128 GB, and 24 hours. DuckDB is capped at 128 GB by default and spills to temporary storage as needed.
HCT_RCID_YEAR_MEMORY_LIMIT 128GB in direct and Slurm runs Sets DuckDB's memory cap.
HCT_RCID_YEAR_THREADS Slurm CPU allocation; otherwise up to 8 Overrides DuckDB's thread count.
cd /hct-tower/analyses
Rscript --vanilla pipelines/build_rcid_year_mq.R \
  --config config/config_full.yaml --mode training --profile worker
Rscript --vanilla pipelines/build_rcid_year_mq.R \
  --config config/config_full.yaml --mode training --profile time
Rscript --vanilla pipelines/build_rcid_year_mq.R \
  --config config/config_full.yaml --mode full_training

For a direct run, the DuckDB cap defaults to 128GB. You can override it and the thread count in the environment, for example HCT_RCID_YEAR_MEMORY_LIMIT=64GB HCT_RCID_YEAR_THREADS=4 before the Rscript command.

cd /hct-tower
slurm/submit.sh --task build-rcid-year-mq \
  --config config_full.yaml --mode training --profile worker
slurm/submit.sh --task build-rcid-year-mq \
  --config config_full.yaml --mode training --profile time
slurm/submit.sh --task build-rcid-year-mq \
  --config config_full.yaml --mode full_training

To change Slurm's default DuckDB cap or memory request, set HCT_RCID_YEAR_MEMORY_LIMIT and pass --mem separately; for example, HCT_RCID_YEAR_MEMORY_LIMIT=64GB slurm/submit.sh --task build-rcid-year-mq --mem 64G ....

Pass --overwrite only to intentionally replace an existing output. The builder writes rcid_year_mq_metadata.yaml beside the Parquet file with the source score manifest, aggregation definitions, resource settings, and row counts. Logs show preflight status, each shard's start and completion, partial-group and score counts, elapsed time and estimated time remaining, followed by partial-merge and publication summaries.