RCID-year match quality¶
This reusable analysis dataset aggregates scored worker-firm relationship periods to the Revelio employer-year level. It retains every scored employer, including private and unmatched firms, and deliberately does not join Compustat, CRSP, or any firm mapping. Training-profile builds keep in-sample and out-of-sample periods separate, yielding one row per RCID-year-status.
flowchart LR
scores[(Profile-specific worker-firm-period scores)]
build[Aggregate MQ and<br/>relationship counts by sample status]
output[(One RCID-year MQ dataset<br/>profile rows include sample_status)]
scores --> build --> output
classDef source fill:#e8eef7,stroke:#52739e
classDef process fill:#fff0cc,stroke:#a66b00
classDef data fill:#e2f2e7,stroke:#3b7a4b
class scores source
class build process
class output data
Inputs
| Dataset | Location |
|---|---|
| Worker-firm-period scores | Training profile: /data/<coverage>/analyses/profile_<profile>/training/worker_firm_period/; full training: /data/<coverage>/analyses/full_training/worker_firm_period/ |
Build
| Component | Location |
|---|---|
| Builder | analyses/modules/build_rcid_year_mq.R |
| Runtime | hct-tower-analyses-cpu |
| Entry point | analyses/pipelines/build_rcid_year_mq.R |
Output
| Dataset | File / location | Grain | Key and variables |
|---|---|---|---|
RCID-year match quality (training, worker profile) |
rcid_year_mq.parquet in /data/<coverage>/analyses/profile_worker/training/ |
Revelio employer-year and sample status | Key: rcid, year, sample_statusMQ statistics: mq_n, mq_sum, mq_sum_sq, mq_mean, mq_variance, mq_sd, mq_min, mq_maxCounts: n_relationships, new_relationships |
RCID-year match quality (training, time profile) |
rcid_year_mq.parquet in /data/<coverage>/analyses/profile_time/training/ |
Revelio employer-year and sample status | Same measures; in-sample and out-of-sample periods are separate rows |
RCID-year match quality (full_training) |
rcid_year_mq.parquet in /data/<coverage>/analyses/full_training/ |
Revelio employer-year | Key: rcid, yearSame MQ statistics and counts; no sample-status column |
mq_n counts finite model score rows. n_relationships counts distinct
(user_id, relationship_number) pairs within each RCID-year; relationship
numbers are worker-specific and must not be counted alone.
new_relationships uses the same relationship key and counts a relationship
only when its relationship_start_date falls in the output year. In training
mode, both relationship counts and MQ statistics are calculated within
sample_status in the same output dataset; a firm-year with periods in both
samples has two rows, one for each status.
For the worker profile, worker-level assignments generally mean each worker's
periods remain on one side of the split, while a firm's workforce can contain
workers from both sides. For the time profile, the same firm-year can also have
periods on both sides of the time cutoff. In both cases, sample_status keeps
the two score populations distinct. The full_training build remains a single
profile-independent RCID-year aggregate and has no sample_status column.
The current full-training artifact covers calendar years 2008--2024. The builder does not impose a separate year filter: 2008 is the first scored source year, while 2024 follows the shared configured observation end date of 2024-12-31. Consumers that construct forward outcomes should form those outcomes before restricting their regression sample to this MQ window.
The additive MQ moments (mq_n, mq_sum, and mq_sum_sq) allow a later
public-firm builder to aggregate several RCIDs into a GVKEY-year statistic
without re-reading score shards. GVKEY, PERMNO, Compustat, and CRSP fields are
intentionally absent so mapping changes never require this dataset to be
rebuilt.
The builder processes one score shard at a time and writes a temporary partial aggregate before merging the partials into the final dataset. Score shards are user-bucketed, so each worker's relationship counts occur in one shard and can be summed at merge time. MQ sample variance and standard deviation are combined from each shard's count, mean, and variance. Temporary partials and DuckDB spill files are removed when the job exits.
The builder defaults DuckDB's memory limit to 128GB for both direct and
Slurm runs. The Slurm entry point requests 128G. DuckDB can spill intermediate
work to the configured temporary directory. Set HCT_RCID_YEAR_MEMORY_LIMIT
to override the DuckDB cap.
HCT_RCID_YEAR_THREADS can override the thread count; otherwise the builder
uses SLURM_CPUS_PER_TASK, capped at eight for direct runs.
Command reference
| Option | Required / default | Source-defined effect |
|---|---|---|
--config PATH |
config/config_one_percent.yaml |
Analyses YAML config. |
--mode training\|full_training |
full_training |
Selects profile-specific training or shared full-training scores. |
--profile worker\|time |
Required with --mode training; omit for full_training |
Selects the training split and output path. |
--overwrite |
Off | Rebuilds an existing output. |
| Option | Required / default | Source-defined effect |
|---|---|---|
--task build-rcid-year-mq |
Required | CPU task. |
--config CONFIG.yaml, --mode, --profile, --overwrite |
Config required; mode full_training; profile required for training; overwrite off |
Passed to the pipeline. |
--partition, --time, --mem, --cpus, --job-name, --dependency, --exclude, --dry-run |
Wrapper defaults | Standard Slurm overrides. The default requests 8 CPUs, 128 GB, and 24 hours. DuckDB is capped at 128 GB by default and spills to temporary storage as needed. |
HCT_RCID_YEAR_MEMORY_LIMIT |
128GB in direct and Slurm runs |
Sets DuckDB's memory cap. |
HCT_RCID_YEAR_THREADS |
Slurm CPU allocation; otherwise up to 8 | Overrides DuckDB's thread count. |
cd /hct-tower/analyses
Rscript --vanilla pipelines/build_rcid_year_mq.R \
--config config/config_full.yaml --mode training --profile worker
Rscript --vanilla pipelines/build_rcid_year_mq.R \
--config config/config_full.yaml --mode training --profile time
Rscript --vanilla pipelines/build_rcid_year_mq.R \
--config config/config_full.yaml --mode full_training
For a direct run, the DuckDB cap defaults to 128GB. You can override it and
the thread count in the environment, for example
HCT_RCID_YEAR_MEMORY_LIMIT=64GB HCT_RCID_YEAR_THREADS=4 before the Rscript
command.
cd /hct-tower
slurm/submit.sh --task build-rcid-year-mq \
--config config_full.yaml --mode training --profile worker
slurm/submit.sh --task build-rcid-year-mq \
--config config_full.yaml --mode training --profile time
slurm/submit.sh --task build-rcid-year-mq \
--config config_full.yaml --mode full_training
To change Slurm's default DuckDB cap or memory request, set
HCT_RCID_YEAR_MEMORY_LIMIT and pass --mem separately; for example,
HCT_RCID_YEAR_MEMORY_LIMIT=64GB slurm/submit.sh --task build-rcid-year-mq --mem 64G ....
Pass --overwrite only to intentionally replace an existing output. The
builder writes rcid_year_mq_metadata.yaml beside the Parquet file with the
source score manifest, aggregation definitions, resource settings, and row
counts. Logs show preflight status, each shard's start and completion,
partial-group and score counts, elapsed time and estimated time remaining,
followed by partial-merge and publication summaries.