Skip to content

Compustat-CRSP MQ dataset

This reusable public-firm annual panel attaches Revelio MQ where available, monthly CRSP information as of the fiscal year-end, and filed and issued KPSS patent measures. The full_training version has one row for each processed Compustat (gvkey, fyear). A worker- or time-profile training version preserves each Compustat firm-year in both sample statuses, yielding one row per (gvkey, fyear, sample_status) so the MQ from each status remains separate. It is intentionally raw: analyses create ratios, logs, cumulative outcomes, winsorisation, and samples themselves. To keep analyses dependent on one reusable file, it also carries coverage-aware raw patent leads and lags through t+5 and t-5.

flowchart TB
    comp[(Processed annual Compustat)]
    mapping[(RCID-GVKEY mapping)]
    mq[(RCID-year MQ)]
    ccm[(Project-raw CCM links)]
    crsp[(Processed monthly CRSP)]
    patents[(Patent firm-year by GVKEY)]
    output[(Compustat-CRSP MQ panel)]

    comp --> output
    mapping --> output
    mq --> output
    ccm --> output
    crsp --> output
    patents --> output

    classDef input fill:#e8eef7,stroke:#52739e,stroke-width:2px
    classDef analysis fill:#eee5f5,stroke:#76528c,stroke-width:2px
    class comp,mapping,mq,ccm,crsp,patents input
    class output analysis
Inputs
Dataset Location Role
Processed annual Compustat /data/full/processed/compustat_annual.parquet Panel spine; one row per (gvkey, fyear)
RCID-GVKEY mapping /data/full/processed/mapping_rcid_gvkey.parquet Preferred one-to-one public-firm mapping
RCID-year MQ (training) /data/<coverage>/analyses/profile_<profile>/training/rcid_year_mq.parquet Annual employer MQ moments by sample_status; build this before Compustat-CRSP MQ
RCID-year MQ (full_training) /data/<coverage>/analyses/full_training/rcid_year_mq.parquet Annual employer MQ moments
Raw CCM link table /data/full/raw/ccm_linktable.parquet Dated GVKEY-PERMNO links
Processed monthly CRSP /data/full/processed/crsp.parquet Returns and volatility as of fiscal year-end
Patent firm-year by GVKEY /data/full/processed/patent_firm_year.parquet Filed and issued patent measures
Build
Component Location
Builder analyses/modules/build_compustat_crsp_mq.R
Runtime hct-tower-analyses-cpu
Entry point analyses/pipelines/build_compustat_crsp_mq.R
Output
Dataset File / location Grain
Compustat-CRSP MQ panel (training, worker profile) compustat_crsp_mq.parquet in /data/<coverage>/analyses/profile_worker/training/ Unique (gvkey, fyear, sample_status)
Compustat-CRSP MQ panel (training, time profile) compustat_crsp_mq.parquet in /data/<coverage>/analyses/profile_time/training/ Unique (gvkey, fyear, sample_status)
Compustat-CRSP MQ panel (full_training) compustat_crsp_mq.parquet in /data/<coverage>/analyses/full_training/ Unique (gvkey, fyear)

Join rules

  • The original zero-padded Compustat GVKEY is retained. A normalized form is used internally to join the mapping and patent data.
  • MQ joins on the mapped RCID and fiscal year (rcid_year_mq.year = fyear); profile builds also join on sample_status. In training mode, the Compustat spine is crossed with in_sample and out_of_sample before joins. Thus every firm-year remains in both status rows; MQ is missing for a status with no scored relationships. Analyses using a profile output should filter the intended sample_status before estimating results.
  • CCM links use LINKTYPE in {LU, LC} and LINKPRIM in {P, C}. Valid links are evaluated at datadate; overlapping links use the latest start.
  • CRSP uses the latest monthly observation at or before datadate.
  • Patent fields retain both timing conventions: *_filed and *_issued, for the current fiscal year and exact t-5 through t+5 fiscal years. A missing GVKEY-year patent record is represented as zero activity only within the fully covered KPSS window. Outside that window it is missing, with matching *_available flags.
  • data.patent_last_complete_year records the final fully covered source year. It is currently 2024: the KPSS 2025 delivery has filing observations only through 2025-08-21, so 2025 cannot be treated as a full filing year.
cd /hct-tower/analyses
Rscript --vanilla pipelines/build_compustat_crsp_mq.R \
  --config config/config_full.yaml --mode training --profile worker
Rscript --vanilla pipelines/build_compustat_crsp_mq.R \
  --config config/config_full.yaml --mode training --profile time
Rscript --vanilla pipelines/build_compustat_crsp_mq.R \
  --config config/config_full.yaml --mode full_training
cd /hct-tower
slurm/submit.sh --task build-compustat-crsp-mq --mode training \
  --profile worker --config config_full.yaml
slurm/submit.sh --task build-compustat-crsp-mq --mode training \
  --profile time --config config_full.yaml
slurm/submit.sh --task build-compustat-crsp-mq --mode full_training \
  --config config_full.yaml