Skip to content

Project-raw Compustat and identifier links

This stage converts immutable WRDS Funda, CCM, and SEC Suite historical-link SAS deliveries into project-owned Parquet files. It selects only the variables declared in shared/config/config_full.yaml, keeping ingestion memory bounded without embedding a vendor-specific field list in code.

flowchart LR
    funda[(Compustat Funda SAS)] --> stage[Stream configured fields]
    ccm[(CCM link-table SAS)] --> stage
    wcik[(WRDS historical CIK--GVKEY SAS)] --> stage
    stage --> annual[(Project-raw annual Compustat)]
    stage --> links[(Project-raw CCM links)]
    stage --> ciklinks[(Project-raw historical CIK--GVKEY links)]

    classDef source fill:#e8eef7,stroke:#52739e,stroke-width:2px
    classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
    class funda,ccm,wcik source
    class annual,links,ciklinks support
Inputs
Dataset Location
Compustat and CRSP vendor extracts /data/compustat/funda.sas7bdat, /data/compustat/ccmxpf_linktable.sas7bdat, and /data/compustat/wciklink_gvkey.sas7bdat
Build
Component Location
Builder shared/data/build_raw_compustat.py
Runtime hct-tower-model-cuda-… (CPU execution)
Entry point python -m shared.data.build_raw_compustat --config shared/config/config_full.yaml
Output
Dataset File / location Grain Key and variables
Project-raw annual Compustat compustat_annual.parquet in /data/full/raw/ Vendor financial statement record Identifiers and reporting metadata; financial fields including at, che, sale, xrd, csho, and prcc_f
Project-raw CCM links ccm_linktable.parquet in /data/full/raw/ GVKEY--security link interval gvkey, lpermno, lpermco, link type/primary status, and link dates
Project-raw historical CIK--GVKEY links wciklink_gvkey.parquet in /data/full/raw/ CIK--GVKEY-source interval CIK, GVKEY, source and quality labels, SEC/link dates, company names, and filing-count diagnostics

The Funda output intentionally preserves all selected source records. The processed Compustat builder applies the industrial, domestic, standardized, consolidated USD record filter and chooses one record per firm fiscal year. The CCM output supersedes the legacy CSV extract and is used by the mapping and patent builders. The WRDS SEC Suite output preserves historical CIK--GVKEY evidence for the audited Revelio--Compustat mapping.

cd /hct-tower
python -m shared.data.build_raw_compustat \
  --config shared/config/config_full.yaml
slurm/submit.sh --task build-raw-compustat --config config_full.yaml