Skip to content

Accounting and market build workflow

Each builder has one responsibility and is invoked directly. Paths and source field selections are defined in shared/config/config_full.yaml.

flowchart LR
    funda[(Funda + CCM + historical CIK--GVKEY SAS)] --> build_raw_comp[Build raw Compustat]
    build_raw_comp --> raw_comp[(Project-raw Compustat + identifier links)]
    msf[(CRSP MSF SAS)] --> build_raw_crsp[Build raw CRSP]
    stocknames[(CRSP Stocknames V2 SAS)] --> build_raw_crsp
    build_raw_crsp --> raw_crsp[(Project-raw CRSP monthly)]
    build_raw_crsp --> raw_names[(Project-raw CRSP security history)]
    raw_comp --> build_processed_comp[Build processed Compustat]
    raw_crsp --> build_processed_crsp[Build processed CRSP]
    build_processed_comp --> comp_out[(Annual Compustat)]
    build_processed_crsp --> crsp_out[(Monthly CRSP)]

    classDef source fill:#e8eef7,stroke:#52739e,stroke-width:2px
    classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
    classDef reusable fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
    class funda,msf,stocknames source
    class raw_comp,raw_crsp,raw_names support
    class comp_out,crsp_out reusable
Inputs
Dataset Location
Compustat and WRDS SEC Suite vendor SAS files /data/compustat/funda.sas7bdat, /data/compustat/ccmxpf_linktable.sas7bdat, /data/compustat/wciklink_gvkey.sas7bdat
CRSP vendor SAS files /data/compustat/msf_v2.sas7bdat, /data/compustat/stocknames_v2.sas7bdat
Build
Stage Direct command Slurm task
Stage Funda and identifier links python -m shared.data.build_raw_compustat --config shared/config/config_full.yaml build-raw-compustat
Stage CRSP MSF and security history python -m shared.data.build_raw_crsp --config shared/config/config_full.yaml build-raw-crsp
Process Compustat python -m shared.data.build_processed_compustat --config shared/config/config_full.yaml build-processed-compustat
Process CRSP python -m shared.data.build_processed_crsp --config shared/config/config_full.yaml build-processed-crsp
Output
Dataset Location Grain
Project-raw Compustat /data/full/raw/compustat_annual.parquet Vendor financial statement record
Project-raw CCM links /data/full/raw/ccm_linktable.parquet GVKEY-security interval
Project-raw historical CIK--GVKEY links /data/full/raw/wciklink_gvkey.parquet CIK-GVKEY-source interval
Project-raw CRSP monthly /data/full/raw/crsp_monthly.parquet Security-month
Project-raw CRSP security history /data/full/raw/crsp_stocknames.parquet Security-name interval
Processed Compustat /data/full/processed/compustat_annual.parquet Firm-fiscal year
Processed CRSP /data/full/processed/crsp.parquet Security-month

Run the raw stage before its corresponding processed stage. For example:

cd /hct-tower
python -m shared.data.build_raw_compustat --config shared/config/config_full.yaml
python -m shared.data.build_processed_compustat --config shared/config/config_full.yaml

The processed annual panel and project-raw CCM and historical CIK--GVKEY links support the audited Revelio--Compustat mapping; the processed CRSP panel is later aligned to each Compustat fiscal datadate.