Accounting and market build workflow¶
Each builder has one responsibility and is invoked directly. Paths and source
field selections are defined in shared/config/config_full.yaml.
flowchart LR
funda[(Funda + CCM + historical CIK--GVKEY SAS)] --> build_raw_comp[Build raw Compustat]
build_raw_comp --> raw_comp[(Project-raw Compustat + identifier links)]
msf[(CRSP MSF SAS)] --> build_raw_crsp[Build raw CRSP]
stocknames[(CRSP Stocknames V2 SAS)] --> build_raw_crsp
build_raw_crsp --> raw_crsp[(Project-raw CRSP monthly)]
build_raw_crsp --> raw_names[(Project-raw CRSP security history)]
raw_comp --> build_processed_comp[Build processed Compustat]
raw_crsp --> build_processed_crsp[Build processed CRSP]
build_processed_comp --> comp_out[(Annual Compustat)]
build_processed_crsp --> crsp_out[(Monthly CRSP)]
classDef source fill:#e8eef7,stroke:#52739e,stroke-width:2px
classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
classDef reusable fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
class funda,msf,stocknames source
class raw_comp,raw_crsp,raw_names support
class comp_out,crsp_out reusable
Inputs
| Dataset | Location |
|---|---|
| Compustat and WRDS SEC Suite vendor SAS files | /data/compustat/funda.sas7bdat, /data/compustat/ccmxpf_linktable.sas7bdat, /data/compustat/wciklink_gvkey.sas7bdat |
| CRSP vendor SAS files | /data/compustat/msf_v2.sas7bdat, /data/compustat/stocknames_v2.sas7bdat |
Build
| Stage | Direct command | Slurm task |
|---|---|---|
| Stage Funda and identifier links | python -m shared.data.build_raw_compustat --config shared/config/config_full.yaml |
build-raw-compustat |
| Stage CRSP MSF and security history | python -m shared.data.build_raw_crsp --config shared/config/config_full.yaml |
build-raw-crsp |
| Process Compustat | python -m shared.data.build_processed_compustat --config shared/config/config_full.yaml |
build-processed-compustat |
| Process CRSP | python -m shared.data.build_processed_crsp --config shared/config/config_full.yaml |
build-processed-crsp |
Output
| Dataset | Location | Grain |
|---|---|---|
| Project-raw Compustat | /data/full/raw/compustat_annual.parquet |
Vendor financial statement record |
| Project-raw CCM links | /data/full/raw/ccm_linktable.parquet |
GVKEY-security interval |
| Project-raw historical CIK--GVKEY links | /data/full/raw/wciklink_gvkey.parquet |
CIK-GVKEY-source interval |
| Project-raw CRSP monthly | /data/full/raw/crsp_monthly.parquet |
Security-month |
| Project-raw CRSP security history | /data/full/raw/crsp_stocknames.parquet |
Security-name interval |
| Processed Compustat | /data/full/processed/compustat_annual.parquet |
Firm-fiscal year |
| Processed CRSP | /data/full/processed/crsp.parquet |
Security-month |
Run the raw stage before its corresponding processed stage. For example:
cd /hct-tower
python -m shared.data.build_raw_compustat --config shared/config/config_full.yaml
python -m shared.data.build_processed_compustat --config shared/config/config_full.yaml
The processed annual panel and project-raw CCM and historical CIK--GVKEY links support the audited
Revelio--Compustat mapping; the processed CRSP panel is later aligned to each
Compustat fiscal datadate.