Analysis-data build workflow¶
Run this after the required worker-firm-period score outputs exist. The first two steps form the dependency chain for the destination-prediction analysis; the later worker-firm, firm-year, executive, and deal branches can then run when their listed inputs are available.
Profile-specific score-derived inputs and outputs are defined under
data.training_profiles.worker and data.training_profiles.time; the shared
full-training paths remain under data.full_training. Some builders still use
the older generic data.training paths and will be migrated separately.
Runtime
Direct R commands run from /hct-tower/analyses in the
hct-tower-analyses-cpu image and always use Rscript --vanilla.
Direct Python commands run from /hct-tower in the model CUDA image
(hct-tower-model-cuda-…). Slurm commands select the corresponding image
through their task wrapper.
-
Build the RCID-year MQ dataset for each training profile you plan to use, then build the shared full-training version if needed. Profile builds aggregate in-sample and out-of-sample periods separately, so a firm-year can produce two rows. The
full_trainingbuild remains one row per RCID-year.cd /hct-tower/analyses Rscript --vanilla pipelines/build_rcid_year_mq.R \ --config config/config_full.yaml --mode training --profile worker Rscript --vanilla pipelines/build_rcid_year_mq.R \ --config config/config_full.yaml --mode training --profile time Rscript --vanilla pipelines/build_rcid_year_mq.R \ --config config/config_full.yaml --mode full_trainingcd /hct-tower slurm/submit.sh --task build-rcid-year-mq --mode training \ --profile worker --config config_full.yaml slurm/submit.sh --task build-rcid-year-mq --mode training \ --profile time --config config_full.yaml slurm/submit.sh --task build-rcid-year-mq --mode full_training \ --config config_full.yamlProduces
Dataset File / dataset directory Folder RCID-year MQ ( full_training)rcid_year_mq.parquet/data/full/analyses/full_training/RCID-year MQ ( training, worker profile)rcid_year_mq.parquet, keyed by RCID-year-sample status/data/full/analyses/profile_worker/training/RCID-year MQ ( training, time profile)rcid_year_mq.parquet, keyed by RCID-year-sample status/data/full/analyses/profile_time/training/RCID-occupation-year MQ ( full_training)rcid_occupation_year_mq.parquet/data/full/analyses/full_training/The same score shards can also be aggregated by occupation with:
slurm/submit.sh --task build-rcid-occupation-year-mq \ --config config_full.yaml --mode full_trainingThis produces
rcid_occupation_year_mq.parquet, keyed byrcid,job_category, andyear. -
Export complete company and movers embedding tables for each needed mode, then build destination ranks. The rank scorer uses the processed prior-year company employment counts for its time-valid candidate pools; it does not depend on the RCID-year MQ aggregation.
Produces
Dataset File / dataset directory Folder Destination ranks ( training)rank Parquet shards /data/full/analyses/training/destination_ranks/Destination ranks ( full_training)rank Parquet shards /data/full/analyses/full_training/destination_ranks/ -
Build the disaggregated period-level regression inputs for each scored profile. Build the separate full-training and worker-firm pair datasets only if those analyses are also needed; full training is profile-independent.
cd /hct-tower/analyses Rscript --vanilla pipelines/build_disaggregated_regression_data.R \ --config config/config_full.yaml --mode training --profile worker --overwrite Rscript --vanilla pipelines/build_disaggregated_regression_data.R \ --config config/config_full.yaml --mode training --profile time --overwrite Rscript --vanilla pipelines/build_disaggregated_regression_data.R \ --config config/config_full.yaml --mode full_training --overwrite Rscript --vanilla pipelines/build_worker_firm_regression_data.R \ --config config/config_full.yaml --mode training --profile worker --overwrite Rscript --vanilla pipelines/build_worker_firm_regression_data.R \ --config config/config_full.yaml --mode training --profile time --overwrite Rscript --vanilla pipelines/build_worker_firm_regression_data.R \ --config config/config_full.yaml --mode full_training --overwritecd /hct-tower slurm/submit.sh --task build-disaggregated-regression-data --mode training \ --profile worker --config config_full.yaml --overwrite slurm/submit.sh --task build-disaggregated-regression-data --mode training \ --profile time --config config_full.yaml --overwrite slurm/submit.sh --task build-disaggregated-regression-data --mode full_training \ --config config_full.yaml --overwrite slurm/submit.sh --task build-worker-firm-regression-data --mode training \ --profile worker --config config_full.yaml --overwrite slurm/submit.sh --task build-worker-firm-regression-data --mode training \ --profile time --config config_full.yaml --overwrite slurm/submit.sh --task build-worker-firm-regression-data --mode full_training \ --config config_full.yaml --overwriteProduces
Dataset File / dataset directory Folder Disaggregated regression data ( training, worker profile)score-named Parquet shards /data/full/analyses/profile_worker/training/disaggregated_regression_data/Disaggregated regression data ( training, time profile)score-named Parquet shards /data/full/analyses/profile_time/training/disaggregated_regression_data/Disaggregated regression data ( full_training)score-named Parquet shards /data/full/analyses/full_training/disaggregated_regression_data/Worker-firm regression data ( full_training)score-named Parquet shards /data/full/analyses/full_training/worker_firm_regression_data/Worker-firm regression data ( training, worker profile)score-named Parquet shards /data/full/analyses/profile_worker/training/worker_firm_regression_data/Worker-firm regression data ( training, time profile)score-named Parquet shards /data/full/analyses/profile_time/training/worker_firm_regression_data/ -
Build the Compustat-CRSP MQ dataset after its matching RCID-year MQ output, the RCID-GVKEY mapping, and patent panel exist. Build the RCID-year and RCID-occupation-year datasets above first so the firm-year products appear in dependency order. The Compustat builder reads RCID-year MQ; the occupation-year product is a separate reusable aggregation.
```bash cd /hct-tower/analyses Rscript --vanilla pipelines/build_compustat_crsp_mq.R \ --config config/config_full.yaml --mode training --profile worker Rscript --vanilla pipelines/build_compustat_crsp_mq.R \ --config config/config_full.yaml --mode training --profile time Rscript --vanilla pipelines/build_compustat_crsp_mq.R \ --config config/config_full.yaml --mode full_training
```
```bash cd /hct-tower slurm/submit.sh --task build-compustat-crsp-mq --mode training \ --profile worker --config config_full.yaml slurm/submit.sh --task build-compustat-crsp-mq --mode training \ --profile time --config config_full.yaml slurm/submit.sh --task build-compustat-crsp-mq --mode full_training \ --config config_full.yaml
```
Produces
Dataset File / dataset directory Folder Compustat-CRSP MQ ( training, worker profile)compustat_crsp_mq.parquet, keyed by GVKEY-fiscal-year-sample status/data/full/analyses/profile_worker/training/Compustat-CRSP MQ ( training, time profile)compustat_crsp_mq.parquet, keyed by GVKEY-fiscal-year-sample status/data/full/analyses/profile_time/training/Compustat-CRSP MQ ( full_training)compustat_crsp_mq.parquet/data/full/analyses/full_training/Patent outcomes come from the GVKEY-year panel
/data/full/processed/patent_firm_year.parquet. Firm-performance and innovation tables are built downstream of this dataset -- see Firm performance and Innovation. -
Build MQ-linked executive data only after matched executive pay and the appropriate score mode exist.
Produces
Dataset File / dataset directory Folder ExecuComp regression data execucomp_regression_table.parquet/data/full/analyses/full_training/ -
Build the deal-level product after company-year MQ is available.
Produces
Dataset / artifact File / dataset directory Folder Goodwill and target match quality goodwill_target_mq.parquet/data/full/raw/Goodwill-MQ build summary goodwill_target_mq_summary.json/hct-tower/shared/data/build_datasets/outputs/Revelio name-key cache revelio_name_key_index.parquet/hct-tower/shared/data/build_datasets/outputs/cache/
The worker-firm, firm-year, compensation and governance, and deals pages document each branch's exact inputs and any current limitations. Run analysis estimators only after their data branch has completed.