Skip to content

Analysis-data build workflow

Run this after the required worker-firm-period score outputs exist. The first two steps form the dependency chain for the destination-prediction analysis; the later worker-firm, firm-year, executive, and deal branches can then run when their listed inputs are available.

Profile-specific score-derived inputs and outputs are defined under data.training_profiles.worker and data.training_profiles.time; the shared full-training paths remain under data.full_training. Some builders still use the older generic data.training paths and will be migrated separately.

Runtime

Direct R commands run from /hct-tower/analyses in the hct-tower-analyses-cpu image and always use Rscript --vanilla. Direct Python commands run from /hct-tower in the model CUDA image (hct-tower-model-cuda-…). Slurm commands select the corresponding image through their task wrapper.

  1. Build the RCID-year MQ dataset for each training profile you plan to use, then build the shared full-training version if needed. Profile builds aggregate in-sample and out-of-sample periods separately, so a firm-year can produce two rows. The full_training build remains one row per RCID-year.

    cd /hct-tower/analyses
    Rscript --vanilla pipelines/build_rcid_year_mq.R \
      --config config/config_full.yaml --mode training --profile worker
    Rscript --vanilla pipelines/build_rcid_year_mq.R \
      --config config/config_full.yaml --mode training --profile time
    Rscript --vanilla pipelines/build_rcid_year_mq.R \
      --config config/config_full.yaml --mode full_training
    
    cd /hct-tower
    slurm/submit.sh --task build-rcid-year-mq --mode training \
      --profile worker --config config_full.yaml
    slurm/submit.sh --task build-rcid-year-mq --mode training \
      --profile time --config config_full.yaml
    slurm/submit.sh --task build-rcid-year-mq --mode full_training \
      --config config_full.yaml
    
    Produces
    Dataset File / dataset directory Folder
    RCID-year MQ (full_training) rcid_year_mq.parquet /data/full/analyses/full_training/
    RCID-year MQ (training, worker profile) rcid_year_mq.parquet, keyed by RCID-year-sample status /data/full/analyses/profile_worker/training/
    RCID-year MQ (training, time profile) rcid_year_mq.parquet, keyed by RCID-year-sample status /data/full/analyses/profile_time/training/
    RCID-occupation-year MQ (full_training) rcid_occupation_year_mq.parquet /data/full/analyses/full_training/

    The same score shards can also be aggregated by occupation with:

    slurm/submit.sh --task build-rcid-occupation-year-mq \
      --config config_full.yaml --mode full_training
    

    This produces rcid_occupation_year_mq.parquet, keyed by rcid, job_category, and year.

  2. Export complete company and movers embedding tables for each needed mode, then build destination ranks. The rank scorer uses the processed prior-year company employment counts for its time-valid candidate pools; it does not depend on the RCID-year MQ aggregation.

    cd /hct-tower/model
    python -m scoring.score_destination_ranks \
      --config config/config_full.yaml --mode training
    python -m scoring.score_destination_ranks \
      --config config/config_full.yaml --mode full_training
    
    cd /hct-tower
    slurm/submit.sh --task score-destination-ranks \
      --mode training --config config_full.yaml
    slurm/submit.sh --task score-destination-ranks \
      --mode full_training --config config_full.yaml
    
    Produces
    Dataset File / dataset directory Folder
    Destination ranks (training) rank Parquet shards /data/full/analyses/training/destination_ranks/
    Destination ranks (full_training) rank Parquet shards /data/full/analyses/full_training/destination_ranks/
  3. Build the disaggregated period-level regression inputs for each scored profile. Build the separate full-training and worker-firm pair datasets only if those analyses are also needed; full training is profile-independent.

    cd /hct-tower/analyses
    Rscript --vanilla pipelines/build_disaggregated_regression_data.R \
      --config config/config_full.yaml --mode training --profile worker --overwrite
    Rscript --vanilla pipelines/build_disaggregated_regression_data.R \
      --config config/config_full.yaml --mode training --profile time --overwrite
    Rscript --vanilla pipelines/build_disaggregated_regression_data.R \
      --config config/config_full.yaml --mode full_training --overwrite
    Rscript --vanilla pipelines/build_worker_firm_regression_data.R \
      --config config/config_full.yaml --mode training --profile worker --overwrite
    Rscript --vanilla pipelines/build_worker_firm_regression_data.R \
      --config config/config_full.yaml --mode training --profile time --overwrite
    Rscript --vanilla pipelines/build_worker_firm_regression_data.R \
      --config config/config_full.yaml --mode full_training --overwrite
    
    cd /hct-tower
    slurm/submit.sh --task build-disaggregated-regression-data --mode training \
      --profile worker --config config_full.yaml --overwrite
    slurm/submit.sh --task build-disaggregated-regression-data --mode training \
      --profile time --config config_full.yaml --overwrite
    slurm/submit.sh --task build-disaggregated-regression-data --mode full_training \
      --config config_full.yaml --overwrite
    slurm/submit.sh --task build-worker-firm-regression-data --mode training \
      --profile worker --config config_full.yaml --overwrite
    slurm/submit.sh --task build-worker-firm-regression-data --mode training \
      --profile time --config config_full.yaml --overwrite
    slurm/submit.sh --task build-worker-firm-regression-data --mode full_training \
      --config config_full.yaml --overwrite
    
    Produces
    Dataset File / dataset directory Folder
    Disaggregated regression data (training, worker profile) score-named Parquet shards /data/full/analyses/profile_worker/training/disaggregated_regression_data/
    Disaggregated regression data (training, time profile) score-named Parquet shards /data/full/analyses/profile_time/training/disaggregated_regression_data/
    Disaggregated regression data (full_training) score-named Parquet shards /data/full/analyses/full_training/disaggregated_regression_data/
    Worker-firm regression data (full_training) score-named Parquet shards /data/full/analyses/full_training/worker_firm_regression_data/
    Worker-firm regression data (training, worker profile) score-named Parquet shards /data/full/analyses/profile_worker/training/worker_firm_regression_data/
    Worker-firm regression data (training, time profile) score-named Parquet shards /data/full/analyses/profile_time/training/worker_firm_regression_data/
  4. Build the Compustat-CRSP MQ dataset after its matching RCID-year MQ output, the RCID-GVKEY mapping, and patent panel exist. Build the RCID-year and RCID-occupation-year datasets above first so the firm-year products appear in dependency order. The Compustat builder reads RCID-year MQ; the occupation-year product is a separate reusable aggregation.

    ```bash cd /hct-tower/analyses Rscript --vanilla pipelines/build_compustat_crsp_mq.R \ --config config/config_full.yaml --mode training --profile worker Rscript --vanilla pipelines/build_compustat_crsp_mq.R \ --config config/config_full.yaml --mode training --profile time Rscript --vanilla pipelines/build_compustat_crsp_mq.R \ --config config/config_full.yaml --mode full_training

    ```

    ```bash cd /hct-tower slurm/submit.sh --task build-compustat-crsp-mq --mode training \ --profile worker --config config_full.yaml slurm/submit.sh --task build-compustat-crsp-mq --mode training \ --profile time --config config_full.yaml slurm/submit.sh --task build-compustat-crsp-mq --mode full_training \ --config config_full.yaml

    ```

    Produces
    Dataset File / dataset directory Folder
    Compustat-CRSP MQ (training, worker profile) compustat_crsp_mq.parquet, keyed by GVKEY-fiscal-year-sample status /data/full/analyses/profile_worker/training/
    Compustat-CRSP MQ (training, time profile) compustat_crsp_mq.parquet, keyed by GVKEY-fiscal-year-sample status /data/full/analyses/profile_time/training/
    Compustat-CRSP MQ (full_training) compustat_crsp_mq.parquet /data/full/analyses/full_training/

    Patent outcomes come from the GVKEY-year panel /data/full/processed/patent_firm_year.parquet. Firm-performance and innovation tables are built downstream of this dataset -- see Firm performance and Innovation.

  5. Build MQ-linked executive data only after matched executive pay and the appropriate score mode exist.

    cd /hct-tower/analyses
    Rscript --vanilla pipelines/disaggregated_execucomp.R \
      build_regression_table --config config/config_full.yaml \
      --mode full_training --overwrite
    
    cd /hct-tower
    slurm/submit.sh --task build-execucomp-regression-table \
      --mode full_training --config config_full.yaml --overwrite
    
    Produces
    Dataset File / dataset directory Folder
    ExecuComp regression data execucomp_regression_table.parquet /data/full/analyses/full_training/
  6. Build the deal-level product after company-year MQ is available.

    cd /hct-tower
    python -m shared.data.build_datasets.goodwill_target_mq
    

    There is currently no project Slurm wrapper for this builder. Run the direct command in a job that uses the model CUDA image.

    Produces
    Dataset / artifact File / dataset directory Folder
    Goodwill and target match quality goodwill_target_mq.parquet /data/full/raw/
    Goodwill-MQ build summary goodwill_target_mq_summary.json /hct-tower/shared/data/build_datasets/outputs/
    Revelio name-key cache revelio_name_key_index.parquet /hct-tower/shared/data/build_datasets/outputs/cache/

The worker-firm, firm-year, compensation and governance, and deals pages document each branch's exact inputs and any current limitations. Run analysis estimators only after their data branch has completed.