Employment build workflow¶
This is the first project build stage. It creates the canonical employment products used by model training, external mappings, and employment-based analysis.
Runtime
Run direct Python commands from /hct-tower in the model CUDA image
(hct-tower-model-cuda-…). The Slurm commands use the corresponding model
CUDA Apptainer image. The model-period step runs from /hct-tower/model
with PYTHONPATH=/hct-tower:/hct-tower/model.
-
Build canonical user-sharded US Revelio position records from the vendor deliveries.
Produces
Dataset File / dataset directory Folder US employment-position records revelio_us_positions_users_education_user_*.parquet/data/full/raw/ -
Build the full reusable employment products: events, workers, relationships, firm-year counts, and firms.
The inclusive observation cutoff is set once in
processed.observation_end_date(currently2024-12-31) in the coverage config. Positions starting after it are excluded and later end dates are made missing, so open relationships are right-censored consistently before model-period construction.cd /hct-tower python -m shared.data.build_processed --config shared/config/config_full.yaml --target employees python -m shared.data.build_processed --config shared/config/config_full.yaml --target relationships python -m shared.data.build_processed --config shared/config/config_full.yaml --target company-year-counts python -m shared.data.build_processed --config shared/config/config_full.yaml --target companies python -m shared.data.validate_processed --config shared/config/config_full.yamlProduces
Dataset File / dataset directory Folder Employment events employment_events_processed//data/full/processed/Workers employees_processed.parquet/data/full/processed/Worker-firm relationships employment_relationships_processed//data/full/processed/Observed firm-year employment company_year_observed_counts.parquet/data/full/processed/Firms companies_processed.parquet/data/full/processed/ -
Build company-description embeddings after the firm table exists.
Produces
Dataset File / dataset directory Folder Company-description embeddings about_emb.parquet/data/full/processed/ -
Derive
one_percentandsamplesonly after the full products validate.cd /hct-tower python -m shared.data.build_processed_sample \ --source-config shared/config/config_full.yaml \ --config shared/config/config_one_percent.yaml --preset one_percent python -m shared.data.build_processed_sample \ --source-config shared/config/config_full.yaml \ --config shared/config/config_samples.yaml --preset samplesProduces — one_percent and samples
Dataset File / dataset directory Folder Employment events employment_events_processed//data/<coverage>/processed/Workers employees_processed.parquet/data/<coverage>/processed/Worker-firm relationships employment_relationships_processed//data/<coverage>/processed/Observed firm-year employment company_year_observed_counts.parquet/data/<coverage>/processed/Firms companies_processed.parquet/data/<coverage>/processed/Company-description embeddings about_emb.parquet/data/<coverage>/processed/ -
Build model relationship periods for every coverage after its processed products and embeddings are ready.
cd /hct-tower/model export PYTHONPATH=/hct-tower:/hct-tower/model python -m data.build_model_datasets --config config/config_full.yaml --no-progress python -m data.build_model_datasets --config config/config_one_percent.yaml --no-progress python -m data.build_model_datasets --config config/config_samples.yaml --no-progressProduces — full, one_percent, and samples
Dataset File / dataset directory Folder Model relationship periods model_relationship_periods//data/<coverage>/processed/Model relationship-period metadata model_relationship_metadata.yaml/data/<coverage>/processed/Model relationship firm counts model_relationship_company_counts.parquet/data/<coverage>/processed/
Finalize processed.observation_end_date before step 2. The same value must be
used by the full and sample coverage configs; changing it invalidates all later
employment, model, scoring, and analysis products. The model configs also set
processed.observation_start_date for left truncation of model relationship
periods; changing it requires rebuilding model periods and all downstream model
artifacts, but not the reusable processed employment products or embeddings.