Skip to content

Employment build workflow

This is the first project build stage. It creates the canonical employment products used by model training, external mappings, and employment-based analysis.

Runtime

Run direct Python commands from /hct-tower in the model CUDA image (hct-tower-model-cuda-…). The Slurm commands use the corresponding model CUDA Apptainer image. The model-period step runs from /hct-tower/model with PYTHONPATH=/hct-tower:/hct-tower/model.

  1. Build canonical user-sharded US Revelio position records from the vendor deliveries.

    cd /hct-tower
    python -m shared.data.build_revelio_us_positions \
      --repartition-users --overwrite
    

    No project Slurm wrapper exists for this vendor-ingestion step. Run it directly, or submit a reviewed custom job using the model CUDA image.

    Produces
    Dataset File / dataset directory Folder
    US employment-position records revelio_us_positions_users_education_user_*.parquet /data/full/raw/
  2. Build the full reusable employment products: events, workers, relationships, firm-year counts, and firms.

    The inclusive observation cutoff is set once in processed.observation_end_date (currently 2024-12-31) in the coverage config. Positions starting after it are excluded and later end dates are made missing, so open relationships are right-censored consistently before model-period construction.

    cd /hct-tower
    python -m shared.data.build_processed --config shared/config/config_full.yaml --target employees
    python -m shared.data.build_processed --config shared/config/config_full.yaml --target relationships
    python -m shared.data.build_processed --config shared/config/config_full.yaml --target company-year-counts
    python -m shared.data.build_processed --config shared/config/config_full.yaml --target companies
    python -m shared.data.validate_processed --config shared/config/config_full.yaml
    
    cd /hct-tower
    slurm/submit.sh --task build-processed --config config_full.yaml
    

    The wrapper runs the complete configured processed-data build; validate the outputs afterwards with the direct validation command above.

    Produces
    Dataset File / dataset directory Folder
    Employment events employment_events_processed/ /data/full/processed/
    Workers employees_processed.parquet /data/full/processed/
    Worker-firm relationships employment_relationships_processed/ /data/full/processed/
    Observed firm-year employment company_year_observed_counts.parquet /data/full/processed/
    Firms companies_processed.parquet /data/full/processed/
  3. Build company-description embeddings after the firm table exists.

    cd /hct-tower
    python -m shared.data.embed_about_text --config shared/config/config_full.yaml
    
    cd /hct-tower
    slurm/submit.sh --task embed-about --config config_full.yaml
    
    Produces
    Dataset File / dataset directory Folder
    Company-description embeddings about_emb.parquet /data/full/processed/
  4. Derive one_percent and samples only after the full products validate.

    cd /hct-tower
    python -m shared.data.build_processed_sample \
      --source-config shared/config/config_full.yaml \
      --config shared/config/config_one_percent.yaml --preset one_percent
    python -m shared.data.build_processed_sample \
      --source-config shared/config/config_full.yaml \
      --config shared/config/config_samples.yaml --preset samples
    
    cd /hct-tower
    slurm/submit.sh --task build-processed-sample --sample one_percent
    slurm/submit.sh --task build-processed-sample --sample samples
    
    Produces — one_percent and samples
    Dataset File / dataset directory Folder
    Employment events employment_events_processed/ /data/<coverage>/processed/
    Workers employees_processed.parquet /data/<coverage>/processed/
    Worker-firm relationships employment_relationships_processed/ /data/<coverage>/processed/
    Observed firm-year employment company_year_observed_counts.parquet /data/<coverage>/processed/
    Firms companies_processed.parquet /data/<coverage>/processed/
    Company-description embeddings about_emb.parquet /data/<coverage>/processed/
  5. Build model relationship periods for every coverage after its processed products and embeddings are ready.

    cd /hct-tower/model
    export PYTHONPATH=/hct-tower:/hct-tower/model
    python -m data.build_model_datasets --config config/config_full.yaml --no-progress
    python -m data.build_model_datasets --config config/config_one_percent.yaml --no-progress
    python -m data.build_model_datasets --config config/config_samples.yaml --no-progress
    
    cd /hct-tower
    slurm/submit.sh --task build-model-periods --config config_full.yaml
    slurm/submit.sh --task build-model-periods --config config_one_percent.yaml
    slurm/submit.sh --task build-model-periods --config config_samples.yaml
    
    Produces — full, one_percent, and samples
    Dataset File / dataset directory Folder
    Model relationship periods model_relationship_periods/ /data/<coverage>/processed/
    Model relationship-period metadata model_relationship_metadata.yaml /data/<coverage>/processed/
    Model relationship firm counts model_relationship_company_counts.parquet /data/<coverage>/processed/

Finalize processed.observation_end_date before step 2. The same value must be used by the full and sample coverage configs; changing it invalidates all later employment, model, scoring, and analysis products. The model configs also set processed.observation_start_date for left truncation of model relationship periods; changing it requires rebuilding model periods and all downstream model artifacts, but not the reusable processed employment products or embeddings.