Skip to content

Workers

The worker table contains one canonical record for each worker observed in the processed employment events.

flowchart LR
    raw[(US employment-position records)] --> select[Select one record<br/>per worker]
    select --> workers[(Workers)]

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#e2f2e7,stroke:#3b7a4b
    class raw source
    class select process
    class workers data
Inputs
Dataset Location
US employment-position records /data/full/raw/revelio_us_positions_users_education_user_*.parquet
Build
Component Location
Builder shared/data/build_processed.py
Runtime hct-tower-model-cuda-… (CPU execution)
Entry point python -m shared.data.build_processed --target employees / build-processed
Output
Dataset File / location Grain Key and variables
Processed workers employees_processed.parquet in /data/<coverage>/processed/ Worker Key: user_id
Variables: highest_degree, sex_predicted, ethnicity_predicted, role_k150, user_location
Command reference
Option Required / default Source-defined effect
--config PATH Required Shared YAML config that supplies inputs and output paths.
--target all\|employees\|companies\|relationships\|company-year-counts all Selects the artifact group; employees builds workers and employment events.
Option Required / default Source-defined effect
--task build-processed Required Invokes the complete processed bundle (--target all); the wrapper does not expose --target.
--config CONFIG.yaml Required Resolves the shared config passed to the builder.
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run Wrapper defaults Standard CPU overrides; default is 8 CPUs, 128 GB, and 6 hours.
python -m shared.data.build_processed \
  --config shared/config/config_full.yaml --target employees
# The Slurm task builds the complete processed-employment bundle.
slurm/submit.sh --task build-processed --config config_full.yaml

Configured fields currently include highest_degree, sex_predicted, ethnicity_predicted, role_k150, and user_location. For each field, the builder keeps the first available value within the worker's ordered source records.

The worker table supplies static employee features to model preparation and worker-level characteristics to descriptive analyses. It does not define model splits; worker splits are assigned later from the model configuration.