Skip to content

US employment-position records

This dataset joins the large Revelio position, worker, and education deliveries into deduplicated US employment-position shards.

flowchart LR
    positions[(Revelio positions)]
    workers[(Revelio workers)]
    education[(Revelio education)]
    build[Join, filter to US,<br/>and repartition workers]
    output[(US employment-position records)]

    positions --> build
    workers --> build
    education --> build
    build --> output

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#e2f2e7,stroke:#3b7a4b
    class positions,workers,education source
    class build process
    class output data
Inputs
Dataset Location
Revelio position, worker, and education deliveries Vendor Revelio deliveries under /data/full/raw/
Build
Component Location
Builder shared/data/build_revelio_us_positions.py
Runtime hct-tower-model-cuda-… (CPU execution)
Entry point python -m shared.data.build_revelio_us_positions
Output
Dataset File / location Grain Key and variables
US employment-position record shards revelio_us_positions_users_education_user_*.parquet in /data/full/raw/ Employment position Key: position_id
Variables include user_id, rcid, start_date, end_date, seniority, job_category, salary, highest_degree

The builder joins each position to worker and education attributes, keeps US positions, removes duplicate position_id values, and repartitions by user_id. All positions for one worker therefore remain in one output shard.

Command reference
Option Required / default Source-defined effect
--revelio-dir PATH, --output-dir PATH, --output-basename NAME Source defaults Override vendor input location and the project-raw output location/name.
--memory-limit SIZE, --threads N, --temp-directory PATH 32GB, 4, unset DuckDB execution and spill settings; 0 threads leaves DuckDB's default.
--overwrite Off Rebuild target Parquet output that already exists.
--repartition-users Off Repartition position-stage output into 20 user_id % 20 buckets.
--combine Off Combine user-stage shards into one Parquet file.
--inspect Off Print output diagnostics; cannot be combined with build flags.
--shard N Unset; repeatable Process only one or more vendor position shard indices (0–19).
Interface Availability
slurm/submit.sh task No task currently invokes shared.data.build_revelio_us_positions; run the direct command in an appropriate CPU environment.
python -m shared.data.build_revelio_us_positions --memory-limit 32GB --threads 4
python -m shared.data.build_revelio_us_positions --repartition-users --overwrite
python -m shared.data.build_revelio_us_positions --inspect

There is currently no project Slurm wrapper for this builder. Run the direct command in a CPU job using the model CUDA image.

Position-sharded intermediate files and an optional combined file may also be written, but the user-sharded files are the canonical input to the processed employment datasets.