US employment-position records

This dataset joins the large Revelio position, worker, and education deliveries into deduplicated US employment-position shards.

flowchart LR
    positions[(Revelio positions)]
    workers[(Revelio workers)]
    education[(Revelio education)]
    build[Join, filter to US,<br/>and repartition workers]
    output[(US employment-position records)]

    positions --> build
    workers --> build
    education --> build
    build --> output

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#e2f2e7,stroke:#3b7a4b
    class positions,workers,education source
    class build process
    class output data

Grain and key: one employment position, position_id
Location: /data/full/raw/revelio_us_positions_users_education_user_*.parquet
Builder: shared/data/build_revelio_us_positions.py

The builder joins each position to worker and education attributes, keeps US positions, removes duplicate position_id values, and repartitions by user_id. All positions for one worker therefore remain in one output shard.

python -m shared.data.build_revelio_us_positions --memory-limit 32GB --threads 4
python -m shared.data.build_revelio_us_positions --repartition-users --overwrite
python -m shared.data.build_revelio_us_positions --inspect

Position-sharded intermediate files and an optional combined file may also be written, but the user-sharded files are the canonical input to the processed employment datasets.