US employment-position records¶
This dataset joins the large Revelio position, worker, and education deliveries into deduplicated US employment-position shards.
flowchart LR
positions[(Revelio positions)]
workers[(Revelio workers)]
education[(Revelio education)]
build[Join, filter to US,<br/>and repartition workers]
output[(US employment-position records)]
positions --> build
workers --> build
education --> build
build --> output
classDef source fill:#e8eef7,stroke:#52739e
classDef process fill:#fff0cc,stroke:#a66b00
classDef data fill:#e2f2e7,stroke:#3b7a4b
class positions,workers,education source
class build process
class output data
Inputs
| Dataset | Location |
|---|---|
| Revelio position, worker, and education deliveries | Vendor Revelio deliveries under /data/full/raw/ |
Build
| Component | Location |
|---|---|
| Builder | shared/data/build_revelio_us_positions.py |
| Runtime | hct-tower-model-cuda-… (CPU execution) |
| Entry point | python -m shared.data.build_revelio_us_positions |
Output
| Dataset | File / location | Grain | Key and variables |
|---|---|---|---|
| US employment-position record shards | revelio_us_positions_users_education_user_*.parquet in /data/full/raw/ |
Employment position | Key: position_idVariables include user_id, rcid, start_date, end_date, seniority, job_category, salary, highest_degree |
The builder joins each position to worker and education attributes, keeps US
positions, removes duplicate position_id values, and repartitions by
user_id. All positions for one worker therefore remain in one output shard.
Command reference
| Option | Required / default | Source-defined effect |
|---|---|---|
--revelio-dir PATH, --output-dir PATH, --output-basename NAME |
Source defaults | Override vendor input location and the project-raw output location/name. |
--memory-limit SIZE, --threads N, --temp-directory PATH |
32GB, 4, unset |
DuckDB execution and spill settings; 0 threads leaves DuckDB's default. |
--overwrite |
Off | Rebuild target Parquet output that already exists. |
--repartition-users |
Off | Repartition position-stage output into 20 user_id % 20 buckets. |
--combine |
Off | Combine user-stage shards into one Parquet file. |
--inspect |
Off | Print output diagnostics; cannot be combined with build flags. |
--shard N |
Unset; repeatable | Process only one or more vendor position shard indices (0–19). |
| Interface | Availability |
|---|---|
slurm/submit.sh task |
No task currently invokes shared.data.build_revelio_us_positions; run the direct command in an appropriate CPU environment. |
There is currently no project Slurm wrapper for this builder. Run the direct command in a CPU job using the model CUDA image.
Position-sharded intermediate files and an optional combined file may also be written, but the user-sharded files are the canonical input to the processed employment datasets.