Employment data

The employment-data pipeline turns Revelio vendor files into the canonical worker, position, relationship, firm, and model-period datasets used throughout the project.

Full-coverage lineage

This diagram describes the full build. Blue nodes are vendor inputs, amber nodes are pipeline-support datasets, green nodes are reusable datasets, and purple nodes are model-specific derivatives.

flowchart TB
    positions[(Revelio positions)]
    users[(Revelio workers)]
    education[(Revelio education)]
    company_mapping[(Revelio company mapping)]

    us_records[(US employment-position records)]
    events[(Employment events)]
    workers[(Workers)]
    relationships[(Worker-firm relationships)]
    counts[(Observed firm-year employment)]
    firms[(Firms)]
    embeddings[(Company-description embeddings)]
    periods[(Model relationship periods)]

    positions --> us_records
    users --> us_records
    education --> us_records

    us_records --> events
    us_records --> workers

    events --> relationships
    relationships --> counts

    company_mapping --> firms
    events --> firms
    counts --> firms

    firms --> embeddings

    events --> periods
    workers --> periods
    relationships --> periods
    firms --> periods

    classDef input fill:#e8eef7,stroke:#52739e,stroke-width:2px
    classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
    classDef reusable fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
    classDef model fill:#eee5f5,stroke:#76528c,stroke-width:2px
    class positions,users,education,company_mapping input
    class us_records,counts support
    class events,workers,relationships,firms reusable
    class embeddings,periods model

For the full firm table, the Revelio company mapping supplies company attributes, employment events determine the observed firm universe, and observed firm-year employment supplies the size category used by the current configuration. Event firms missing from the mapping are retained as explicit orphan firms.

Output datasets

The pages below follow the dependency order shown in the diagram. Scope describes how a stored dataset is used: pipeline-support datasets exist to build another product, reusable datasets are consumed across project areas, and model-specific derivatives are built primarily for model training or scoring.

Dataset	Scope	Grain	Coverage	Location within /data/<coverage>/
US employment-position records	Pipeline support	Position	full only	raw/revelio_us_positions_users_education_user_*.parquet
Employment events	Reusable	Position	All	processed/employment_events_processed/
Workers	Reusable	Worker	All	processed/employees_processed.parquet
Worker-firm relationships	Reusable	Consecutive worker-firm spell	All	processed/employment_relationships_processed/
Observed firm-year employment	Pipeline support	Firm-year	All	processed/company_year_observed_counts.parquet
Firms	Reusable	Firm	All	processed/companies_processed.parquet
Company-description embeddings	Model-specific	Firm	All	processed/about_emb.parquet
Model relationship periods	Model-specific	Annual worker-firm risk-set period	All	processed/model_relationship_periods/

“All” means that the schema is available under full, one_percent, and samples. The datasets are not constructed identically across those coverages.

Coverage variants

The full employment events, workers, relationships, observed firm-year employment, firms, and company-description embeddings are built first. Two reproducible worker-level variants are then derived from those stored datasets. Model relationship periods are built separately within each coverage.

For both sampled coverages, employment events and relationships are filtered to selected workers; workers, firms, and company-description embeddings are subset from the full tables; and observed firm-year employment is recomputed from the sampled relationships. See Coverage variants for the dataset-by-dataset construction rules and commands.

Build order

Build the US employment-position records first. The canonical processed-data entry point is then shared/data/build_processed.py; for a clean build, use the dependency order below:

python -m shared.data.build_processed --config shared/config/config_full.yaml --target employees
python -m shared.data.build_processed --config shared/config/config_full.yaml --target relationships
python -m shared.data.build_processed --config shared/config/config_full.yaml --target company-year-counts
python -m shared.data.build_processed --config shared/config/config_full.yaml --target companies

The employees target writes both employment events and workers. Build company-description embeddings and model relationship periods after the core processed products are available.

Validate the core processed products with:

python -m shared.data.validate_processed --config shared/config/config_full.yaml