Coverage variants¶

The full, one_percent, and samples coverages expose the same four reusable employment datasets and two model-specific datasets. The smaller coverages are reproducible worker-level subsets of the completed full products, not separate vendor samples.

Coverage	Worker selection	Construction
`full`	All eligible workers	Built from the Revelio vendor inputs
`one_percent`	1% of workers in each nonempty event shard by default	Derived from the full processed datasets
`samples`	10,000 workers allocated across event shards by default	Derived from the full processed datasets

Products in each coverage¶

Each coverage contains the four green reusable datasets and the two purple model-specific datasets below. Observed firm-year employment is an amber pipeline-support artifact that is also written for each coverage, but it is not one of the six principal downstream products.

flowchart TB
    subgraph reusable[Four reusable datasets]
        events[(Employment events)]
        workers[(Workers)]
        relationships[(Worker-firm relationships)]
        firms[(Firms)]
    end

    counts[(Observed firm-year employment)]

    subgraph model_ready[Two model-specific datasets]
        embeddings[(Company-description embeddings)]
        periods[(Model relationship periods)]
    end

    events --> relationships
    relationships -.->|coverage-specific counts| counts
    firms -->|aligned firm IDs| embeddings

    events --> periods
    workers --> periods
    relationships --> periods
    firms --> periods

    classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
    classDef reusable fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
    classDef model fill:#eee5f5,stroke:#76528c,stroke-width:2px
    class counts support
    class events,workers,relationships,firms reusable
    class embeddings,periods model

All seven stored products use the same filenames and schemas under /data/<coverage>/processed/. Model relationship periods also have a metadata file and a firm-count companion file; these are part of that model-period artifact rather than additional datasets in the diagram.

How a sampled coverage is built¶

The same flow applies to one_percent and samples. Only the worker-selection rule changes. Blue nodes are completed full-coverage inputs; green, amber, and purple nodes are outputs under /data/<coverage>/processed/.

flowchart TB
    subgraph full_inputs[Completed inputs under /data/full/processed]
        full_events[(Full employment events)]
        full_workers[(Full workers)]
        full_relationships[(Full worker-firm relationships)]
        full_firms[(Full firms)]
        full_embeddings[(Full company-description embeddings)]
    end

    subgraph sampled_reusable[Four reusable outputs]
        sampled_events[(Sampled employment events)]
        sampled_workers[(Sampled workers)]
        sampled_relationships[(Sampled worker-firm relationships)]
        sampled_firms[(Sampled firms)]
    end

    sampled_counts[(Sampled observed firm-year employment)]

    subgraph sampled_model[Two model-specific outputs]
        sampled_embeddings[(Sampled company-description embeddings)]
        sampled_periods[(Sampled model relationship periods)]
    end

    full_events -->|filter selected workers| sampled_events
    full_workers -->|retain selected workers| sampled_workers
    full_relationships -->|filter selected workers| sampled_relationships

    full_firms -->|retain referenced firms| sampled_firms
    sampled_events -->|observed firms| sampled_firms
    sampled_relationships -->|current and next firms| sampled_firms

    sampled_relationships -->|recompute| sampled_counts
    full_embeddings -->|retain referenced firms| sampled_embeddings

    sampled_events --> sampled_periods
    sampled_workers --> sampled_periods
    sampled_relationships --> sampled_periods
    sampled_firms --> sampled_periods

    classDef input fill:#e8eef7,stroke:#52739e,stroke-width:2px
    classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
    classDef reusable fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
    classDef model fill:#eee5f5,stroke:#76528c,stroke-width:2px
    class full_events,full_workers,full_relationships,full_firms,full_embeddings input
    class sampled_counts support
    class sampled_events,sampled_workers,sampled_relationships,sampled_firms reusable
    class sampled_embeddings,sampled_periods model

The stable hash selection uses user_id and a configurable seed. All events and relationships for a selected worker remain together, and source shard names are preserved. Sampled firms include firms referenced by employment events, relationships, or an observed next employer. The sampled firm table retains attributes—including observed_size_bucket—from the full firm table; the recomputed sampled firm-year counts do not replace those attributes.

Product	Location within `/data/<coverage>/processed/`	Sampled construction
Employment events	`employment_events_processed/`	Filter full event shards to selected workers
Workers	`employees_processed.parquet`	Retain the selected workers from the full worker table
Worker-firm relationships	`employment_relationships_processed/`	Filter full relationship shards to selected workers
Firms	`companies_processed.parquet`	Retain full-table firms referenced by sampled events, relationships, or next employers
Observed firm-year employment	`company_year_observed_counts.parquet`	Recompute counts from sampled relationships
Company-description embeddings	`about_emb.parquet`	Retain full embeddings for the sampled firm universe
Model relationship periods	`model_relationship_periods/`	Build from the four coverage-specific reusable datasets

Build sequence¶

Order	Builder	Products created
1	`shared.data.build_processed_sample`	Four reusable datasets, observed firm-year employment, and company-description embeddings
2	`data.build_model_datasets`	Model relationship periods and their metadata and firm-count companions

Prerequisites¶

Before building either sampled coverage, /data/full/processed/ must contain the completed full employment events, workers, worker-firm relationships, firms, and company-description embeddings. Full model relationship periods are not required.

Starting from the staged US employment-position records, the clean full build order is:

python -m shared.data.build_processed --config shared/config/config_full.yaml --target employees
python -m shared.data.build_processed --config shared/config/config_full.yaml --target relationships
python -m shared.data.build_processed --config shared/config/config_full.yaml --target company-year-counts
python -m shared.data.build_processed --config shared/config/config_full.yaml --target companies
python -m shared.data.embed_about_text --config shared/config/config_full.yaml

These commands create the four full reusable datasets, the support counts, and the full embeddings. See Employment data and the US employment-position records if the staged input has not yet been built.

To complete the full coverage's second model-specific dataset, run the model-period builder from model/:

python -m data.build_model_datasets --config config/config_full.yaml

Its Slurm equivalent is:

slurm/submit.sh --task build-model-periods --config config_full.yaml

Run the processed-sample and validation commands below from the repository root inside the model container. Run the direct model-period commands from model/ as indicated. The explicit seed records the current default and makes the selection intent clear.

Build `one_percent`¶

First create the four reusable datasets, observed firm-year employment, and company-description embeddings:

python -m shared.data.build_processed_sample \
  --source-config shared/config/config_full.yaml \
  --config shared/config/config_one_percent.yaml \
  --preset one_percent \
  --seed 42

Then create model relationship periods from those coverage-specific reusable datasets. Run the direct command from model/:

python -m data.build_model_datasets --config config/config_one_percent.yaml

Alternatively, submit only this second step through Slurm from the repository root:

slurm/submit.sh --task build-model-periods --config config_one_percent.yaml

Build `samples`¶

First create the four reusable datasets, observed firm-year employment, and company-description embeddings:

python -m shared.data.build_processed_sample \
  --source-config shared/config/config_full.yaml \
  --config shared/config/config_samples.yaml \
  --preset samples \
  --seed 42

Then run the model-period builder from model/:

python -m data.build_model_datasets --config config/config_samples.yaml

Or submit that second step through Slurm:

slurm/submit.sh --task build-model-periods --config config_samples.yaml

The submit wrapper does not currently expose the processed-sample builder as a Slurm task. Run the first step directly in the model container or in an interactive cluster session with access to /data. The sample builder recreates its configured outputs; do not run two builds for the same coverage at the same time.

Validate the sampled products¶

Validate each sampled coverage against the full processed products after the first build step:

python -m shared.data.validate_processed \
  --config shared/config/config_one_percent.yaml \
  --reference-root /data/full/processed

python -m shared.data.validate_processed \
  --config shared/config/config_samples.yaml \
  --reference-root /data/full/processed

The model-period builder validates its own inputs before writing periods and reports the output row count, split counts, metadata path, and companion firm-count path when it completes.