Skip to content

Observed firm-year employment

This dataset measures the number of distinct workers observed at each firm in each calendar year.

flowchart LR
    relationships[(Worker-firm relationships)] --> expand[Expand active years<br/>and count workers]
    expand --> counts[(Observed firm-year employment)]

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#e2f2e7,stroke:#3b7a4b
    class relationships source
    class expand process
    class counts data
Inputs
Dataset Location
Worker-firm relationships /data/<coverage>/processed/employment_relationships_processed/
Build
Component Location
Builder shared/data/build_processed.py
Runtime hct-tower-model-cuda-… (CPU execution)
Entry point python -m shared.data.build_processed --target company-year-counts / build-processed
Output
Dataset File / location Grain Key and variables
Observed firm-year employment company_year_observed_counts.parquet in /data/<coverage>/processed/ Firm-year Key: company_id, year
Variables: observed_active_users
Command reference
Option Required / default Source-defined effect
--config PATH Required Shared YAML config that supplies raw inputs and processed output paths.
--target all\|employees\|companies\|relationships\|company-year-counts all Selects the artifact group; use company-year-counts for this dataset.
Option Required / default Source-defined effect
--task build-processed Required Invokes the complete processed bundle (--target all); the wrapper does not expose --target.
--config CONFIG.yaml Required Resolves the shared config passed to the builder.
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run Wrapper defaults Standard CPU overrides; default is 8 CPUs, 128 GB, and 6 hours.
python -m shared.data.build_processed \
  --config shared/config/config_full.yaml --target company-year-counts
# The Slurm task builds the complete processed-employment bundle.
slurm/submit.sh --task build-processed --config config_full.yaml

Each relationship is expanded from its start year through its end year. An open-ended relationship is expanded through the latest year observed in the relationship data. The builder then counts distinct user_id values by firm and year as observed_active_users.

These are counts in Revelio, not true firm employment. Their main use is to construct the observed_size_bucket attached to the firm table.