Skip to content

Firms

The firm table combines Revelio company attributes with firms observed in the employment events.

The diagram describes the full build.

flowchart TB
    mapping[(Revelio company mapping)]
    events[(Employment events)]
    counts[(Observed firm-year employment)]
    build[Construct firm universe<br/>and attributes]
    firms[(Firms)]

    mapping --> build
    events --> build
    counts --> build
    build --> firms

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#e2f2e7,stroke:#3b7a4b
    class mapping,events,counts source
    class build process
    class firms data
Inputs
Dataset Location
Revelio company mapping Vendor Revelio company delivery under /data/full/raw/
Employment events /data/<coverage>/processed/employment_events_processed/
Observed firm-year employment /data/<coverage>/processed/company_year_observed_counts.parquet
Build
Component Location
Builder shared/data/build_processed.py / shared/data/build_processed_sample.py
Runtime hct-tower-model-cuda-… (CPU execution)
Entry point python -m shared.data.build_processed --target companies / build-processed
Output
Dataset File / location Grain Key and variables
Processed firms companies_processed.parquet in /data/<coverage>/processed/ Firm Key: company_id
Variables: naics, industry2, rics_k50, rics_k200, rics_k400, hq_city, hq_metro_area, hq_state, hq_country, hq_region, observed_size_bucket, about_clean, is_orphan_company
Command reference
Option Required / default Source-defined effect
python -m shared.data.build_processed --config PATH --target companies Required --config; target default is all Builds the full-coverage firm table. --target accepts all, employees, companies, relationships, and company-year-counts.
python -m shared.data.build_processed_sample --source-config PATH --config PATH --preset one_percent\|samples --config and --preset required; source config defaults to the full shared config Produces sampled processed outputs. Optional source-defined controls are --sample-user-count N, --one-percent-fraction F, and --seed N.
Option Required / default Source-defined effect
--task build-processed Required Builds the full processed bundle; accepts required --config.
--task build-processed-sample --sample one_percent\|samples --sample required; config defaults to the matching coverage config Runs the sample builder. --source-config PATH optionally overrides the full source config.
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run Wrapper defaults Standard CPU overrides. Full build default: 8 CPUs, 128 GB, 6 hours; sample build: 8 CPUs, 64 GB, 6 hours.
python -m shared.data.build_processed \
  --config shared/config/config_full.yaml --target companies
# The Slurm task builds the complete processed-employment bundle.
slurm/submit.sh --task build-processed --config config_full.yaml

For full coverage, the Revelio company mapping supplies company attributes and descriptions. Employment events restrict it to observed firms and identify orphan company IDs. Under the current canonical column contract, observed firm-year employment supplies the observed_size_bucket, based on active users observed through 2024. Firms present in employment events but absent from the vendor mapping are retained and marked with is_orphan_company = TRUE.

The one_percent and samples firm tables are subsets of the completed full firm table for firms referenced by sampled events, relationships, or observed next employers. They are not rebuilt directly from the vendor mapping or from sample-specific firm-year counts.

The table defines the company universe and firm-side categorical features. about_clean is separately embedded into the company-description embeddings.