Skip to content

Embedding tables

Embedding tables retain reusable company vectors and time-varying worker vectors from a trained model. Its company and worker targets are independent: export a company table once per checkpoint, then export any worker population without reading or recreating that saved company table.

flowchart LR
    periods[(Model relationship periods)]
    firms[(Processed firms)]
    checkpoint[(Trained HCT checkpoint)]
    firm_export[Select model-period firms<br/>and encode firm vectors]
    mover_export[Select destination-positive periods<br/>and encode worker states]
    all_worker_export[Encode every worker<br/>relationship-period state]
    companies[(Company embeddings)]
    movers[(Mover worker embeddings)]
    all_workers[(All worker-period embeddings)]

    firms --> firm_export
    checkpoint --> firm_export
    periods --> mover_export
    firms --> mover_export
    checkpoint --> mover_export
    periods --> all_worker_export
    firms --> all_worker_export
    checkpoint --> all_worker_export
    firm_export --> companies
    mover_export --> movers
    all_worker_export --> all_workers

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#eee5f5,stroke:#76528c
    class periods,firms,checkpoint source
    class firm_export,mover_export,all_worker_export process
    class companies,movers,all_workers data
Inputs
Dataset Location
Model relationship periods /data/<coverage>/processed/model_relationship_periods/; defines the company export universe and supplies worker rows
Processed firms /data/<coverage>/processed/companies_processed.parquet
Selected fitted checkpoint For training, training_profiles.<profile>.scoring_checkpoints.training; for full_training, full_training_artifacts.scoring_checkpoint
Build
Component Location
Builder model/scoring/export_embedding_tables.py
Runtime hct-tower-model-cuda-… (GPU task)
Entry point python -m scoring.export_embedding_tables --target company\|workers / export-embedding-tables --target company\|workers
Output
Dataset File / location Grain Key and variables
Company embeddings (worker profile) /data/<coverage>/analyses/profile_worker/training/embeddings/company_embeddings.parquet One distinct incumbent or observed-destination company in model relationship periods Key: company_id
Variables: structured firm attributes, fixed-width raw embedding_int8, per-vector FP16 embedding_scale
Company embeddings (time profile) /data/<coverage>/analyses/profile_time/training/embeddings/company_embeddings.parquet One distinct incumbent or observed-destination company in model relationship periods Same company-vector fields, from the time-profile checkpoint
Company embeddings (full_training) /data/<coverage>/analyses/full_training/embeddings/company_embeddings.parquet One distinct incumbent or observed-destination company in model relationship periods Same company-vector fields, from the shared full-training checkpoint
Mover worker embeddings Source-named shards in <embedding-root>/worker_embeddings_movers/ Worker relationship-period with an observed destination Key: user_id, relationship_number, company_id, period_start
Variables: incumbent and destination company IDs, relationship and destination dates, split/sample flags, raw embedding_int8, FP16 embedding_scale
All worker-period embeddings Source-named shards in <embedding-root>/worker_embeddings_worker_firm_period/ Every worker relationship-period in the model-period input Same worker-state fields; includes rows with and without an observed destination
Metadata _metadata.yaml in the selected embeddings directory One embedding directory Per-table checkpoint metadata, vector dimension, int8 quantization contract, FP32 MQ dimension weights, population-specific source-shard manifests, row counts, and truncation status

For profile-aware training exports, the profile root is either /data/<coverage>/analyses/profile_worker/ or /data/<coverage>/analyses/profile_time/, selected with --profile worker or --profile time. These outputs are separate even though they use the same processed relationship-period shards. full_training has one shared output root and does not accept --profile. In this table, <embedding-root> means /data/<coverage>/analyses/profile_<profile>/training/embeddings/ for a profile-aware training export and /data/<coverage>/analyses/full_training/embeddings/ for full_training.

Company vectors are fixed for a checkpoint and are therefore stored once. The company target scans the narrow company_id and destination_company_id columns of every model-relationship-period shard, retains their distinct non-null union, and exports vectors only for that universe. This includes observed next employers even when they have no later incumbent period. It validates that every selected ID is in the current model cache and processed company table.

To avoid repeated static joins in destination and embedding analyses, the company file also contains every structured processed-company attribute: naics, industry2, rics_k50, rics_k200, rics_k400, hq_city, hq_metro_area, hq_state, hq_country, hq_region, observed_size_bucket, and is_orphan_company. about_clean is deliberately excluded: it is unstructured text, may be large, and has already informed the company-tower representation.

Worker vectors encode the worker's history and relationship context for the selected relationship-period row. The worker target uses the fitted model and its normal firm inputs while running inference, but it does not read company_embeddings.parquet; the two output tables can be built in either order. movers retains only destination-positive rows; worker_firm_period retains every row in the model relationship-period input. This means one vector per observed worker relationship-period, not one row per unique worker. The two populations can coexist under the same embeddings directory. Each worker output shard follows its corresponding source shard, so the export can checkpoint and resume after every completed shard. Worker-export logs include UTC timestamps, total elapsed time, the current source shard, vectors written, and a rough remaining-time estimate based on completed source shards. The estimate is deliberately shard-based—mover counts vary by shard—so it is directional rather than a promise. Smoke samples report that their --max-workers cap was reached instead of estimating a full export.

The output directory depends only on coverage and scoring mode. Checkpoint identity belongs in metadata, never in the directory name. Vectors are always stored as raw, unweighted, per-vector symmetric int8 quantizations. For a row, embedding_scale * embedding_int8 approximately reconstructs its raw vector; the scale is FP16 and the vector is a fixed-width signed-int8 list. All-zero vectors use scale one. This representation is for compact archival storage, not direct integer similarity arithmetic.

Metadata stores the checkpoint's positive FP32 match_score_dimension_weights vector. To reproduce MQ approximately, cast int8 vectors to FP16 or FP32, multiply dimensions by sqrt(match_score_dimension_weights), normalize each vector, then take the dot product. Per-vector quantization scales cancel under this weighted cosine normalization, but are retained for approximate raw-space reconstruction and embedding-geometry analysis. The one-percent int8 check found mean absolute weighted-cosine error around 5.2e-4 over representative destination candidates, with 99.2% top-1 agreement.

Company IDs are the durable external join keys. The exporter does not retain model-internal company_idx or destination_idx; consumers load the static company table once and construct an in-memory company-ID-to-vector lookup. The metadata records the period-source path, incumbent-plus-destination selection rule, retained attributes, and the exclusion of about_clean. The worker metadata keeps observed destinations and dates so an ex-post destination analysis can sample candidates on demand without joining the large relationship tables. It does not retain sampled candidate sets, so the saved destination-rank outputs remain the canonical record of an earlier exact candidate draw.

--max-workers N writes a source-order smoke sample and records truncated: true in metadata. It is useful to validate a setup but is not a general destination-evaluation sample. For an interrupted full export, rerun the identical command with --resume; do not combine --resume and --overwrite.

Command reference
Option Required / default Source-defined effect
--config PATH Required Model YAML config; determines the data coverage and checkpoint path.
--mode training\|full_training Required Selects held-out profile training or the shared full-training checkpoint.
--profile worker\|time Required with --mode training; omit for full_training Selects the split profile, checkpoint, and embedding output root.
--target company Required Selects the static company-vector table.
--overwrite Off Replaces only company_embeddings.parquet; worker-population output is preserved. Required to replace an archive built under an earlier company-universe contract.
--show-setup-logs Off Shows verbose company metadata-loading logs.
Option Required / default Source-defined effect
--config PATH Required Model YAML config; determines the data coverage and checkpoint paths.
--mode training\|full_training Required Selects held-out profile training or the shared full-training checkpoint.
--profile worker\|time Required with --mode training; omit for full_training Selects the split profile, checkpoint, and embedding output root.
--target workers Required Selects one time-varying worker-vector population.
--population movers\|worker_firm_period movers Exports destination-positive rows or every model relationship-period row, respectively. The corresponding directory is worker_embeddings_<population>/.
--max-workers N Unset Stops after at most N selected worker rows in source-shard order and marks metadata as truncated. N must be positive.
--overwrite Off Replaces only the requested worker-population directory; the company table and other worker populations are preserved.
--resume Off Continues a compatible incomplete export; cannot be combined with --overwrite.
--show-setup-logs Off Shows verbose company and employee metadata-loading logs.
Option Required / default Source-defined effect
--task export-embedding-tables Required Selects slurm/model/export_embedding_tables.slurm.
--config CONFIG.yaml Required Resolves a model config under /hct-tower/model/config/ unless an in-container or repository-relative path is supplied.
--mode training\|full_training Required Passed to the direct exporter.
--profile worker\|time Required with --mode training; omit for full_training Selects the profile-specific checkpoint and output root.
--target company\|workers workers Passed to the direct exporter. company writes the static table; workers writes one population.
--population movers\|worker_firm_period movers, --target workers only Passed to the worker target.
--resume / --overwrite Off Passed to the direct exporter. --resume is valid only for the worker target; the two flags remain mutually exclusive.
--partition, --time, --mem, --cpus, --gpus, --constraint, --exclude, --job-name, --dependency, --dry-run Wrapper defaults Standard Slurm resource/submission overrides. Default request: one GPU, 8 CPUs, 64 GB, 24 hours.
--repo-root, --data-root, --artifact-root, --container-root, --sif Wrapper defaults Override host paths or the Apptainer image used by the job.

Company embeddings (run once for each checkpoint whose company vectors you need):

cd /hct-tower/model
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile worker --mode training --target company
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile time --mode training --target company
python -m scoring.export_embedding_tables --config config/config_full.yaml --mode full_training --target company

Mover-only worker embeddings (destination-positive relationship-periods):

cd /hct-tower/model
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile worker --mode training --target workers --population movers
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile time --mode training --target workers --population movers
python -m scoring.export_embedding_tables --config config/config_full.yaml --mode full_training --target workers --population movers

All worker relationship-period embeddings (every row in the model-period input):

cd /hct-tower/model
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile worker --mode training --target workers --population worker_firm_period
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile time --mode training --target workers --population worker_firm_period
python -m scoring.export_embedding_tables --config config/config_full.yaml --mode full_training --target workers --population worker_firm_period

Company embeddings (run once for each checkpoint whose company vectors you need):

cd /hct-tower
slurm/submit.sh --task export-embedding-tables --profile worker --mode training --target company --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile time --mode training --target company --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --mode full_training --target company --config config_full.yaml

Mover-only worker embeddings (destination-positive relationship-periods):

cd /hct-tower
slurm/submit.sh --task export-embedding-tables --profile worker --mode training --target workers --population movers --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile time --mode training --target workers --population movers --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --mode full_training --target workers --population movers --config config_full.yaml

All worker relationship-period embeddings (every row in the model-period input):

cd /hct-tower
slurm/submit.sh --task export-embedding-tables --profile worker --mode training --target workers --population worker_firm_period --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile time --mode training --target workers --population worker_firm_period --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --mode full_training --target workers --population worker_firm_period --config config_full.yaml

If the company tables for the required checkpoint are already complete, skip the company commands. Run the worker-population groups only when you need those outputs. --mode full_training selects the profile-independent full-training checkpoint; --population worker_firm_period selects every worker-period row. These options control different things.