Embedding tables¶
Embedding tables retain reusable company vectors and time-varying worker vectors from a trained model. Its company and worker targets are independent: export a company table once per checkpoint, then export any worker population without reading or recreating that saved company table.
flowchart LR
periods[(Model relationship periods)]
firms[(Processed firms)]
checkpoint[(Trained HCT checkpoint)]
firm_export[Select model-period firms<br/>and encode firm vectors]
mover_export[Select destination-positive periods<br/>and encode worker states]
all_worker_export[Encode every worker<br/>relationship-period state]
companies[(Company embeddings)]
movers[(Mover worker embeddings)]
all_workers[(All worker-period embeddings)]
firms --> firm_export
checkpoint --> firm_export
periods --> mover_export
firms --> mover_export
checkpoint --> mover_export
periods --> all_worker_export
firms --> all_worker_export
checkpoint --> all_worker_export
firm_export --> companies
mover_export --> movers
all_worker_export --> all_workers
classDef source fill:#e8eef7,stroke:#52739e
classDef process fill:#fff0cc,stroke:#a66b00
classDef data fill:#eee5f5,stroke:#76528c
class periods,firms,checkpoint source
class firm_export,mover_export,all_worker_export process
class companies,movers,all_workers data
Inputs
| Dataset | Location |
|---|---|
| Model relationship periods | /data/<coverage>/processed/model_relationship_periods/; defines the company export universe and supplies worker rows |
| Processed firms | /data/<coverage>/processed/companies_processed.parquet |
| Selected fitted checkpoint | For training, training_profiles.<profile>.scoring_checkpoints.training; for full_training, full_training_artifacts.scoring_checkpoint |
Build
| Component | Location |
|---|---|
| Builder | model/scoring/export_embedding_tables.py |
| Runtime | hct-tower-model-cuda-… (GPU task) |
| Entry point | python -m scoring.export_embedding_tables --target company\|workers / export-embedding-tables --target company\|workers |
Output
| Dataset | File / location | Grain | Key and variables |
|---|---|---|---|
| Company embeddings (worker profile) | /data/<coverage>/analyses/profile_worker/training/embeddings/company_embeddings.parquet |
One distinct incumbent or observed-destination company in model relationship periods | Key: company_idVariables: structured firm attributes, fixed-width raw embedding_int8, per-vector FP16 embedding_scale |
| Company embeddings (time profile) | /data/<coverage>/analyses/profile_time/training/embeddings/company_embeddings.parquet |
One distinct incumbent or observed-destination company in model relationship periods | Same company-vector fields, from the time-profile checkpoint |
Company embeddings (full_training) |
/data/<coverage>/analyses/full_training/embeddings/company_embeddings.parquet |
One distinct incumbent or observed-destination company in model relationship periods | Same company-vector fields, from the shared full-training checkpoint |
| Mover worker embeddings | Source-named shards in <embedding-root>/worker_embeddings_movers/ |
Worker relationship-period with an observed destination | Key: user_id, relationship_number, company_id, period_startVariables: incumbent and destination company IDs, relationship and destination dates, split/sample flags, raw embedding_int8, FP16 embedding_scale |
| All worker-period embeddings | Source-named shards in <embedding-root>/worker_embeddings_worker_firm_period/ |
Every worker relationship-period in the model-period input | Same worker-state fields; includes rows with and without an observed destination |
| Metadata | _metadata.yaml in the selected embeddings directory |
One embedding directory | Per-table checkpoint metadata, vector dimension, int8 quantization contract, FP32 MQ dimension weights, population-specific source-shard manifests, row counts, and truncation status |
For profile-aware training exports, the profile root is either
/data/<coverage>/analyses/profile_worker/ or
/data/<coverage>/analyses/profile_time/, selected with --profile worker or
--profile time. These outputs are separate even though they use the same
processed relationship-period shards. full_training has one shared output
root and does not accept --profile. In this table, <embedding-root> means
/data/<coverage>/analyses/profile_<profile>/training/embeddings/ for a
profile-aware training export and
/data/<coverage>/analyses/full_training/embeddings/ for full_training.
Company vectors are fixed for a checkpoint and are therefore stored once. The
company target scans the narrow company_id and destination_company_id
columns of every model-relationship-period shard, retains their distinct
non-null union, and exports vectors only for that universe. This includes
observed next employers even when they have no later incumbent period. It
validates that every selected ID is in the current model cache and processed
company table.
To avoid repeated static joins in destination and embedding analyses, the
company file also contains every structured processed-company attribute:
naics, industry2, rics_k50, rics_k200, rics_k400, hq_city,
hq_metro_area, hq_state, hq_country, hq_region,
observed_size_bucket, and is_orphan_company. about_clean is deliberately
excluded: it is unstructured text, may be large, and has already informed the
company-tower representation.
Worker vectors encode the worker's history and relationship context for the
selected relationship-period row. The worker target uses the fitted model and
its normal firm inputs while running inference, but it does not read
company_embeddings.parquet; the two output tables can be built in either
order. movers retains only destination-positive rows; worker_firm_period
retains every row in the model relationship-period input. This means one vector
per observed worker relationship-period, not one row per unique worker. The two
populations can coexist under the same embeddings directory. Each worker output shard
follows its corresponding source shard, so the export can checkpoint and
resume after every completed shard. Worker-export logs include UTC timestamps,
total elapsed time, the current source shard, vectors written, and a rough
remaining-time estimate based on completed source shards. The estimate is
deliberately shard-based—mover counts vary by shard—so it is directional rather
than a promise. Smoke samples report that their --max-workers cap was reached
instead of estimating a full export.
The output directory depends only on coverage and scoring mode. Checkpoint
identity belongs in metadata, never in the directory name. Vectors are always
stored as raw, unweighted, per-vector symmetric int8 quantizations. For a row,
embedding_scale * embedding_int8 approximately reconstructs its raw vector;
the scale is FP16 and the vector is a fixed-width signed-int8 list. All-zero
vectors use scale one. This representation is for compact archival storage,
not direct integer similarity arithmetic.
Metadata stores the checkpoint's positive FP32 match_score_dimension_weights
vector. To reproduce MQ approximately, cast int8 vectors to FP16 or FP32,
multiply dimensions by sqrt(match_score_dimension_weights), normalize each
vector, then take the dot product. Per-vector quantization scales cancel under
this weighted cosine normalization, but are retained for approximate raw-space
reconstruction and embedding-geometry analysis. The one-percent int8 check
found mean absolute weighted-cosine error around 5.2e-4 over representative
destination candidates, with 99.2% top-1 agreement.
Company IDs are the durable external join keys. The exporter does not retain
model-internal company_idx or destination_idx; consumers load the static
company table once and construct an in-memory company-ID-to-vector lookup. The
metadata records the period-source path, incumbent-plus-destination selection
rule, retained attributes, and the exclusion of about_clean.
The worker metadata keeps observed destinations and dates so an ex-post
destination analysis can sample candidates on demand without joining the large
relationship tables. It does not retain sampled candidate sets, so the saved
destination-rank outputs remain the canonical record of an earlier exact
candidate draw.
--max-workers N writes a source-order smoke sample and records
truncated: true in metadata. It is useful to validate a setup but is not a
general destination-evaluation sample. For an interrupted full export, rerun
the identical command with --resume; do not combine --resume and
--overwrite.
Command reference
| Option | Required / default | Source-defined effect |
|---|---|---|
--config PATH |
Required | Model YAML config; determines the data coverage and checkpoint path. |
--mode training\|full_training |
Required | Selects held-out profile training or the shared full-training checkpoint. |
--profile worker\|time |
Required with --mode training; omit for full_training |
Selects the split profile, checkpoint, and embedding output root. |
--target company |
Required | Selects the static company-vector table. |
--overwrite |
Off | Replaces only company_embeddings.parquet; worker-population output is preserved. Required to replace an archive built under an earlier company-universe contract. |
--show-setup-logs |
Off | Shows verbose company metadata-loading logs. |
| Option | Required / default | Source-defined effect |
|---|---|---|
--config PATH |
Required | Model YAML config; determines the data coverage and checkpoint paths. |
--mode training\|full_training |
Required | Selects held-out profile training or the shared full-training checkpoint. |
--profile worker\|time |
Required with --mode training; omit for full_training |
Selects the split profile, checkpoint, and embedding output root. |
--target workers |
Required | Selects one time-varying worker-vector population. |
--population movers\|worker_firm_period |
movers |
Exports destination-positive rows or every model relationship-period row, respectively. The corresponding directory is worker_embeddings_<population>/. |
--max-workers N |
Unset | Stops after at most N selected worker rows in source-shard order and marks metadata as truncated. N must be positive. |
--overwrite |
Off | Replaces only the requested worker-population directory; the company table and other worker populations are preserved. |
--resume |
Off | Continues a compatible incomplete export; cannot be combined with --overwrite. |
--show-setup-logs |
Off | Shows verbose company and employee metadata-loading logs. |
| Option | Required / default | Source-defined effect |
|---|---|---|
--task export-embedding-tables |
Required | Selects slurm/model/export_embedding_tables.slurm. |
--config CONFIG.yaml |
Required | Resolves a model config under /hct-tower/model/config/ unless an in-container or repository-relative path is supplied. |
--mode training\|full_training |
Required | Passed to the direct exporter. |
--profile worker\|time |
Required with --mode training; omit for full_training |
Selects the profile-specific checkpoint and output root. |
--target company\|workers |
workers |
Passed to the direct exporter. company writes the static table; workers writes one population. |
--population movers\|worker_firm_period |
movers, --target workers only |
Passed to the worker target. |
--resume / --overwrite |
Off | Passed to the direct exporter. --resume is valid only for the worker target; the two flags remain mutually exclusive. |
--partition, --time, --mem, --cpus, --gpus, --constraint, --exclude, --job-name, --dependency, --dry-run |
Wrapper defaults | Standard Slurm resource/submission overrides. Default request: one GPU, 8 CPUs, 64 GB, 24 hours. |
--repo-root, --data-root, --artifact-root, --container-root, --sif |
Wrapper defaults | Override host paths or the Apptainer image used by the job. |
Company embeddings (run once for each checkpoint whose company vectors you need):
cd /hct-tower/model
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile worker --mode training --target company
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile time --mode training --target company
python -m scoring.export_embedding_tables --config config/config_full.yaml --mode full_training --target company
Mover-only worker embeddings (destination-positive relationship-periods):
cd /hct-tower/model
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile worker --mode training --target workers --population movers
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile time --mode training --target workers --population movers
python -m scoring.export_embedding_tables --config config/config_full.yaml --mode full_training --target workers --population movers
All worker relationship-period embeddings (every row in the model-period input):
cd /hct-tower/model
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile worker --mode training --target workers --population worker_firm_period
python -m scoring.export_embedding_tables --config config/config_full.yaml --profile time --mode training --target workers --population worker_firm_period
python -m scoring.export_embedding_tables --config config/config_full.yaml --mode full_training --target workers --population worker_firm_period
Company embeddings (run once for each checkpoint whose company vectors you need):
cd /hct-tower
slurm/submit.sh --task export-embedding-tables --profile worker --mode training --target company --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile time --mode training --target company --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --mode full_training --target company --config config_full.yaml
Mover-only worker embeddings (destination-positive relationship-periods):
cd /hct-tower
slurm/submit.sh --task export-embedding-tables --profile worker --mode training --target workers --population movers --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile time --mode training --target workers --population movers --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --mode full_training --target workers --population movers --config config_full.yaml
All worker relationship-period embeddings (every row in the model-period input):
cd /hct-tower
slurm/submit.sh --task export-embedding-tables --profile worker --mode training --target workers --population worker_firm_period --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile time --mode training --target workers --population worker_firm_period --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --mode full_training --target workers --population worker_firm_period --config config_full.yaml
If the company tables for the required checkpoint are already complete, skip
the company commands. Run the worker-population groups only when you need those
outputs. --mode full_training selects the profile-independent full-training
checkpoint; --population worker_firm_period selects every worker-period row.
These options control different things.