Skip to content

Company-description embeddings

This dataset encodes cleaned company descriptions for the HCT company tower.

flowchart LR
    firms[(Firms with about text)] --> embed[Sentence-transformer<br/>encoding]
    embed --> output[(Company-description embeddings)]

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#e2f2e7,stroke:#3b7a4b
    class firms source
    class embed process
    class output data
Inputs
Dataset Location
Processed firms, including about_clean /data/<coverage>/processed/companies_processed.parquet
Build
Component Location
Builder shared/data/embed_about_text.py
Runtime hct-tower-model-cuda-… (GPU task)
Entry point python -m shared.data.embed_about_text / embed-about
Output
Dataset File / location Grain Key and variables
Company-description embeddings about_emb.parquet in /data/<coverage>/processed/ Described firm Key: company_id
Variables: emb_0 … emb_<D-1> (one float column per embedding dimension)

The builder reads about_clean from the firm table and writes the identifier plus wide emb_* columns. Firms without usable text are omitted and receive zeros at model runtime. The current default encoder is sentence-transformers/all-MiniLM-L6-v2.

Command reference
Option Required / default Source-defined effect
--config PATH Required Shared YAML config; supplies company input/output paths and the about_embeddings model, batch-size, and truncation settings.
Option Required / default Source-defined effect
--task embed-about Required Selects slurm/shared/embed_about_text.slurm.
--config CONFIG.yaml Required Resolves the shared config passed to the builder.
--partition, --time, --mem, --cpus, --gpus, --constraint, --exclude, --job-name, --dependency, --dry-run Wrapper defaults Standard overrides; default request is one GPU, 8 CPUs, 64 GB, and 4 hours.
--repo-root, --data-root, --artifact-root, --container-root, --sif Wrapper defaults Override host paths or the Apptainer image.
python -m shared.data.embed_about_text --config shared/config/config_full.yaml
slurm/submit.sh --task embed-about --config config_full.yaml

Sampled coverages normally subset the full embedding file rather than re-encoding descriptions.