Company-description embeddings

This dataset encodes cleaned company descriptions for the HCT company tower.

flowchart LR
    firms[(Firms with about text)] --> embed[Sentence-transformer<br/>encoding]
    embed --> output[(Company-description embeddings)]

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#e2f2e7,stroke:#3b7a4b
    class firms source
    class embed process
    class output data

Grain and key: one described company_id
Location: /data/<coverage>/processed/about_emb.parquet
Builder: shared/data/embed_about_text.py

The builder reads about_clean from the firm table and writes the identifier plus wide emb_* columns. Firms without usable text are omitted and receive zeros at model runtime. The current default encoder is sentence-transformers/all-MiniLM-L6-v2.

python -m shared.data.embed_about_text --config shared/config/config_full.yaml

On Slurm:

slurm/submit.sh --task embed-about --config config_full.yaml

Sampled coverages normally subset the full embedding file rather than re-encoding descriptions.