Skip to content

XBRL target counterfactual MQ

This GPU scorer measures the pre-deal target workforce's counterfactual match quality with its mapped acquirer. It is deliberately separate from the clean XBRL deal-panel builder: the output is a compact, merge-ready deal-level file that later R analyses can join on xbrl_deal_target_id. Profile-aware training outputs retain one row per deal and sample_status, so those should be joined on both keys.

flowchart LR
    deals[(XBRL deals regression data<br/>selected mode/profile)]
    workers[(Worker-firm-period embeddings<br/>selected mode/profile)]
    firms[(Company embeddings + MQ weights<br/>selected mode/profile)]
    score[Select prior-year target workers<br/>matching the sample split, then score]
    output[(Counterfactual MQ by deal<br/>and sample status when training)]

    deals --> score
    workers --> score
    firms --> score
    score --> output

    classDef input fill:#e8eef7,stroke:#52739e,stroke-width:2px
    classDef process fill:#fff0d8,stroke:#b7791f,stroke-width:2px
    classDef output fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
    class deals,workers,firms input
    class score process
    class output output
Inputs
Dataset Location Role
XBRL deals regression data Full training: /data/<coverage>/analyses/full_training/xbrl_deals_regression_data.parquet; worker/time profile: /data/<coverage>/analyses/profile_<profile>/training/xbrl_deals_regression_data.parquet Deal key, mapped target/acquirer RCIDs, observed_mq_year; profile data also carries sample_status
Company embeddings Full training: /data/<coverage>/analyses/full_training/embeddings/company_embeddings.parquet; profile: /data/<coverage>/analyses/profile_<profile>/training/embeddings/company_embeddings.parquet Acquirer vectors from the selected mode/profile
Worker-firm-period embeddings Full training: /data/<coverage>/analyses/full_training/embeddings/worker_embeddings_worker_firm_period/; profile: /data/<coverage>/analyses/profile_<profile>/training/embeddings/worker_embeddings_worker_firm_period/ Target worker states; training archives retain the split label on each row
Embedding metadata Beside each embedding archive as _metadata.yaml Archive completion, checkpoint identity, mode, and MQ dimension weights
Build
Component Location Purpose
Builder model/scoring/score_xbrl_target_counterfactual_mq.py Filters worker embeddings with DuckDB, scores on CUDA, and aggregates exact deal summaries
Runtime hct-tower-model-cuda-… Required GPU Python environment
Slurm task score-xbrl-target-counterfactual-mq Runs the selected full-training mode or worker/time training profile on CUDA
Output
Dataset Location Grain Content
XBRL target counterfactual MQ Full training: /data/<coverage>/analyses/full_training/xbrl_target_counterfactual_mq.parquet; profile: /data/<coverage>/analyses/profile_<profile>/training/xbrl_target_counterfactual_mq.parquet Full training: one row per deal; profile: one row per deal and sample_status Deal key/RCIDs/year, status, target-worker count, and counterfactual MQ distribution summary
Scoring metadata Beside the Parquet as xbrl_target_counterfactual_mq_metadata.yaml One scoring run Source archive manifest, checkpoint, selection/scoring method, coverage, and runtime settings

Timing and scoring contract

The scorer supports full_training and profile-aware training modes. For training, select --profile worker or --profile time; the model config resolves that profile's checkpoint, deals panel, and embedding archive. The same config determines coverage, so config_one_percent.yaml and config_full.yaml run against their respective /data/<coverage>/… archives.

For each deal, observed_mq_year = xbrl_year - 1. The scorer selects the latest target-firm relationship-period embedding for each user_id during that calendar year. A person therefore contributes at most once to a deal's counterfactual workforce, even if multiple target employment relationships appear during the year. In training mode, each deal's in-sample or out-of-sample row is scored only against worker rows with the corresponding status: train and valid worker split labels map to in_sample, while test maps to out_of_sample. This prevents workers from crossing the selected profile's sample boundary. Full-training mode has one row per deal and uses the full worker archive.

The target worker vectors and acquirer firm vector are archived raw, unweighted int8 representations. The scorer casts them to FP16, multiplies each dimension by the square root of the stored positive MQ weight, L2 normalizes, and takes their dot product. Per-vector quantization scales cancel under this cosine normalization. Summary statistics are exact over the scored worker values: count, mean, sample SD, min/max, and p10/p25/p50/p75/p90.

The output preserves every row in the regression panel. Training-profile output preserves both sample-status rows where present. A non-scored row has a zero worker count and null summary values. counterfactual_mq_status records whether its target RCID, acquirer RCID, acquirer embedding, or prior-year target workforce was unavailable.

The worker files are user-sharded, so DuckDB must inspect each source shard to find target-firm rows. It projects and filters before returning only matching vectors to CUDA; the output itself remains small. The job is one atomic run, not resumable. Logs report startup validation, eligible-deal coverage, periodic selected-worker scoring progress, aggregation, and final coverage.

Command reference
Option Required / default Source-defined effect
--config PATH Required Model YAML configuration; determines coverage, checkpoint, embedding root, and sibling input/output paths.
--mode MODE full_training Selects shared full-training outputs or profile-aware training outputs.
--profile PROFILE Required with --mode training Selects worker or time; omit for full_training.
--overwrite Off Replaces the output Parquet and metadata.
--batch-size N 65536 Selected target-worker embeddings sent to CUDA per scoring batch. Must be positive.
Option Required / default Source-defined effect
--task score-xbrl-target-counterfactual-mq Required Selects slurm/model/score_xbrl_target_counterfactual_mq.slurm.
--mode MODE full_training Selects full_training or profile-aware training scoring.
--profile PROFILE Required with --mode training Selects the worker or time profile; omit with full_training.
--config CONFIG.yaml Required Resolves a model config under /hct-tower/model/config/ unless an in-container or repository-relative path is supplied.
--overwrite Off Passed to the direct scorer.
--counterfactual-mq-args ARGS Unset Extra direct scorer arguments, such as '--batch-size 32768'.
--partition, --time, --mem, --cpus, --gpus, --constraint, --exclude, --job-name, --dependency, --dry-run Wrapper defaults Standard Slurm overrides. Default request: one GPU, 8 CPUs, 64 GB, and 24 hours.
--repo-root, --data-root, --artifact-root, --container-root, --sif Wrapper defaults Override host paths or the Apptainer image.
cd /hct-tower/model
python -m scoring.score_xbrl_target_counterfactual_mq \
  --config config/config_one_percent.yaml --mode full_training
python -m scoring.score_xbrl_target_counterfactual_mq \
  --config config/config_full.yaml --profile worker --mode training
python -m scoring.score_xbrl_target_counterfactual_mq \
  --config config/config_full.yaml --profile time --mode training
cd /hct-tower
slurm/submit.sh --task score-xbrl-target-counterfactual-mq \
  --config config_one_percent.yaml --mode full_training
slurm/submit.sh --task score-xbrl-target-counterfactual-mq \
  --profile worker --mode training --config config_full.yaml
slurm/submit.sh --task score-xbrl-target-counterfactual-mq \
  --profile time --mode training --config config_full.yaml \
  --counterfactual-mq-args '--batch-size 32768'