XBRL target counterfactual MQ¶
This GPU scorer measures the pre-deal target workforce's counterfactual match
quality with its mapped acquirer. It is deliberately separate from the clean
XBRL deal-panel builder: the output is a compact, merge-ready deal-level file
that later R analyses can join on xbrl_deal_target_id. Profile-aware
training outputs retain one row per deal and sample_status, so those should
be joined on both keys.
flowchart LR
deals[(XBRL deals regression data<br/>selected mode/profile)]
workers[(Worker-firm-period embeddings<br/>selected mode/profile)]
firms[(Company embeddings + MQ weights<br/>selected mode/profile)]
score[Select prior-year target workers<br/>matching the sample split, then score]
output[(Counterfactual MQ by deal<br/>and sample status when training)]
deals --> score
workers --> score
firms --> score
score --> output
classDef input fill:#e8eef7,stroke:#52739e,stroke-width:2px
classDef process fill:#fff0d8,stroke:#b7791f,stroke-width:2px
classDef output fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
class deals,workers,firms input
class score process
class output output
Inputs
| Dataset | Location | Role |
|---|---|---|
| XBRL deals regression data | Full training: /data/<coverage>/analyses/full_training/xbrl_deals_regression_data.parquet; worker/time profile: /data/<coverage>/analyses/profile_<profile>/training/xbrl_deals_regression_data.parquet |
Deal key, mapped target/acquirer RCIDs, observed_mq_year; profile data also carries sample_status |
| Company embeddings | Full training: /data/<coverage>/analyses/full_training/embeddings/company_embeddings.parquet; profile: /data/<coverage>/analyses/profile_<profile>/training/embeddings/company_embeddings.parquet |
Acquirer vectors from the selected mode/profile |
| Worker-firm-period embeddings | Full training: /data/<coverage>/analyses/full_training/embeddings/worker_embeddings_worker_firm_period/; profile: /data/<coverage>/analyses/profile_<profile>/training/embeddings/worker_embeddings_worker_firm_period/ |
Target worker states; training archives retain the split label on each row |
| Embedding metadata | Beside each embedding archive as _metadata.yaml |
Archive completion, checkpoint identity, mode, and MQ dimension weights |
Build
| Component | Location | Purpose |
|---|---|---|
| Builder | model/scoring/score_xbrl_target_counterfactual_mq.py |
Filters worker embeddings with DuckDB, scores on CUDA, and aggregates exact deal summaries |
| Runtime | hct-tower-model-cuda-… |
Required GPU Python environment |
| Slurm task | score-xbrl-target-counterfactual-mq |
Runs the selected full-training mode or worker/time training profile on CUDA |
Output
| Dataset | Location | Grain | Content |
|---|---|---|---|
| XBRL target counterfactual MQ | Full training: /data/<coverage>/analyses/full_training/xbrl_target_counterfactual_mq.parquet; profile: /data/<coverage>/analyses/profile_<profile>/training/xbrl_target_counterfactual_mq.parquet |
Full training: one row per deal; profile: one row per deal and sample_status |
Deal key/RCIDs/year, status, target-worker count, and counterfactual MQ distribution summary |
| Scoring metadata | Beside the Parquet as xbrl_target_counterfactual_mq_metadata.yaml |
One scoring run | Source archive manifest, checkpoint, selection/scoring method, coverage, and runtime settings |
Timing and scoring contract¶
The scorer supports full_training and profile-aware training modes. For
training, select --profile worker or --profile time; the model config
resolves that profile's checkpoint, deals panel, and embedding archive. The
same config determines coverage, so config_one_percent.yaml and
config_full.yaml run against their respective /data/<coverage>/… archives.
For each deal, observed_mq_year = xbrl_year - 1. The scorer selects the
latest target-firm relationship-period embedding for each user_id during
that calendar year. A person therefore contributes at most once to a deal's
counterfactual workforce, even if multiple target employment relationships
appear during the year. In training mode, each deal's in-sample or
out-of-sample row is scored only against worker rows with the corresponding
status: train and valid worker split labels map to in_sample, while
test maps to out_of_sample. This prevents workers from crossing the
selected profile's sample boundary. Full-training mode has one row per deal
and uses the full worker archive.
The target worker vectors and acquirer firm vector are archived raw, unweighted int8 representations. The scorer casts them to FP16, multiplies each dimension by the square root of the stored positive MQ weight, L2 normalizes, and takes their dot product. Per-vector quantization scales cancel under this cosine normalization. Summary statistics are exact over the scored worker values: count, mean, sample SD, min/max, and p10/p25/p50/p75/p90.
The output preserves every row in the regression panel. Training-profile
output preserves both sample-status rows where present. A non-scored row has a
zero worker count and null summary values. counterfactual_mq_status records
whether its target RCID, acquirer RCID, acquirer embedding, or prior-year
target workforce was unavailable.
The worker files are user-sharded, so DuckDB must inspect each source shard to find target-firm rows. It projects and filters before returning only matching vectors to CUDA; the output itself remains small. The job is one atomic run, not resumable. Logs report startup validation, eligible-deal coverage, periodic selected-worker scoring progress, aggregation, and final coverage.
Command reference
| Option | Required / default | Source-defined effect |
|---|---|---|
--config PATH |
Required | Model YAML configuration; determines coverage, checkpoint, embedding root, and sibling input/output paths. |
--mode MODE |
full_training |
Selects shared full-training outputs or profile-aware training outputs. |
--profile PROFILE |
Required with --mode training |
Selects worker or time; omit for full_training. |
--overwrite |
Off | Replaces the output Parquet and metadata. |
--batch-size N |
65536 |
Selected target-worker embeddings sent to CUDA per scoring batch. Must be positive. |
| Option | Required / default | Source-defined effect |
|---|---|---|
--task score-xbrl-target-counterfactual-mq |
Required | Selects slurm/model/score_xbrl_target_counterfactual_mq.slurm. |
--mode MODE |
full_training |
Selects full_training or profile-aware training scoring. |
--profile PROFILE |
Required with --mode training |
Selects the worker or time profile; omit with full_training. |
--config CONFIG.yaml |
Required | Resolves a model config under /hct-tower/model/config/ unless an in-container or repository-relative path is supplied. |
--overwrite |
Off | Passed to the direct scorer. |
--counterfactual-mq-args ARGS |
Unset | Extra direct scorer arguments, such as '--batch-size 32768'. |
--partition, --time, --mem, --cpus, --gpus, --constraint, --exclude, --job-name, --dependency, --dry-run |
Wrapper defaults | Standard Slurm overrides. Default request: one GPU, 8 CPUs, 64 GB, and 24 hours. |
--repo-root, --data-root, --artifact-root, --container-root, --sif |
Wrapper defaults | Override host paths or the Apptainer image. |
cd /hct-tower/model
python -m scoring.score_xbrl_target_counterfactual_mq \
--config config/config_one_percent.yaml --mode full_training
python -m scoring.score_xbrl_target_counterfactual_mq \
--config config/config_full.yaml --profile worker --mode training
python -m scoring.score_xbrl_target_counterfactual_mq \
--config config/config_full.yaml --profile time --mode training
cd /hct-tower
slurm/submit.sh --task score-xbrl-target-counterfactual-mq \
--config config_one_percent.yaml --mode full_training
slurm/submit.sh --task score-xbrl-target-counterfactual-mq \
--profile worker --mode training --config config_full.yaml
slurm/submit.sh --task score-xbrl-target-counterfactual-mq \
--profile time --mode training --config config_full.yaml \
--counterfactual-mq-args '--batch-size 32768'