Skip to content

Predictive performance

This analysis evaluates separation predictions out of sample for the HCT model and five logit benchmarks. One invocation produces separate tables for the worker and time training profiles.

flowchart LR
    worker[(Worker-profile disaggregated data)]
    time[(Time-profile disaggregated data)]
    fit[[Fit logits in sample<br/>and predict OOS]]
    predictions[(OOS prediction Parquet files)]
    metrics[(AUC, BCE, pseudo-R2,<br/>and sample statistics)]
    latex[/model_performance.tex/]

    worker --> fit
    time --> fit
    fit --> predictions --> metrics --> latex

    classDef upstream fill:#e8eef7,stroke:#52739e,stroke-width:2px
    classDef input fill:#eee5f5,stroke:#76528c,stroke-width:2px
    classDef code fill:#f3f3f3,stroke:#666,stroke-width:2px
    classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
    classDef tableOutput fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
    class worker,time input
    class fit code
    class predictions,metrics support
    class latex tableOutput
Inputs
Dataset Location
Disaggregated regression data (training, worker profile) /data/<coverage>/analyses/profile_worker/training/disaggregated_regression_data/
Disaggregated regression data (training, time profile) /data/<coverage>/analyses/profile_time/training/disaggregated_regression_data/
Build
Component Location
Builder analyses/modules/model_performance.R
Runtime hct-tower-analyses-cpu
Entry point analyses/pipelines/model_performance.R / model-performance
Output
Dataset File / location Grain and variables
Worker-profile predictive-performance results Metrics, OOS predictions, cached fits, and model_performance.tex in /artifacts/<coverage>/analyses/model_performance/profile_worker/ One metric row and one OOS prediction file per neural or logit specification
Time-profile predictive-performance results Metrics, OOS predictions, cached fits, and model_performance.tex in /artifacts/<coverage>/analyses/model_performance/profile_time/ One metric row and one OOS prediction file per neural or logit specification

Each profile is analyzed separately. Benchmark logits are estimated on that profile's in-sample rows and evaluated on its out-of-sample rows. For the worker profile, the held-out sample contains workers excluded from training. For the time profile, the held-out sample contains test-period observations; under the current split, train and validation periods through 2016 are in-sample and 2017 onward is out-of-sample. The HCT panel reports the saved tenure baseline and full tenure-plus-MQ hazard predictions. The logit panel reports:

  • a constant separation rate;
  • tenure fixed effects;
  • tenure and calendar-year fixed effects;
  • tenure, calendar-year, and observable controls;
  • the same controls plus MQ.

Metrics are AUC, binary cross-entropy, and pseudo-$R^2$. Observation, worker, and firm counts are saved separately for the estimation, evaluation, and total samples.

Command reference
Option Required / default Source-defined effect
--config PATH config/config_one_percent.yaml Analyses YAML config.
--shards ID[,ID,…] Unset Restricts the user-shard subset processed.
--rerun Off Refits all saved models for both profiles instead of reusing compatible profile-specific fits.
Option Required / default Source-defined effect
--task model-performance Required Selects the CPU analyses task.
--config CONFIG.yaml, --shards ID[,ID,…], --rerun Config required; others off Passed to the R pipeline.
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run Wrapper defaults Standard CPU overrides; default is 8 CPUs, 800 GB, and two days.
cd /hct-tower/analyses
Rscript --vanilla pipelines/model_performance.R \
  --config config/config_one_percent.yaml
cd /hct-tower
slurm/submit.sh --task model-performance \
  --config config_full.yaml

One command builds both profile tables. --shards selects the same user shards for each profile. Outputs and caches are kept in separate profile_worker/ and profile_time/ directories. Saved OOS predictions are the durable expensive output. A normal run reuses the complete compatible cache separately for each profile; use --rerun to replace predictions and fits for both profiles.

Sharded runs

Use --shards with comma-separated user-shard identifiers for a scoped, exploratory result. Outputs are isolated under shards/user_<ids>/.

cd /hct-tower/analyses
Rscript --vanilla pipelines/model_performance.R \
  --config config/config_full.yaml --shards 00,01
cd /hct-tower
slurm/submit.sh --task model-performance --config config_full.yaml \
  --shards 00,01