Predictive performance

This analysis evaluates separation predictions out of sample for the HCT model and five logit benchmarks.

flowchart LR
    scores[(Training scores)]
    data[(Training disaggregated data)]
    fit[[Fit logits in sample<br/>and predict OOS]]
    predictions[(OOS prediction Parquet files)]
    metrics[(AUC, BCE, pseudo-R2,<br/>and sample statistics)]
    latex[/model_performance.tex/]

    scores --> predictions
    data --> fit --> predictions --> metrics --> latex

    classDef upstream fill:#e8eef7,stroke:#52739e,stroke-width:2px
    classDef input fill:#eee5f5,stroke:#76528c,stroke-width:2px
    classDef code fill:#f3f3f3,stroke:#666,stroke-width:2px
    classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
    classDef tableOutput fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
    class scores upstream
    class data input
    class fit code
    class predictions,metrics support
    class latex tableOutput

All benchmark logits are estimated on training in-sample rows and evaluated on training out-of-sample workers. The HCT panel reports the saved tenure baseline and full tenure-plus-MQ hazard predictions. The logit panel reports:

• a constant separation rate;
• tenure fixed effects;
• tenure and calendar-year fixed effects;
• tenure, calendar-year, and observable controls;
• the same controls plus MQ.

Metrics are AUC, binary cross-entropy, and pseudo-$R^2$. Observation, worker, and firm counts are saved separately for the estimation, evaluation, and total samples.

Inputs: Worker-firm-period scores and disaggregated regression data Module: analyses/modules/model_performance.R

Rscript pipelines/model_performance.R \
  --config config/config_one_percent.yaml

slurm/submit.sh --task model-performance \
  --config config_full.yaml

Saved OOS predictions are the durable expensive output. If metrics are missing or change, the pipeline can recompute them from those Parquet files without refitting. Use --rerun only when predictions and fits must be replaced.

/artifacts/<coverage>/analyses/model_performance/
  model_performance_models/
    *_oos_predictions.parquet
    *_metrics.rds
  model_performance.csv
  model_performance_statistics.csv
  model_performance_statistics.rds
  model_performance.tex
  model_performance_metadata.yaml