Predictive performance¶
This analysis evaluates separation predictions out of sample for the HCT model and five logit benchmarks. One invocation produces separate tables for the worker and time training profiles.
flowchart LR
worker[(Worker-profile disaggregated data)]
time[(Time-profile disaggregated data)]
fit[[Fit logits in sample<br/>and predict OOS]]
predictions[(OOS prediction Parquet files)]
metrics[(AUC, BCE, pseudo-R2,<br/>and sample statistics)]
latex[/model_performance.tex/]
worker --> fit
time --> fit
fit --> predictions --> metrics --> latex
classDef upstream fill:#e8eef7,stroke:#52739e,stroke-width:2px
classDef input fill:#eee5f5,stroke:#76528c,stroke-width:2px
classDef code fill:#f3f3f3,stroke:#666,stroke-width:2px
classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
classDef tableOutput fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
class worker,time input
class fit code
class predictions,metrics support
class latex tableOutput
Inputs
| Dataset | Location |
|---|---|
Disaggregated regression data (training, worker profile) |
/data/<coverage>/analyses/profile_worker/training/disaggregated_regression_data/ |
Disaggregated regression data (training, time profile) |
/data/<coverage>/analyses/profile_time/training/disaggregated_regression_data/ |
Build
| Component | Location |
|---|---|
| Builder | analyses/modules/model_performance.R |
| Runtime | hct-tower-analyses-cpu |
| Entry point | analyses/pipelines/model_performance.R / model-performance |
Output
| Dataset | File / location | Grain and variables |
|---|---|---|
| Worker-profile predictive-performance results | Metrics, OOS predictions, cached fits, and model_performance.tex in /artifacts/<coverage>/analyses/model_performance/profile_worker/ |
One metric row and one OOS prediction file per neural or logit specification |
| Time-profile predictive-performance results | Metrics, OOS predictions, cached fits, and model_performance.tex in /artifacts/<coverage>/analyses/model_performance/profile_time/ |
One metric row and one OOS prediction file per neural or logit specification |
Each profile is analyzed separately. Benchmark logits are estimated on that profile's in-sample rows and evaluated on its out-of-sample rows. For the worker profile, the held-out sample contains workers excluded from training. For the time profile, the held-out sample contains test-period observations; under the current split, train and validation periods through 2016 are in-sample and 2017 onward is out-of-sample. The HCT panel reports the saved tenure baseline and full tenure-plus-MQ hazard predictions. The logit panel reports:
- a constant separation rate;
- tenure fixed effects;
- tenure and calendar-year fixed effects;
- tenure, calendar-year, and observable controls;
- the same controls plus MQ.
Metrics are AUC, binary cross-entropy, and pseudo-$R^2$. Observation, worker, and firm counts are saved separately for the estimation, evaluation, and total samples.
Command reference
| Option | Required / default | Source-defined effect |
|---|---|---|
--config PATH |
config/config_one_percent.yaml |
Analyses YAML config. |
--shards ID[,ID,…] |
Unset | Restricts the user-shard subset processed. |
--rerun |
Off | Refits all saved models for both profiles instead of reusing compatible profile-specific fits. |
| Option | Required / default | Source-defined effect |
|---|---|---|
--task model-performance |
Required | Selects the CPU analyses task. |
--config CONFIG.yaml, --shards ID[,ID,…], --rerun |
Config required; others off | Passed to the R pipeline. |
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run |
Wrapper defaults | Standard CPU overrides; default is 8 CPUs, 800 GB, and two days. |
One command builds both profile tables. --shards selects the same user shards
for each profile. Outputs and caches are kept in separate profile_worker/ and
profile_time/ directories. Saved OOS predictions are the durable expensive
output. A normal run reuses the complete compatible cache separately for each
profile; use --rerun to replace predictions and fits for both profiles.
Sharded runs¶
Use --shards with comma-separated user-shard identifiers for a scoped,
exploratory result. Outputs are isolated under shards/user_<ids>/.