Disaggregated regression data

This dataset materializes the period-level columns repeatedly used by the hazard, predictive-performance, and wage analyses.

flowchart TB
    scores[(Worker-firm-period scores)]
    periods[(Model relationship periods)]
    firms[(Firms)]
    build[Validate aligned shards<br/>and select analysis columns]
    output[(Disaggregated regression data)]

    scores --> build
    periods --> build
    firms --> build
    build --> output

    classDef source fill:#e8eef7,stroke:#52739e
    classDef process fill:#fff0cc,stroke:#a66b00
    classDef data fill:#eee5f5,stroke:#76528c
    class scores,periods,firms source
    class build process
    class output data

Grain: one scored worker-firm-period
Location: /data/<coverage>/analyses/<mode>/disaggregated_regression_table/
Builder: analyses/modules/build_disaggregated_data.R
Entry point: analyses/pipelines/build_disaggregated_data.R

The builder combines MQ, hazard and wage outcomes, sample flags, tenure and calendar controls, job attributes, worker characteristics, and firm characteristics. It validates score metadata, matching period-shard filenames, row counts, keys, and row order before writing. Output shards retain the source filenames and include _metadata.yaml.

Rscript pipelines/build_disaggregated_data.R \
  --config config/config_one_percent.yaml --mode training --overwrite
Rscript pipelines/build_disaggregated_data.R \
  --config config/config_one_percent.yaml --mode full_training --overwrite

On Slurm, use build-disaggregated-data. The primary consumers are hazard estimates, predictive performance, and wage regressions.