Skip to content

Next-employer prediction

This analysis produces two compact, publication-facing destination-prediction tables from the v10 embedding-native destination-rank data: one for the worker split and one for the time split. Each table evaluates HCT on held-out moves and compares its ranks with a uniform random ordering of the same 1,000 candidate firms. The profile tables use all years present in their rank data, including 2024. The analysis does not estimate a firm-size-adjusted effect.

flowchart LR
    ranks[(Worker and time<br/>destination ranks)]
    summarize[[Summarize held-out moves<br/>separately by profile]]
    metrics[(Profile-specific metrics CSVs)]
    latex[/Two profile-specific LaTeX tables/]

    ranks --> summarize --> metrics
    summarize --> latex

    classDef input fill:#eee5f5,stroke:#76528c,stroke-width:2px
    classDef code fill:#f3f3f3,stroke:#666,stroke-width:2px
    classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
    classDef tableOutput fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
    class ranks input
    class summarize code
    class metrics support
    class latex tableOutput
Inputs
Dataset Location
Destination ranks (training, v10) /data/<coverage>/analyses/profile_worker/training/destination_ranks/ and /data/<coverage>/analyses/profile_time/training/destination_ranks/, or their matching shards/user_00_01/ scopes
Build
Component Location
Builder analyses/modules/destination_prediction.R
Runtime hct-tower-analyses-cpu
Entry point analyses/pipelines/destination_prediction.R / destination-prediction
Output
Dataset File / location Grain and variables
Worker-split results CSV, LaTeX table, and metadata in /artifacts/<coverage>/analyses/destination_prediction/profile_worker/, or its shard subdirectory One row per candidate-market and mover group; event and worker counts; random and HCT Recall@1, Recall@10, and MRR
Time-split results CSV, LaTeX table, and metadata in /artifacts/<coverage>/analyses/destination_prediction/profile_time/, or its shard subdirectory Same measures, calculated independently for the time split

The LaTex table has two panels. Panel A uses the global candidate pool and reports all held-out movers, same-industry movers, and industry-switching movers. The latter two rows are independently balanced samples from global_industry_transition; they are not partitions of the all-mover row. Panel B repeats these three rows using candidates drawn within the observed destination's broad industry (industry2). Its transition rows are filters of the industry-stratified all-mover sample, rather than independently balanced transition samples.

Every retained event has the realised next employer plus 999 uniformly sampled, time-valid alternatives. The random benchmark is consequently calculated from the actual candidate count for every event (0.1% Recall@1 and 1% Recall@10 with the current 1,000-firm design). HCT's MRR is the full mean reciprocal rank, not MRR truncated at ten candidates.

The second panel is an oracle diagnostic: it uses the realised destination's broad industry to form the candidate pool. It asks whether HCT can distinguish firms within that broad market, not whether it can forecast the industry of a move. One table-wide random-baseline row is calculated from the common 1,000-candidate design. The table intentionally omits firm-size rows and makes no claim that HCT adds information beyond firm size; that requires a future, fine-size-matched candidate experiment.

The pipeline creates both profile tables in one run. Pass --shards 00,01 to summarize the matching destination-rank scope for both profiles. The table builder accepts either a combined scorer output at destination_ranks/shards/user_00_01/ or separate completed outputs at destination_ranks/shards/user_00/ and .../user_01/; in the latter case it combines the source Parquet files itself. It writes matching table scopes at destination_prediction/profile_worker/shards/user_00_01/ and destination_prediction/profile_time/shards/user_00_01/. Use the all-shard run for final estimates.

Command reference
Option Required / default Source-defined effect
--config PATH config/config_one_percent.yaml Analyses YAML config; the pipeline reads both profile paths from it.
--shards IDS All destination-rank shards Comma- or whitespace-separated IDs from 00 through 19; reads the matching destination_ranks/shards/user_<IDs>/ scope and writes a matching result scope.
Option Required / default Source-defined effect
--task destination-prediction Required Selects the CPU analyses task and writes both profile tables. --mode is not used.
--config CONFIG.yaml Required Resolves the analyses config passed to the pipeline.
--shards IDS All destination-rank shards Passes the matching source/output shard scope to the R table builder.
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run Wrapper defaults Standard CPU overrides; default is 8 CPUs, 128 GB, and 12 hours.
cd /hct-tower/analyses
Rscript --vanilla pipelines/destination_prediction.R \
  --config config/config_full.yaml
Rscript --vanilla pipelines/destination_prediction.R \
  --config config/config_full.yaml --shards 00,01
cd /hct-tower
slurm/submit.sh --task destination-prediction \
  --config config_full.yaml
slurm/submit.sh --task destination-prediction \
  --config config_full.yaml --shards 00,01