Next-employer prediction¶
This analysis produces two compact, publication-facing destination-prediction tables from the v10 embedding-native destination-rank data: one for the worker split and one for the time split. Each table evaluates HCT on held-out moves and compares its ranks with a uniform random ordering of the same 1,000 candidate firms. The profile tables use all years present in their rank data, including 2024. The analysis does not estimate a firm-size-adjusted effect.
flowchart LR
ranks[(Worker and time<br/>destination ranks)]
summarize[[Summarize held-out moves<br/>separately by profile]]
metrics[(Profile-specific metrics CSVs)]
latex[/Two profile-specific LaTeX tables/]
ranks --> summarize --> metrics
summarize --> latex
classDef input fill:#eee5f5,stroke:#76528c,stroke-width:2px
classDef code fill:#f3f3f3,stroke:#666,stroke-width:2px
classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
classDef tableOutput fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
class ranks input
class summarize code
class metrics support
class latex tableOutput
Inputs
| Dataset | Location |
|---|---|
Destination ranks (training, v10) |
/data/<coverage>/analyses/profile_worker/training/destination_ranks/ and /data/<coverage>/analyses/profile_time/training/destination_ranks/, or their matching shards/user_00_01/ scopes |
Build
| Component | Location |
|---|---|
| Builder | analyses/modules/destination_prediction.R |
| Runtime | hct-tower-analyses-cpu |
| Entry point | analyses/pipelines/destination_prediction.R / destination-prediction |
Output
| Dataset | File / location | Grain and variables |
|---|---|---|
| Worker-split results | CSV, LaTeX table, and metadata in /artifacts/<coverage>/analyses/destination_prediction/profile_worker/, or its shard subdirectory |
One row per candidate-market and mover group; event and worker counts; random and HCT Recall@1, Recall@10, and MRR |
| Time-split results | CSV, LaTeX table, and metadata in /artifacts/<coverage>/analyses/destination_prediction/profile_time/, or its shard subdirectory |
Same measures, calculated independently for the time split |
The LaTex table has two panels. Panel A uses the global candidate pool and
reports all held-out movers, same-industry movers, and industry-switching
movers. The latter two rows are independently balanced samples from
global_industry_transition; they are not partitions of the all-mover row.
Panel B repeats these three rows using candidates drawn within the observed
destination's broad industry (industry2). Its transition rows are filters of
the industry-stratified all-mover sample, rather than independently balanced
transition samples.
Every retained event has the realised next employer plus 999 uniformly sampled, time-valid alternatives. The random benchmark is consequently calculated from the actual candidate count for every event (0.1% Recall@1 and 1% Recall@10 with the current 1,000-firm design). HCT's MRR is the full mean reciprocal rank, not MRR truncated at ten candidates.
The second panel is an oracle diagnostic: it uses the realised destination's broad industry to form the candidate pool. It asks whether HCT can distinguish firms within that broad market, not whether it can forecast the industry of a move. One table-wide random-baseline row is calculated from the common 1,000-candidate design. The table intentionally omits firm-size rows and makes no claim that HCT adds information beyond firm size; that requires a future, fine-size-matched candidate experiment.
The pipeline creates both profile tables in one run. Pass --shards 00,01 to
summarize the matching destination-rank scope for both profiles. The table
builder accepts either a combined scorer output at
destination_ranks/shards/user_00_01/ or separate completed outputs at
destination_ranks/shards/user_00/ and .../user_01/; in the latter case it
combines the source Parquet files itself. It writes matching table scopes at
destination_prediction/profile_worker/shards/user_00_01/ and
destination_prediction/profile_time/shards/user_00_01/. Use the all-shard
run for final estimates.
Command reference
| Option | Required / default | Source-defined effect |
|---|---|---|
--config PATH |
config/config_one_percent.yaml |
Analyses YAML config; the pipeline reads both profile paths from it. |
--shards IDS |
All destination-rank shards | Comma- or whitespace-separated IDs from 00 through 19; reads the matching destination_ranks/shards/user_<IDs>/ scope and writes a matching result scope. |
| Option | Required / default | Source-defined effect |
|---|---|---|
--task destination-prediction |
Required | Selects the CPU analyses task and writes both profile tables. --mode is not used. |
--config CONFIG.yaml |
Required | Resolves the analyses config passed to the pipeline. |
--shards IDS |
All destination-rank shards | Passes the matching source/output shard scope to the R table builder. |
--partition, --time, --mem, --cpus, --job-name, --dependency, --dry-run |
Wrapper defaults | Standard CPU overrides; default is 8 CPUs, 128 GB, and 12 hours. |