Skip to content

Slurm on Stanford Clusters (Yen / Sherlock)

This guide explains the recommended workflow for running the model on Stanford Slurm clusters, with examples oriented toward the Stanford GSB Yen environment.

Context

Relevant Stanford documentation:

  • Yen Slurm overview: https://rcpedia.stanford.edu/_user_guide/slurm/
  • Yen GPU jobs: https://rcpedia.stanford.edu/_user_guide/using_gpu/
  • Yen software modules: https://rcpedia.stanford.edu/_getting_started/modules/
  • Sherlock Apptainer: https://www.sherlock.stanford.edu/docs/software/containers/apptainer/
  • Sherlock GSB partition: https://rcpedia.stanford.edu/_user_guide/sherlock/

This project may be run on both Yen and Sherlock. General lab-server Podman setup is documented in lab-containers.md. The Slurm container workflow is the same at a high level:

  1. build the Podman CUDA image from model/Containerfile
  2. export it to a container archive
  3. convert that archive to a .sif image
  4. submit a Slurm job that runs the code inside the .sif

This guide uses Apptainer terminology. On both Sherlock and Yen, the documented workflow for this project should be expressed in terms of apptainer.

Repo Structure

All cluster-execution assets live under the top-level slurm/ directory:

  • slurm/submit.sh: main user-facing submission wrapper
  • slurm/common.sh: shared runtime helpers for Slurm jobs
  • slurm/model/: model job templates
  • slurm/shared/: shared data-building job templates
  • slurm/analyses/: CPU job templates for analyses data building and tests

The recommended user workflow is to call slurm/submit.sh rather than sbatch the task templates directly.

The wrapper keeps the user interface small while centralizing the Slurm and Apptainer logic.

Examples:

slurm/submit.sh --task train --profile worker --mode training --config config_samples.yaml
slurm/submit.sh --task train --mode full_training --config config_samples.yaml
# Start only after job 123456 completes successfully:
slurm/submit.sh --task train --mode full_training --config config_full.yaml --dependency afterok:123456
slurm/submit.sh --task train --mode full_training --config config_full.yaml --resume-latest-epoch
slurm/submit.sh --task train --mode full_training --config config_full.yaml --resume-from-checkpoint /artifacts/full/full_training/checkpoints/model_epoch1_iter12345.pt
CV_TRIAL=learning_rate__low slurm/submit.sh --task cross-validate --profile time --config config_one_percent.yaml
slurm/submit.sh --task benchmark-training --profile worker --config config_one_percent.yaml --benchmark-args '--workers 4 --prefetch-factor 2 --tensor-block-rows 32768 --period-batch-rows 500000'
slurm/submit.sh --task score --profile worker --mode training --config config_full.yaml
slurm/submit.sh --task score --profile time --mode training --config config_full.yaml
slurm/submit.sh --task score --mode full_training --config config_full.yaml
# Build destination-rank analysis datasets for the R destination-prediction task:
slurm/submit.sh --task score-destination-ranks --profile worker --mode training --config config_full.yaml
slurm/submit.sh --task score-destination-ranks --profile time --mode training --config config_full.yaml
slurm/submit.sh --task score-destination-ranks --mode full_training --config config_full.yaml
# Score and summarize the same selected user shards without overwriting full outputs:
slurm/submit.sh --task score-destination-ranks --profile worker --mode training --config config_full.yaml --shards 00,01
slurm/submit.sh --task destination-prediction --config config_full.yaml --shards 00,01
slurm/submit.sh --task score-xbrl-target-counterfactual-mq --profile worker --mode training --config config_full.yaml
slurm/submit.sh --task score-xbrl-target-counterfactual-mq --profile time --mode training --config config_full.yaml
slurm/submit.sh --task score-xbrl-target-counterfactual-mq --mode full_training --config config_full.yaml
# Export reusable company and mover embedding tables:
slurm/submit.sh --task export-embedding-tables --profile worker --target company --mode training --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile time --target company --mode training --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile worker --target workers --mode training --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile time --target workers --mode training --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --target workers --mode full_training --config config_full.yaml
# Resume an interrupted scoring run:
slurm/submit.sh --task score --profile worker --mode training --config config_full.yaml --resume
# Resume an interrupted destination-rank scoring run:
slurm/submit.sh --task score-destination-ranks --profile worker --mode training --config config_full.yaml --resume
slurm/submit.sh --task build-processed --config config_full.yaml
slurm/submit.sh --task build-processed-sample --sample one_percent
slurm/submit.sh --task build-processed-sample --sample samples
slurm/submit.sh --task embed-about --config config_samples.yaml
slurm/submit.sh --task build-rcid-year-mq --config config_full.yaml
slurm/submit.sh --task build-disaggregated-regression-data --mode training --profile worker --config config_full.yaml --overwrite
slurm/submit.sh --task build-disaggregated-regression-data --mode training --profile time --config config_full.yaml --overwrite
slurm/submit.sh --task build-execucomp-regression-table --mode training --config config_one_percent.yaml --overwrite
slurm/submit.sh --task build-execucomp-regression-table --mode full_training --config config_one_percent.yaml --overwrite
slurm/submit.sh --task pairs-variations --config config_one_percent.yaml
slurm/submit.sh --task pairs-variations --config config_one_percent.yaml --shards 00,01
slurm/submit.sh --task destination-prediction --config config_one_percent.yaml
slurm/submit.sh --task wage-regressions --config config_one_percent.yaml
slurm/submit.sh --task execucomp-regressions --config config_one_percent.yaml
# Saved wage models are reused automatically; refit all models only if intended:
slurm/submit.sh --task wage-regressions --config config_one_percent.yaml --rerun

The analyses tasks are independent. Use --dependency to ask Slurm to hold a job until its input-building job reaches a specified state. The value is passed directly to sbatch; for example, --dependency afterok:123456 starts the job only if job 123456 exits successfully. Analyses tasks use the CPU image hct-tower-analyses-cpu.sif by default. The default requests are 64 GB and 6 hours for the disaggregated-data builder, 256 GB and 12 hours for pair variations, 128 GB and 12 hours for destination prediction, and 800 GB and two days for hazard estimates, model performance, and wage regressions. The ExecuComp table builder requests 64 GB and 24 hours; ExecuComp regressions request 64 GB and four hours. Override these with --mem, --time, --cpus, --partition, or --sif when needed.

For a GPU node with a driver or device fault, add --exclude <node> to any wrapper task. For example:

slurm/submit.sh --task embed-about --config config_full.yaml --exclude gpu042
slurm/submit.sh --task train --profile worker --mode training --config config_full.yaml --exclude gpu042

The option is passed directly to sbatch, so it works for every GPU wrapper task: embedding, training, cross-validation, benchmarking, scoring, destination rank scoring, and smoke tests.

The builder writes the configured regression-ready dataset. It defaults to full_training; for --mode training, also select --profile worker or --profile time. Use --overwrite only when intentionally replacing an existing output. For example:

slurm/submit.sh --task build-disaggregated-regression-data --mode full_training \
  --config config_one_percent.yaml --dry-run

All four jobs write their Slurm stdout and stderr files to /artifacts/slurm/<job-name>.<job-id>.{out,err}. The builder emits a live progress bar only when attached to a terminal; under Slurm it emits periodic newline-delimited progress records, including rows, elapsed time, and ETA.

build-processed builds the full canonical processed data from the staged US employment-position records. build-processed-sample derives either one_percent or samples from that full output; --sample selects the corresponding output config and sampling rule. Use --source-config or --config only to override those default paths. It runs on the model CUDA image without a GPU request and defaults to 64 GB, 8 CPUs, and six hours. embed-about is a one-time setup job for the full processed company data; sampled builds subset that output. If the configured outputs already exist, downstream jobs can reuse them. Rebuild only when inputs, processing logic, feature choices, or output paths change.

The accounting-market builders are independent model-container CPU tasks. Run the raw source stage before its corresponding processed stage:

slurm/submit.sh --task build-raw-compustat --config config_full.yaml
slurm/submit.sh --task build-processed-compustat --config config_full.yaml
slurm/submit.sh --task build-raw-crsp --config config_full.yaml
slurm/submit.sh --task build-processed-crsp --config config_full.yaml

The ExecuComp builder is separate from the main disaggregated regression-table builder. Run it once for training and once for full_training, then submit the regression task after both compact tables are present:

slurm/submit.sh --task build-execucomp-regression-table --mode training \
  --config config_full.yaml --overwrite
slurm/submit.sh --task build-execucomp-regression-table --mode full_training \
  --config config_full.yaml --overwrite
slurm/submit.sh --task execucomp-regressions --config config_full.yaml

The ExecuComp builder logs one flushed line for every score shard to the Slurm .out file. The regression runner writes a line for each fitted specification; errors remain in the .err file.

The wrapper automatically:

  • chooses the correct underlying job template
  • maps the config file to model/config/ or shared/config/
  • uses sensible defaults for repo, data, artifacts, and container roots
  • chooses CPU vs GPU Slurm resources based on the task
  • runs the job inside the shared .sif

Task behavior itself stays in config. For example:

  • cross-validate reads cross_validation_search.*; set CV_TRIAL to run one trial
  • benchmark-training runs training.benchmark_training and writes one JSON result to the Slurm stdout log
  • train --profile PROFILE --mode training trains on that profile's train+valid splits and evaluates on its test split
  • train --mode full_training trains once on train, validation, and test rows and writes to the shared full-training artifact root; it does not use a worker/time profile
  • train accepts direct resume flags: --resume-latest-epoch, --resume-latest-iteration, or --resume-from-checkpoint PATH; use --train-args for other training-entrypoint flags
  • score --profile PROFILE --mode ... selects that profile's checkpoint and writes under its score_root
  • score-destination-ranks --profile PROFILE --mode ... reads that profile's worker/company embedding archives and ranks observed movers using prior-year company employment counts. It is a GPU task. --shards 00,01 restricts mover events while retaining the full candidate-firm universe and writes an isolated matching shards/user_00_01/ output. It checkpoints after each selected mover-embedding source shard, so rerun the same command with --resume after a wall-time limit or interruption. Do not combine --resume with --overwrite.
  • score-xbrl-target-counterfactual-mq --mode training --profile PROFILE reads that worker/time profile's XBRL deal panel and completed company/worker-firm-period embedding archives. It preserves separate in-sample and out-of-sample deal rows and scores each against only the matching worker split. --mode full_training reads the shared full-training panel and archives and writes one counterfactual summary row per deal. This is a GPU task.
  • export-embedding-tables --profile PROFILE --target company\|workers --mode ... writes either the static company-vector table or one source-sharded worker-state population under that profile's score root. Vectors are raw per-vector int8 quantizations with FP16 reconstruction scales; checkpoint MQ dimension weights are recorded in metadata. The targets are independent: worker exports do not read or recreate the saved company table. The worker target checkpoints after each source shard; rerun it with --resume after interruption. Do not combine --resume with --overwrite.
  • wage-regressions saves each fitted model before moving to the next one. Rerun the same command after a wall-time limit to reuse saved models and fit only missing ones. Pass --rerun only to refit all models. Its progress lines are written to the Slurm .out file.
  • embed-about reads about_embeddings.*

To inspect the resolved submission command without launching a job:

slurm/submit.sh --task train --profile worker --mode training --config config_samples.yaml --dry-run

Cross-validation On Slurm

For CV sweeps, submit one GPU job per trial and let Slurm place jobs across available GPUs:

CV_TRIAL=learning_rate__low slurm/submit.sh --task cross-validate --profile worker --config config_one_percent.yaml
CV_TRIAL=learning_rate__high slurm/submit.sh --task cross-validate --profile time --config config_one_percent.yaml

Dry-run a trial submission with:

CV_TRIAL=learning_rate__low slurm/submit.sh --task cross-validate --profile worker --config config_one_percent.yaml --dry-run

After the trial jobs finish, regenerate the combined CSV and plots from inside the model container or an interactive job:

cd /hct-tower/model
python -m training.cross_validate --config config/config_one_percent.yaml --profile worker --summarize-only

Slurm normally reserves the requested GPU resource for a job until that job ends. For this project, prefer multiple one-GPU trial jobs over running several heavy trials on the same GPU.

Interactive Testing Before Submission

Before submitting a full batch job, it is often useful to start an interactive Slurm allocation and verify that:

  • the requested resources are available
  • apptainer loads correctly
  • the .sif image runs
  • the bind mounts and Python command behave as expected

There are two common Slurm patterns for this:

  • srun --pty /bin/bash: quickest way to land on a compute node and test commands interactively
  • salloc: fuller interactive allocation, especially useful on Sherlock when you may want to launch additional srun steps inside the allocation

For this project, the most practical choice before a GPU batch submission is usually a short interactive GPU shell.

Examples:

Yen interactive GPU shell:

srun -p gpu -G 1 -n 1 -c 4 --mem=32G -t 01:00:00 --pty /bin/bash

Sherlock interactive GPU shell:

srun -p gpu -G 1 -n 1 -c 4 --mem=32G -t 01:00:00 --pty /bin/bash

Add -C "GPU_MODEL:A40" only when a run specifically needs that GPU class.

If you specifically want a Sherlock interactive allocation rather than a one-shot shell command, the Slurm docs also support:

salloc -p gpu --gpus 1 -c 4 --mem=32G -t 01:00:00

Once the allocation starts, you can test the container directly:

module purge
module load apptainer

REPO_ROOT="${HOME}/hct-tower"
DATA_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/data"
ARTIFACT_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/artifacts"
CONTAINER_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/containers"
SIF_PATH="${CONTAINER_ROOT}/hct-tower-model-cuda.sif"

apptainer exec --nv \
  --bind "${REPO_ROOT}:/hct-tower,${DATA_ROOT}:/data,${ARTIFACT_ROOT}:/artifacts" \
  "${SIF_PATH}" \
  bash -lc "cd /hct-tower/model && python -c 'import torch; print(torch.cuda.is_available())'"

Useful quick follow-up checks inside the interactive session:

nvidia-smi
python -c "import torch; print(torch.cuda.is_available())"
slurm/submit.sh --task train --profile worker --config config_samples.yaml --dry-run

This is the easiest way to debug image, mount, GPU, and config issues before moving to sbatch.

The following instructions assume you are trying to run a slurm job for the model.

1. Build the CUDA image

From inside model/:

podman build -t hct-tower-model-cuda .

2. Export the image to an archive

The tarball is just an intermediate build artifact. To avoid polluting the repo, create it in a temporary or personal working directory outside /hct-tower.

cd ~

Use an explicit Docker archive so the next step is unambiguous:

podman save --format docker-archive -o hct-tower-model-cuda.tar hct-tower-model-cuda

3. Convert the archive to a .sif

On a Stanford cluster node with the container runtime available, build the .sif:

The commands below will save the .sif to a shared folder so it simplifies the slurm scripts and can be reused across users.

The shared folder for .sif files is /zfs/projects/faculty/jungho-idhlab/drstrange/containers/.

module purge
module load apptainer

CONTAINER_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/containers"
mkdir -p "${CONTAINER_ROOT}"

apptainer build "${CONTAINER_ROOT}/hct-tower-model-cuda.sif" \
  docker-archive://$PWD/hct-tower-model-cuda.tar

You only need to rebuild the .sif when the underlying image changes.

Direct Templates

If needed, you can still submit the underlying templates directly with sbatch, but slurm/submit.sh is the recommended entrypoint.

Monitoring and Debugging

Useful commands:

sinfo -p gpu
squeue -u $USER
squeue --start
scancel <JOBID>

Training progress is environment-aware. In an interactive terminal, training uses a tqdm progress bar. In non-interactive runs such as sbatch, it skips the bar and writes plain progress lines to stdout every training.log_every_iterations iterations, defaulting to 100. You can override this with training.progress_mode: auto, bar, log, or none.

Inside a running GPU allocation, nvidia-smi is the fastest basic sanity check.

References

  • Yen Slurm overview: https://rcpedia.stanford.edu/_user_guide/slurm/
  • Yen GPU jobs: https://rcpedia.stanford.edu/_user_guide/using_gpu/
  • Yen software modules: https://rcpedia.stanford.edu/_getting_started/modules/
  • Sherlock Apptainer: https://www.sherlock.stanford.edu/docs/software/containers/apptainer/
  • Sherlock GSB partition: https://rcpedia.stanford.edu/_user_guide/sherlock/