Skip to content

Slurm on Stanford Clusters (Yen / Sherlock)

This guide explains the recommended workflow for running the model on Stanford Slurm clusters, with examples oriented toward the Stanford GSB Yen environment.

Context

Relevant Stanford documentation:

  • Yen Slurm overview: https://rcpedia.stanford.edu/_user_guide/slurm/
  • Yen GPU jobs: https://rcpedia.stanford.edu/_user_guide/using_gpu/
  • Yen software modules: https://rcpedia.stanford.edu/_getting_started/modules/
  • Sherlock Apptainer: https://www.sherlock.stanford.edu/docs/software/containers/apptainer/
  • Sherlock GSB partition: https://rcpedia.stanford.edu/_user_guide/sherlock/

This project may be run on both Yen and Sherlock. General lab-server Podman setup is documented in lab-containers.md. The Slurm container workflow is the same at a high level:

  1. build the Podman CUDA image from model/Containerfile
  2. export it to a container archive
  3. convert that archive to a .sif image
  4. submit a Slurm job that runs the code inside the .sif

This guide uses Apptainer terminology. On both Sherlock and Yen, the documented workflow for this project should be expressed in terms of apptainer.

Repo Structure

All cluster-execution assets live under the top-level slurm/ directory:

  • slurm/submit.sh: main user-facing submission wrapper
  • slurm/common.sh: shared runtime helpers for Slurm jobs
  • slurm/model/: model job templates
  • slurm/shared/: shared data-building job templates
  • slurm/analyses/: CPU job templates for analyses data building and tests

The recommended user workflow is to call slurm/submit.sh rather than sbatch the task templates directly.

The wrapper keeps the user interface small while centralizing the Slurm and Apptainer logic.

Examples:

slurm/submit.sh --task train --mode training --config config_samples.yaml
slurm/submit.sh --task train --mode full_training --config config_samples.yaml
slurm/submit.sh --task train --mode full_training --config config_full.yaml --train-args '--resume-from-checkpoint /artifacts/full/full_training/epoch_checkpoints/model_epoch1_iter12345.pt'
CV_TRIAL=learning_rate__low slurm/submit.sh --task cross-validate --config config_one_percent.yaml
slurm/submit.sh --task benchmark-training --config config_one_percent.yaml --benchmark-args '--workers 4 --prefetch-factor 2 --tensor-block-rows 32768 --period-batch-rows 500000'
slurm/submit.sh --task score --mode training --config config_full.yaml
slurm/submit.sh --task score --mode full_training --config config_full.yaml
# Build sampled destination-rank datasets for the R destination-prediction task:
slurm/submit.sh --task score-destination-ranks --mode training --config config_full.yaml
slurm/submit.sh --task score-destination-ranks --mode full_training --config config_full.yaml
# Resume an interrupted scoring run:
slurm/submit.sh --task score --mode training --config config_full.yaml --resume
# Resume an interrupted destination-rank scoring run:
slurm/submit.sh --task score-destination-ranks --mode training --config config_full.yaml --resume
slurm/submit.sh --task build-processed --config config_full.yaml
slurm/submit.sh --task embed-about --config config_samples.yaml
slurm/submit.sh --task build-disaggregated-data --mode training --config config_one_percent.yaml --overwrite
slurm/submit.sh --task build-execucomp-regression-table --mode training --config config_one_percent.yaml --overwrite
slurm/submit.sh --task build-execucomp-regression-table --mode full_training --config config_one_percent.yaml --overwrite
slurm/submit.sh --task pairs-variations --config config_one_percent.yaml
slurm/submit.sh --task destination-prediction --config config_one_percent.yaml
slurm/submit.sh --task wage-regressions --config config_one_percent.yaml
slurm/submit.sh --task execucomp-regressions --config config_one_percent.yaml
# Saved wage models are reused automatically; refit all models only if intended:
slurm/submit.sh --task wage-regressions --config config_one_percent.yaml --rerun

The analyses tasks are independent. Submit them one at a time after their inputs are available; the wrapper does not create job dependencies. Analyses tasks use the CPU image hct-tower-analyses-cpu.sif by default. The default requests are 64 GB and 6 hours for the disaggregated-data builder, 256 GB and 12 hours for pair variations, 128 GB and 12 hours for destination prediction, and 800 GB and two days for hazard estimates, model performance, and wage regressions. The ExecuComp table builder requests 64 GB and 24 hours; ExecuComp regressions request 64 GB and four hours. Override these with --mem, --time, --cpus, --partition, or --sif when needed.

The builder writes the configured regression-ready dataset. It defaults to full_training; use --mode training for the training dataset, and use --overwrite only when intentionally replacing an existing output. For example:

slurm/submit.sh --task build-disaggregated-data --mode full_training \
  --config config_one_percent.yaml --dry-run

All four jobs write their Slurm stdout and stderr files to /artifacts/slurm/<job-name>.<job-id>.{out,err}. The builder emits a live progress bar only when attached to a terminal; under Slurm it emits periodic newline-delimited progress records, including rows, elapsed time, and ETA.

build-processed builds the full canonical processed data from the staged US employment-position records. The samples and one_percent processed datasets use the separate shared.data.build_processed_sample entry point documented in Employment-data coverage variants. embed-about is a one-time setup job for the full processed company data; sampled builds subset that output. If the configured outputs already exist, downstream jobs can reuse them. Rebuild only when inputs, processing logic, feature choices, or output paths change.

The ExecuComp builder is separate from the main disaggregated regression-table builder. Run it once for training and once for full_training, then submit the regression task after both compact tables are present:

slurm/submit.sh --task build-execucomp-regression-table --mode training \
  --config config_full.yaml --overwrite
slurm/submit.sh --task build-execucomp-regression-table --mode full_training \
  --config config_full.yaml --overwrite
slurm/submit.sh --task execucomp-regressions --config config_full.yaml

The ExecuComp builder logs one flushed line for every score shard to the Slurm .out file. The regression runner writes a line for each fitted specification; errors remain in the .err file.

The wrapper automatically:

  • chooses the correct underlying job template
  • maps the config file to model/config/ or shared/config/
  • uses sensible defaults for repo, data, artifacts, and container roots
  • chooses CPU vs GPU Slurm resources based on the task
  • runs the job inside the shared .sif

Task behavior itself stays in config. For example:

  • cross-validate reads cross_validation_search.*; set CV_TRIAL to run one trial
  • benchmark-training runs training.benchmark_training and writes one JSON result to the Slurm stdout log
  • train --mode training trains on the configured train+valid splits and evaluates on the configured test split
  • train --mode full_training trains on all configured splits for final scoring/analysis use
  • train accepts --train-args for operational flags such as --resume-from-checkpoint
  • score --mode training reads scoring.training.* and writes scoring.training.output_path
  • score --mode full_training reads scoring.full_training.* and writes scoring.full_training.output_path
  • score-destination-ranks --mode ... reads scoring.<mode>.datasets.destination_ranks.* and writes the compact sampled destination-rank dataset; it is a GPU model-scoring task, which must finish before the CPU destination-prediction analyses task. It checkpoints after each input relationship-period shard, so rerun the same command with --resume after a wall-time limit or interruption. Do not combine --resume with --overwrite.
  • wage-regressions saves each fitted model before moving to the next one. Rerun the same command after a wall-time limit to reuse saved models and fit only missing ones. Pass --rerun only to refit all models. Its progress lines are written to the Slurm .out file.
  • embed-about reads about_embeddings.*

To inspect the resolved submission command without launching a job:

slurm/submit.sh --task train --mode training --config config_samples.yaml --dry-run

Cross-validation On Slurm

For CV sweeps, submit one GPU job per trial and let Slurm place jobs across available GPUs:

CV_TRIAL=learning_rate__low slurm/submit.sh --task cross-validate --config config_one_percent.yaml
CV_TRIAL=learning_rate__high slurm/submit.sh --task cross-validate --config config_one_percent.yaml

Dry-run a trial submission with:

CV_TRIAL=learning_rate__low slurm/submit.sh --task cross-validate --config config_one_percent.yaml --dry-run

After the trial jobs finish, regenerate the combined CSV and plots from inside the model container or an interactive job:

cd /hct-tower/model
python -m training.cross_validate --config config/config_one_percent.yaml --summarize-only

Slurm normally reserves the requested GPU resource for a job until that job ends. For this project, prefer multiple one-GPU trial jobs over running several heavy trials on the same GPU.

Interactive Testing Before Submission

Before submitting a full batch job, it is often useful to start an interactive Slurm allocation and verify that:

  • the requested resources are available
  • apptainer loads correctly
  • the .sif image runs
  • the bind mounts and Python command behave as expected

There are two common Slurm patterns for this:

  • srun --pty /bin/bash: quickest way to land on a compute node and test commands interactively
  • salloc: fuller interactive allocation, especially useful on Sherlock when you may want to launch additional srun steps inside the allocation

For this project, the most practical choice before a GPU batch submission is usually a short interactive GPU shell.

Examples:

Yen interactive GPU shell:

srun -p gpu -G 1 -n 1 -c 4 --mem=32G -t 01:00:00 --pty /bin/bash

Sherlock interactive GPU shell:

srun -p gpu -G 1 -n 1 -c 4 --mem=32G -t 01:00:00 --pty /bin/bash

Add -C "GPU_MODEL:A40" only when a run specifically needs that GPU class.

If you specifically want a Sherlock interactive allocation rather than a one-shot shell command, the Slurm docs also support:

salloc -p gpu --gpus 1 -c 4 --mem=32G -t 01:00:00

Once the allocation starts, you can test the container directly:

module purge
module load apptainer

REPO_ROOT="${HOME}/hct-tower"
DATA_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/data"
ARTIFACT_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/artifacts"
CONTAINER_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/containers"
SIF_PATH="${CONTAINER_ROOT}/hct-tower-model-cuda.sif"

apptainer exec --nv \
  --bind "${REPO_ROOT}:/hct-tower,${DATA_ROOT}:/data,${ARTIFACT_ROOT}:/artifacts" \
  "${SIF_PATH}" \
  bash -lc "cd /hct-tower/model && python -c 'import torch; print(torch.cuda.is_available())'"

Useful quick follow-up checks inside the interactive session:

nvidia-smi
python -c "import torch; print(torch.cuda.is_available())"
slurm/submit.sh --task train --config config_samples.yaml --dry-run

This is the easiest way to debug image, mount, GPU, and config issues before moving to sbatch.

The following instructions assume you are trying to run a slurm job for the model.

1. Build the CUDA image

From inside model/:

podman build -t hct-tower-model-cuda .

2. Export the image to an archive

The tarball is just an intermediate build artifact. To avoid polluting the repo, create it in a temporary or personal working directory outside /hct-tower.

cd ~

Use an explicit Docker archive so the next step is unambiguous:

podman save --format docker-archive -o hct-tower-model-cuda.tar hct-tower-model-cuda

3. Convert the archive to a .sif

On a Stanford cluster node with the container runtime available, build the .sif:

The commands below will save the .sif to a shared folder so it simplifies the slurm scripts and can be reused across users.

The shared folder for .sif files is /zfs/projects/faculty/jungho-idhlab/drstrange/containers/.

module purge
module load apptainer

CONTAINER_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/containers"
mkdir -p "${CONTAINER_ROOT}"

apptainer build "${CONTAINER_ROOT}/hct-tower-model-cuda.sif" \
  docker-archive://$PWD/hct-tower-model-cuda.tar

You only need to rebuild the .sif when the underlying image changes.

Direct Templates

If needed, you can still submit the underlying templates directly with sbatch, but slurm/submit.sh is the recommended entrypoint.

Monitoring and Debugging

Useful commands:

sinfo -p gpu
squeue -u $USER
squeue --start
scancel <JOBID>

Training progress is environment-aware. In an interactive terminal, training uses a tqdm progress bar. In non-interactive runs such as sbatch, it skips the bar and writes plain progress lines to stdout every training.log_every_iterations iterations, defaulting to 100. You can override this with training.progress_mode: auto, bar, log, or none.

Inside a running GPU allocation, nvidia-smi is the fastest basic sanity check.

References

  • Yen Slurm overview: https://rcpedia.stanford.edu/_user_guide/slurm/
  • Yen GPU jobs: https://rcpedia.stanford.edu/_user_guide/using_gpu/
  • Yen software modules: https://rcpedia.stanford.edu/_getting_started/modules/
  • Sherlock Apptainer: https://www.sherlock.stanford.edu/docs/software/containers/apptainer/
  • Sherlock GSB partition: https://rcpedia.stanford.edu/_user_guide/sherlock/