Slurm on Stanford Clusters (Yen / Sherlock)¶
This guide explains the recommended workflow for running the model on Stanford Slurm clusters, with examples oriented toward the Stanford GSB Yen environment.
Context¶
Relevant Stanford documentation:
- Yen Slurm overview: https://rcpedia.stanford.edu/_user_guide/slurm/
- Yen GPU jobs: https://rcpedia.stanford.edu/_user_guide/using_gpu/
- Yen software modules: https://rcpedia.stanford.edu/_getting_started/modules/
- Sherlock Apptainer: https://www.sherlock.stanford.edu/docs/software/containers/apptainer/
- Sherlock GSB partition: https://rcpedia.stanford.edu/_user_guide/sherlock/
This project may be run on both Yen and Sherlock. General lab-server Podman setup is documented in lab-containers.md. The Slurm container workflow is the same at a high level:
- build the Podman CUDA image from
model/Containerfile - export it to a container archive
- convert that archive to a
.sifimage - submit a Slurm job that runs the code inside the
.sif
This guide uses Apptainer terminology. On both Sherlock and Yen, the documented workflow for this project should be expressed in terms of apptainer.
Repo Structure¶
All cluster-execution assets live under the top-level slurm/ directory:
slurm/submit.sh: main user-facing submission wrapperslurm/common.sh: shared runtime helpers for Slurm jobsslurm/model/: model job templatesslurm/shared/: shared data-building job templatesslurm/analyses/: CPU job templates for analyses data building and tests
The recommended user workflow is to call slurm/submit.sh rather than sbatch the task templates directly.
Recommended Submission Interface¶
The wrapper keeps the user interface small while centralizing the Slurm and Apptainer logic.
Examples:
slurm/submit.sh --task train --profile worker --mode training --config config_samples.yaml
slurm/submit.sh --task train --mode full_training --config config_samples.yaml
# Start only after job 123456 completes successfully:
slurm/submit.sh --task train --mode full_training --config config_full.yaml --dependency afterok:123456
slurm/submit.sh --task train --mode full_training --config config_full.yaml --resume-latest-epoch
slurm/submit.sh --task train --mode full_training --config config_full.yaml --resume-from-checkpoint /artifacts/full/full_training/checkpoints/model_epoch1_iter12345.pt
CV_TRIAL=learning_rate__low slurm/submit.sh --task cross-validate --profile time --config config_one_percent.yaml
slurm/submit.sh --task benchmark-training --profile worker --config config_one_percent.yaml --benchmark-args '--workers 4 --prefetch-factor 2 --tensor-block-rows 32768 --period-batch-rows 500000'
slurm/submit.sh --task score --profile worker --mode training --config config_full.yaml
slurm/submit.sh --task score --profile time --mode training --config config_full.yaml
slurm/submit.sh --task score --mode full_training --config config_full.yaml
# Build destination-rank analysis datasets for the R destination-prediction task:
slurm/submit.sh --task score-destination-ranks --profile worker --mode training --config config_full.yaml
slurm/submit.sh --task score-destination-ranks --profile time --mode training --config config_full.yaml
slurm/submit.sh --task score-destination-ranks --mode full_training --config config_full.yaml
# Score and summarize the same selected user shards without overwriting full outputs:
slurm/submit.sh --task score-destination-ranks --profile worker --mode training --config config_full.yaml --shards 00,01
slurm/submit.sh --task destination-prediction --config config_full.yaml --shards 00,01
slurm/submit.sh --task score-xbrl-target-counterfactual-mq --profile worker --mode training --config config_full.yaml
slurm/submit.sh --task score-xbrl-target-counterfactual-mq --profile time --mode training --config config_full.yaml
slurm/submit.sh --task score-xbrl-target-counterfactual-mq --mode full_training --config config_full.yaml
# Export reusable company and mover embedding tables:
slurm/submit.sh --task export-embedding-tables --profile worker --target company --mode training --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile time --target company --mode training --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile worker --target workers --mode training --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --profile time --target workers --mode training --config config_full.yaml
slurm/submit.sh --task export-embedding-tables --target workers --mode full_training --config config_full.yaml
# Resume an interrupted scoring run:
slurm/submit.sh --task score --profile worker --mode training --config config_full.yaml --resume
# Resume an interrupted destination-rank scoring run:
slurm/submit.sh --task score-destination-ranks --profile worker --mode training --config config_full.yaml --resume
slurm/submit.sh --task build-processed --config config_full.yaml
slurm/submit.sh --task build-processed-sample --sample one_percent
slurm/submit.sh --task build-processed-sample --sample samples
slurm/submit.sh --task embed-about --config config_samples.yaml
slurm/submit.sh --task build-rcid-year-mq --config config_full.yaml
slurm/submit.sh --task build-disaggregated-regression-data --mode training --profile worker --config config_full.yaml --overwrite
slurm/submit.sh --task build-disaggregated-regression-data --mode training --profile time --config config_full.yaml --overwrite
slurm/submit.sh --task build-execucomp-regression-table --mode training --config config_one_percent.yaml --overwrite
slurm/submit.sh --task build-execucomp-regression-table --mode full_training --config config_one_percent.yaml --overwrite
slurm/submit.sh --task pairs-variations --config config_one_percent.yaml
slurm/submit.sh --task pairs-variations --config config_one_percent.yaml --shards 00,01
slurm/submit.sh --task destination-prediction --config config_one_percent.yaml
slurm/submit.sh --task wage-regressions --config config_one_percent.yaml
slurm/submit.sh --task execucomp-regressions --config config_one_percent.yaml
# Saved wage models are reused automatically; refit all models only if intended:
slurm/submit.sh --task wage-regressions --config config_one_percent.yaml --rerun
The analyses tasks are independent. Use --dependency to ask Slurm to hold a
job until its input-building job reaches a specified state. The value is passed
directly to sbatch; for example, --dependency afterok:123456 starts the job
only if job 123456 exits successfully. Analyses tasks use the CPU image
hct-tower-analyses-cpu.sif by default. The default
requests are 64 GB and 6 hours for the disaggregated-data builder, 256 GB and
12 hours for pair variations, 128 GB and 12 hours for destination prediction,
and 800 GB and two days for hazard estimates, model performance, and wage
regressions. The ExecuComp table
builder requests 64 GB and 24 hours; ExecuComp regressions request 64 GB and
four hours. Override these with
--mem, --time, --cpus, --partition, or --sif when needed.
For a GPU node with a driver or device fault, add --exclude <node> to any
wrapper task. For example:
slurm/submit.sh --task embed-about --config config_full.yaml --exclude gpu042
slurm/submit.sh --task train --profile worker --mode training --config config_full.yaml --exclude gpu042
The option is passed directly to sbatch, so it works for every GPU wrapper
task: embedding, training, cross-validation, benchmarking, scoring, destination
rank scoring, and smoke tests.
The builder writes the configured regression-ready dataset. It defaults to
full_training; for --mode training, also select --profile worker or
--profile time. Use --overwrite only when intentionally replacing an
existing output. For example:
slurm/submit.sh --task build-disaggregated-regression-data --mode full_training \
--config config_one_percent.yaml --dry-run
All four jobs write their Slurm stdout and stderr files to
/artifacts/slurm/<job-name>.<job-id>.{out,err}. The builder emits a live
progress bar only when attached to a terminal; under Slurm it emits periodic
newline-delimited progress records, including rows, elapsed time, and ETA.
build-processed builds the full canonical processed data from the staged US
employment-position records. build-processed-sample derives either
one_percent or samples from that full output; --sample selects the
corresponding output config and sampling rule. Use --source-config or
--config only to override those default paths. It runs on the model CUDA
image without a GPU request and defaults to 64 GB, 8 CPUs, and six hours. embed-about is a
one-time setup job for the full processed company data; sampled builds subset
that output. If the configured outputs already exist, downstream jobs can reuse
them. Rebuild only when inputs, processing logic, feature choices, or output
paths change.
The accounting-market builders are independent model-container CPU tasks. Run the raw source stage before its corresponding processed stage:
slurm/submit.sh --task build-raw-compustat --config config_full.yaml
slurm/submit.sh --task build-processed-compustat --config config_full.yaml
slurm/submit.sh --task build-raw-crsp --config config_full.yaml
slurm/submit.sh --task build-processed-crsp --config config_full.yaml
The ExecuComp builder is separate from the main disaggregated regression-table
builder. Run it once for training and once for full_training, then submit
the regression task after both compact tables are present:
slurm/submit.sh --task build-execucomp-regression-table --mode training \
--config config_full.yaml --overwrite
slurm/submit.sh --task build-execucomp-regression-table --mode full_training \
--config config_full.yaml --overwrite
slurm/submit.sh --task execucomp-regressions --config config_full.yaml
The ExecuComp builder logs one flushed line for every score shard to the Slurm
.out file. The regression runner writes a line for each fitted specification;
errors remain in the .err file.
The wrapper automatically:
- chooses the correct underlying job template
- maps the config file to
model/config/orshared/config/ - uses sensible defaults for repo, data, artifacts, and container roots
- chooses CPU vs GPU Slurm resources based on the task
- runs the job inside the shared
.sif
Task behavior itself stays in config. For example:
cross-validatereadscross_validation_search.*; setCV_TRIALto run one trialbenchmark-trainingrunstraining.benchmark_trainingand writes one JSON result to the Slurm stdout logtrain --profile PROFILE --mode trainingtrains on that profile's train+valid splits and evaluates on its test splittrain --mode full_trainingtrains once on train, validation, and test rows and writes to the shared full-training artifact root; it does not use a worker/time profiletrainaccepts direct resume flags:--resume-latest-epoch,--resume-latest-iteration, or--resume-from-checkpoint PATH; use--train-argsfor other training-entrypoint flagsscore --profile PROFILE --mode ...selects that profile's checkpoint and writes under itsscore_rootscore-destination-ranks --profile PROFILE --mode ...reads that profile's worker/company embedding archives and ranks observed movers using prior-year company employment counts. It is a GPU task.--shards 00,01restricts mover events while retaining the full candidate-firm universe and writes an isolated matchingshards/user_00_01/output. It checkpoints after each selected mover-embedding source shard, so rerun the same command with--resumeafter a wall-time limit or interruption. Do not combine--resumewith--overwrite.score-xbrl-target-counterfactual-mq --mode training --profile PROFILEreads that worker/time profile's XBRL deal panel and completed company/worker-firm-period embedding archives. It preserves separate in-sample and out-of-sample deal rows and scores each against only the matching worker split.--mode full_trainingreads the shared full-training panel and archives and writes one counterfactual summary row per deal. This is a GPU task.export-embedding-tables --profile PROFILE --target company\|workers --mode ...writes either the static company-vector table or one source-sharded worker-state population under that profile's score root. Vectors are raw per-vector int8 quantizations with FP16 reconstruction scales; checkpoint MQ dimension weights are recorded in metadata. The targets are independent: worker exports do not read or recreate the saved company table. The worker target checkpoints after each source shard; rerun it with--resumeafter interruption. Do not combine--resumewith--overwrite.wage-regressionssaves each fitted model before moving to the next one. Rerun the same command after a wall-time limit to reuse saved models and fit only missing ones. Pass--rerunonly to refit all models. Its progress lines are written to the Slurm.outfile.embed-aboutreadsabout_embeddings.*
To inspect the resolved submission command without launching a job:
slurm/submit.sh --task train --profile worker --mode training --config config_samples.yaml --dry-run
Cross-validation On Slurm¶
For CV sweeps, submit one GPU job per trial and let Slurm place jobs across available GPUs:
CV_TRIAL=learning_rate__low slurm/submit.sh --task cross-validate --profile worker --config config_one_percent.yaml
CV_TRIAL=learning_rate__high slurm/submit.sh --task cross-validate --profile time --config config_one_percent.yaml
Dry-run a trial submission with:
CV_TRIAL=learning_rate__low slurm/submit.sh --task cross-validate --profile worker --config config_one_percent.yaml --dry-run
After the trial jobs finish, regenerate the combined CSV and plots from inside the model container or an interactive job:
cd /hct-tower/model
python -m training.cross_validate --config config/config_one_percent.yaml --profile worker --summarize-only
Slurm normally reserves the requested GPU resource for a job until that job ends. For this project, prefer multiple one-GPU trial jobs over running several heavy trials on the same GPU.
Interactive Testing Before Submission¶
Before submitting a full batch job, it is often useful to start an interactive Slurm allocation and verify that:
- the requested resources are available
apptainerloads correctly- the
.sifimage runs - the bind mounts and Python command behave as expected
There are two common Slurm patterns for this:
srun --pty /bin/bash: quickest way to land on a compute node and test commands interactivelysalloc: fuller interactive allocation, especially useful on Sherlock when you may want to launch additionalsrunsteps inside the allocation
For this project, the most practical choice before a GPU batch submission is usually a short interactive GPU shell.
Examples:
Yen interactive GPU shell:
Sherlock interactive GPU shell:
Add -C "GPU_MODEL:A40" only when a run specifically needs that GPU class.
If you specifically want a Sherlock interactive allocation rather than a one-shot shell command, the Slurm docs also support:
Once the allocation starts, you can test the container directly:
module purge
module load apptainer
REPO_ROOT="${HOME}/hct-tower"
DATA_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/data"
ARTIFACT_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/artifacts"
CONTAINER_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/containers"
SIF_PATH="${CONTAINER_ROOT}/hct-tower-model-cuda.sif"
apptainer exec --nv \
--bind "${REPO_ROOT}:/hct-tower,${DATA_ROOT}:/data,${ARTIFACT_ROOT}:/artifacts" \
"${SIF_PATH}" \
bash -lc "cd /hct-tower/model && python -c 'import torch; print(torch.cuda.is_available())'"
Useful quick follow-up checks inside the interactive session:
nvidia-smi
python -c "import torch; print(torch.cuda.is_available())"
slurm/submit.sh --task train --profile worker --config config_samples.yaml --dry-run
This is the easiest way to debug image, mount, GPU, and config issues before moving to sbatch.
Recommended Image Workflow¶
The following instructions assume you are trying to run a slurm job for the model.
1. Build the CUDA image¶
From inside model/:
2. Export the image to an archive¶
The tarball is just an intermediate build artifact. To avoid polluting the repo, create it in a temporary or personal working directory outside /hct-tower.
Use an explicit Docker archive so the next step is unambiguous:
3. Convert the archive to a .sif¶
On a Stanford cluster node with the container runtime available, build the .sif:
The commands below will save the .sif to a shared folder so it simplifies the slurm scripts and can be reused across users.
The shared folder for .sif files is /zfs/projects/faculty/jungho-idhlab/drstrange/containers/.
module purge
module load apptainer
CONTAINER_ROOT="/zfs/projects/faculty/jungho-idhlab/drstrange/containers"
mkdir -p "${CONTAINER_ROOT}"
apptainer build "${CONTAINER_ROOT}/hct-tower-model-cuda.sif" \
docker-archive://$PWD/hct-tower-model-cuda.tar
You only need to rebuild the .sif when the underlying image changes.
Direct Templates¶
If needed, you can still submit the underlying templates directly with sbatch, but slurm/submit.sh is the recommended entrypoint.
Monitoring and Debugging¶
Useful commands:
Training progress is environment-aware. In an interactive terminal, training
uses a tqdm progress bar. In non-interactive runs such as sbatch, it skips
the bar and writes plain progress lines to stdout every
training.log_every_iterations iterations, defaulting to 100. You can override
this with training.progress_mode: auto, bar, log, or none.
Inside a running GPU allocation, nvidia-smi is the fastest basic sanity check.
References¶
- Yen Slurm overview: https://rcpedia.stanford.edu/_user_guide/slurm/
- Yen GPU jobs: https://rcpedia.stanford.edu/_user_guide/using_gpu/
- Yen software modules: https://rcpedia.stanford.edu/_getting_started/modules/
- Sherlock Apptainer: https://www.sherlock.stanford.edu/docs/software/containers/apptainer/
- Sherlock GSB partition: https://rcpedia.stanford.edu/_user_guide/sherlock/