Course contentsShow
← Machine Learning and Deep Learning
Lesson 2799 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Cluster Resource Management: SLURM

Using SLURM to schedule and manage multi-node training jobs on HPC clusters: job scripts and resource allocation.

Cluster Resource Management: SLURM

What you'll learn: How to use SLURM to schedule and manage multi-node distributed training jobs on high-performance computing clusters.

What is SLURM?

SLURM (Simple Linux Utility for Resource Management) is a job scheduler that manages access to shared computing resources on HPC clusters. Think of it as an air traffic controller for compute jobs—it decides which jobs run on which nodes, when they run, and ensures fair resource allocation among users.

When you're training large models across multiple nodes, you can't just SSH into machines and start processes manually. SLURM provides the orchestration layer that allocates GPUs, CPUs, and memory across nodes, then launches your distributed training processes in a coordinated fashion.

SLURM Job Scripts

A typical SLURM job script specifies resource requirements and the commands to execute:

#!/bin/bash
#SBATCH --job-name=my_training
#SBATCH --nodes=4
#SBATCH --gres=gpu:8
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32
#SBATCH --time=24:00:00

# SLURM sets SLURM_PROCID, SLURM_NODEID, etc.
srun torchrun --nnodes=4 --nproc_per_node=8 train.py

The #SBATCH directives request 4 nodes with 8 GPUs each. SLURM then coordinates launching your torchrun command across all allocated nodes simultaneously, automatically setting environment variables that your training script uses to establish process groups and ranks.

Resource Allocation Strategy

SLURM helps you match workload requirements to available hardware—requesting appropriate GPU counts, memory, and walltime ensures efficient cluster utilization and faster queue times.

Key Takeaway: SLURM automates multi-node job scheduling by managing resource allocation and coordinated process launching, making distributed training on HPC clusters practical and reproducible.