Written and maintained by CASRAI Editorial Board
Published
On this page
Slurm Workload Manager is an open source, fault-tolerant and highly scalable cluster management and job scheduling system for large and small Linux clusters. Researchers use it to ask a shared cluster for compute nodes, run work on them and wait their turn when many people want the same machines. You interact with it through a small set of commands: sinfo to see the machines, sbatch or srun to submit work, squeue to watch the queue, scancel to stop a job and sacct to review what a finished job used.
This page is the overview. It explains the vocabulary, walks one job through its life with the commands as the SchedMD documentation shows them, and decodes the status codes and pending reasons you will see in the queue. Each task has its own deeper guide, linked where it comes up.
What Slurm is used for in research
According to the SchedMD overview, Slurm does three things. It allocates exclusive or non-exclusive access to compute nodes to users for some duration of time so they can do work. It provides a framework for starting, executing and monitoring work, normally a parallel job, on the set of allocated nodes. And it arbitrates contention for resources by managing a queue of pending work.
For a researcher this means you do not log in to a compute node and start a program by hand. You describe what you need (how many tasks, how many nodes, how long, which partition), submit it, and the scheduler decides when and where it runs. That is why a simulation, an assembly pipeline or a model-training run can sit in a queue for a while and then start without you being present.
The vocabulary the commands rely on
The Slurm quick start defines four terms that appear in every command’s output:
- Node: a compute resource in the cluster.
- Partition: a logical group of nodes. Partitions may overlap and act as job queues, each with its own limits such as job size or time limit.
- Job: an allocation of resources assigned to a user for a specified amount of time.
- Job step: a set of, possibly parallel, tasks within a job.
The three daemons behind the commands
You never start these yourself, but their names appear in error messages and in support conversations. The overview lists slurmctld, the central manager that monitors resources and work (a backup can take over if it fails); slurmd, which runs on each compute node, waits for work, executes it and returns status; and the optional slurmdbd, the database daemon that can record accounting information for multiple Slurm-managed clusters in a single database. The accounting history that sacct reports depends on that accounting storage being enabled on your cluster.
Install or access: use your institution’s cluster
Most researchers do not install Slurm. They get an account on a cluster that their university, national laboratory or computing centre already runs, then connect to its login node and use the commands there. The quick start notes that Slurm commands can be run anywhere in the cluster. Administrators who need to install and configure Slurm themselves should use the official sources only: the download page and the administrator quick start (quickstart_admin.html) on the SchedMD site at slurm.schedmd.com.
One point catches new users. Slurm does not move your executables or data to the allocated nodes. The quick start says the files must be on local disk or on a shared file system such as NFS or Lustre, and that the sbcast command can copy a file to local storage on the allocated nodes. In practice, keep your inputs and your script on the shared file system your site provides, and check your institution’s documentation for which directories that means.
Step by step: one job from first look to review
The quick start’s command examples map to a simple sequence. Options are case sensitive, every command accepts --help, and each has a man page.
1. Look at the cluster with sinfo
sinfo reports the state of partitions and nodes. The documentation’s sample output has the columns PARTITION, AVAIL, TIMELIMIT, NODES, STATE and NODELIST, with one row per partition and state, for example a debug partition with a 30:00 time limit and a batch partition whose nodes show as alloc. An asterisk after a partition name marks the default partition. Reading the time limit here tells you the longest job a partition will accept before you write a script.
2. Submit work with sbatch (or srun)
sbatch submits a script for later execution; srun submits or starts job steps in real time. The documentation’s examples are srun -N3 -l /bin/hostname for an immediate three-node run and the following batch submission:
sbatch -n4 -w "adev[9-10]" -o my.stdout my.script
sbatch: Submitted batch job 469
Options can be set on #SBATCH lines inside the script, and options given on the command line override the ones in the script. The number printed by sbatch is the job ID that every later command uses. The full anatomy of a batch script is in how to write an sbatch job script, and the difference between the submission commands is compared in srun vs sbatch vs salloc. Slurm also supports job arrays and GPU requests, each covered in its own guide on this site.
3. Watch the queue with squeue
squeue reports the state of running and pending jobs. Running jobs are listed first, then pending ones, each group in priority order. The default format is the job ID, partition, name, user, compact state (ST), time used, number of nodes and either the node list or the reason a job is waiting. Time is shown as days-hours:minutes:seconds.
4. Cancel a job with scancel
scancel cancels a pending or running job; the quick start’s example is scancel 473. Options and safeguards are in cancelling Slurm jobs with scancel.
5. Review what the job used with sacct
sacct reports accounting data for active and completed jobs. Its --format option takes a comma-separated field list, for example sacct -j 469 --format=jobid,elapsed,ncpus,ntasks,state. Job steps are named after the job ID, such as the batch step, and array tasks appear as ArrayJobID_ArrayTaskID. sacct --helpformat lists every field. By default sacct shows jobs from 00:00:00 of the current day unless you give a start time or a job ID, so a job from yesterday needs -S or -j. A worked reading of memory and CPU figures is in sacct and seff job resource usage.
6. Inspect details with scontrol
scontrol views and, for administrators, modifies state: scontrol show partition, scontrol show node adev1 and scontrol show job display the configuration the scheduler is using. The documentation notes that many scontrol commands are available only to root.
Output and how to read it
The two columns that decide what you do next are ST, the compact job state, and NODELIST(REASON). The quick start shows R for running and PD for pending, with reasons such as Resources and Priority appearing for pending jobs. The squeue reference defines the full set of state codes; the data table below the article lists sixteen of the states, with their compact and extended names and meanings.
Reading a pending reason
| Reason shown | What the squeue documentation says it means | Where to go next |
|---|---|---|
| Priority | One or more higher-priority jobs exist for the partition or advanced reservation. | Read how priority is calculated in Slurm priority and fairshare. |
| Resources | The job is waiting for resources to become available. | Compare your request with the node and partition limits from sinfo. |
| Dependency | The job depends on another job whose condition has not yet been met. | Check the state of the job it depends on with squeue or sacct. |
Reading a finished job
After a job leaves the queue, use sacct. Useful fields from the reference are AllocCPUS (CPUs allocated), Elapsed (wall-clock time, formatted [DD-[HH:]]MM:SS), ReqMem (the minimum memory required), MaxRSS (maximum resident set size of all tasks in the job) and ExitCode (the exit code, where the number after the colon is the signal that caused a process to terminate). Comparing ReqMem with MaxRSS shows how much of the memory you asked for the job actually touched. Accounting queries require accounting storage to be enabled in the cluster’s configuration (AccountingStorageType in slurm.conf); if a site has not turned it on, sacct has nothing to report.
Troubleshooting
A job stays in PD (pending)
Read the reason in the last squeue column first. Priority means other jobs are ahead of yours; Resources means the scheduler is waiting for capacity; Dependency means an upstream job has not met its condition. None of these is an error by itself.
A job ended as TIMEOUT (TO)
TIMEOUT means the job terminated upon reaching its time limit. Compare the Elapsed field from sacct with the limit you requested and with the partition’s TIMELIMIT from sinfo.
A job ended as OUT_OF_MEMORY (OOM)
OUT_OF_MEMORY means the job experienced an out of memory error. Compare MaxRSS with ReqMem in sacct to see how far the job exceeded its request, then adjust the memory request in the script.
A job ended as FAILED (F) or NODE_FAIL (NF)
FAILED means the job terminated with a non-zero exit code or another failure condition, so the ExitCode field and your program’s own output files are the place to look. NODE_FAIL means the job terminated due to the failure of one or more allocated nodes. The documentation lists REQUEUED (RQ) as a separate state for a job that was completing and then requeued.
Commands behave differently than you expect
Remember that command-line options override #SBATCH lines in the script, that options are case sensitive, and that Slurm does not copy your files to the compute nodes. Run the command with --help or read its man page for the exact options on your installed version.
Free and open source options
Slurm is itself open source, as the overview states, so there is nothing to purchase to use it on a cluster you administer. Whether another scheduler such as PBS or LSF is a better fit usually depends on what your institution already runs, because researchers rarely choose the scheduler. For a side-by-side of the three, see Slurm vs LSF vs PBS.
Frequently asked questions
What is Slurm Workload Manager?
It is an open source cluster management and job scheduling system for Linux clusters. It allocates nodes to users, provides a framework for starting and monitoring jobs on them, and manages the queue of pending work.
Is Slurm the same as sbatch?
No. Slurm is the whole system; sbatch is one of its commands, used to submit a script for later execution. The others include srun, squeue, scancel, sinfo, scontrol and sacct.
Do I have to install Slurm to use it?
Not as a user. Access normally comes through an account on a cluster someone else administers. Administrators install it from the SchedMD download page using the administrator quick start.
What does PD mean in squeue?
PD is the compact code for PENDING: the job is waiting. The reason column explains why, for example Priority or Resources.
Why can I not find my job in sacct?
By default sacct shows jobs from the start of the current day. Give a start time with -S or a job ID with -j to see earlier jobs, and note that the cluster must have accounting storage enabled.
Does Slurm copy my data to the compute nodes?
No. Files must be on local disk of the allocated nodes or on a shared file system such as NFS or Lustre. The sbcast command can copy a file to local storage on allocated nodes.
Sources
- SchedMD, Slurm Workload Manager overview, https://slurm.schedmd.com/overview.html (read 2026-11-15).
- SchedMD, Slurm quick start user guide, https://slurm.schedmd.com/quickstart.html (read 2026-11-15).
- SchedMD, squeue reference, https://slurm.schedmd.com/squeue.html (read 2026-11-15).
- SchedMD, sacct reference, https://slurm.schedmd.com/sacct.html (read 2026-11-15).
Data: Slurm squeue job state codes (compact and extended names with meanings)
Compact and extended job state codes reported by the Slurm squeue command, with the meaning of each as stated in the SchedMD squeue reference, compiled into one lookup table for researchers reading queue output.
| Compact code | Extended state | Meaning |
|---|---|---|
| CA | CANCELLED | Job was explicitly cancelled by the user or system administrator; it may or may not have been initiated. |
| CD | COMPLETED | Job has terminated all processes on all nodes with an exit code of zero. |
| CG | COMPLETING | Job is in the process of completing; some processes on some nodes may still be active. |
| CF | CONFIGURING | Job has been allocated resources but is waiting for them to become ready for use (for example booting). |
| DL | DEADLINE | The job terminated on its deadline. |
| F | FAILED | The job terminated with a non-zero exit code or other failure condition. |
| NF | NODE_FAIL | The job terminated due to failure of one or more allocated nodes. |
| OOM | OUT_OF_MEMORY | The job experienced an out of memory error. |
| PD | PENDING | The job is awaiting resource allocation. |
| PR | PREEMPTED | The job terminated due to preemption. |
| R | RUNNING | The job currently has an allocation. |
| S | SUSPENDED | Job has an allocation, but execution has been suspended and CPUs have been released for other jobs. |
| ST | STOPPED | Job has an allocation, but execution has been stopped with a SIGSTOP signal; CPUs have been retained by the job. |
| TO | TIMEOUT | The job terminated upon reaching its time limit. |
| BF | BOOT_FAIL | Job terminated due to launch failure, typically due to a hardware failure. |
| RQ | REQUEUED | The job was completing and then requeued. |
Sources
- SchedMD squeue reference (Job State Codes) (accessed 2026-11-15)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







