Skip to main content
v2026.11,986 entries · CC-BY 4.0

Slurm Workload Manager: What It Does and the Commands Researchers Use

What Slurm does on a research cluster, the sinfo, sbatch, squeue, scancel and sacct commands in order, and how to read job states and pending reasons.

Written and maintained by CASRAI Editorial Board

Published

On this page

Slurm Workload Manager is an open source, fault-tolerant and highly scalable cluster management and job scheduling system for large and small Linux clusters. Researchers use it to ask a shared cluster for compute nodes, run work on them and wait their turn when many people want the same machines. You interact with it through a small set of commands: sinfo to see the machines, sbatch or srun to submit work, squeue to watch the queue, scancel to stop a job and sacct to review what a finished job used.

This page is the overview. It explains the vocabulary, walks one job through its life with the commands as the SchedMD documentation shows them, and decodes the status codes and pending reasons you will see in the queue. Each task has its own deeper guide, linked where it comes up.

What Slurm is used for in research

According to the SchedMD overview, Slurm does three things. It allocates exclusive or non-exclusive access to compute nodes to users for some duration of time so they can do work. It provides a framework for starting, executing and monitoring work, normally a parallel job, on the set of allocated nodes. And it arbitrates contention for resources by managing a queue of pending work.

For a researcher this means you do not log in to a compute node and start a program by hand. You describe what you need (how many tasks, how many nodes, how long, which partition), submit it, and the scheduler decides when and where it runs. That is why a simulation, an assembly pipeline or a model-training run can sit in a queue for a while and then start without you being present.

The vocabulary the commands rely on

The Slurm quick start defines four terms that appear in every command’s output:

  • Node: a compute resource in the cluster.
  • Partition: a logical group of nodes. Partitions may overlap and act as job queues, each with its own limits such as job size or time limit.
  • Job: an allocation of resources assigned to a user for a specified amount of time.
  • Job step: a set of, possibly parallel, tasks within a job.

The three daemons behind the commands

You never start these yourself, but their names appear in error messages and in support conversations. The overview lists slurmctld, the central manager that monitors resources and work (a backup can take over if it fails); slurmd, which runs on each compute node, waits for work, executes it and returns status; and the optional slurmdbd, the database daemon that can record accounting information for multiple Slurm-managed clusters in a single database. The accounting history that sacct reports depends on that accounting storage being enabled on your cluster.

Install or access: use your institution’s cluster

Most researchers do not install Slurm. They get an account on a cluster that their university, national laboratory or computing centre already runs, then connect to its login node and use the commands there. The quick start notes that Slurm commands can be run anywhere in the cluster. Administrators who need to install and configure Slurm themselves should use the official sources only: the download page and the administrator quick start (quickstart_admin.html) on the SchedMD site at slurm.schedmd.com.

One point catches new users. Slurm does not move your executables or data to the allocated nodes. The quick start says the files must be on local disk or on a shared file system such as NFS or Lustre, and that the sbcast command can copy a file to local storage on the allocated nodes. In practice, keep your inputs and your script on the shared file system your site provides, and check your institution’s documentation for which directories that means.

Step by step: one job from first look to review

The quick start’s command examples map to a simple sequence. Options are case sensitive, every command accepts --help, and each has a man page.

1. Look at the cluster with sinfo

sinfo reports the state of partitions and nodes. The documentation’s sample output has the columns PARTITION, AVAIL, TIMELIMIT, NODES, STATE and NODELIST, with one row per partition and state, for example a debug partition with a 30:00 time limit and a batch partition whose nodes show as alloc. An asterisk after a partition name marks the default partition. Reading the time limit here tells you the longest job a partition will accept before you write a script.

2. Submit work with sbatch (or srun)

sbatch submits a script for later execution; srun submits or starts job steps in real time. The documentation’s examples are srun -N3 -l /bin/hostname for an immediate three-node run and the following batch submission:

sbatch -n4 -w "adev[9-10]" -o my.stdout my.script
sbatch: Submitted batch job 469

Options can be set on #SBATCH lines inside the script, and options given on the command line override the ones in the script. The number printed by sbatch is the job ID that every later command uses. The full anatomy of a batch script is in how to write an sbatch job script, and the difference between the submission commands is compared in srun vs sbatch vs salloc. Slurm also supports job arrays and GPU requests, each covered in its own guide on this site.

3. Watch the queue with squeue

squeue reports the state of running and pending jobs. Running jobs are listed first, then pending ones, each group in priority order. The default format is the job ID, partition, name, user, compact state (ST), time used, number of nodes and either the node list or the reason a job is waiting. Time is shown as days-hours:minutes:seconds.

4. Cancel a job with scancel

scancel cancels a pending or running job; the quick start’s example is scancel 473. Options and safeguards are in cancelling Slurm jobs with scancel.

5. Review what the job used with sacct

sacct reports accounting data for active and completed jobs. Its --format option takes a comma-separated field list, for example sacct -j 469 --format=jobid,elapsed,ncpus,ntasks,state. Job steps are named after the job ID, such as the batch step, and array tasks appear as ArrayJobID_ArrayTaskID. sacct --helpformat lists every field. By default sacct shows jobs from 00:00:00 of the current day unless you give a start time or a job ID, so a job from yesterday needs -S or -j. A worked reading of memory and CPU figures is in sacct and seff job resource usage.

6. Inspect details with scontrol

scontrol views and, for administrators, modifies state: scontrol show partition, scontrol show node adev1 and scontrol show job display the configuration the scheduler is using. The documentation notes that many scontrol commands are available only to root.

Output and how to read it

The two columns that decide what you do next are ST, the compact job state, and NODELIST(REASON). The quick start shows R for running and PD for pending, with reasons such as Resources and Priority appearing for pending jobs. The squeue reference defines the full set of state codes; the data table below the article lists sixteen of the states, with their compact and extended names and meanings.

Reading a pending reason

Reason shown What the squeue documentation says it means Where to go next
Priority One or more higher-priority jobs exist for the partition or advanced reservation. Read how priority is calculated in Slurm priority and fairshare.
Resources The job is waiting for resources to become available. Compare your request with the node and partition limits from sinfo.
Dependency The job depends on another job whose condition has not yet been met. Check the state of the job it depends on with squeue or sacct.

Reading a finished job

After a job leaves the queue, use sacct. Useful fields from the reference are AllocCPUS (CPUs allocated), Elapsed (wall-clock time, formatted [DD-[HH:]]MM:SS), ReqMem (the minimum memory required), MaxRSS (maximum resident set size of all tasks in the job) and ExitCode (the exit code, where the number after the colon is the signal that caused a process to terminate). Comparing ReqMem with MaxRSS shows how much of the memory you asked for the job actually touched. Accounting queries require accounting storage to be enabled in the cluster’s configuration (AccountingStorageType in slurm.conf); if a site has not turned it on, sacct has nothing to report.

Troubleshooting

A job stays in PD (pending)

Read the reason in the last squeue column first. Priority means other jobs are ahead of yours; Resources means the scheduler is waiting for capacity; Dependency means an upstream job has not met its condition. None of these is an error by itself.

A job ended as TIMEOUT (TO)

TIMEOUT means the job terminated upon reaching its time limit. Compare the Elapsed field from sacct with the limit you requested and with the partition’s TIMELIMIT from sinfo.

A job ended as OUT_OF_MEMORY (OOM)

OUT_OF_MEMORY means the job experienced an out of memory error. Compare MaxRSS with ReqMem in sacct to see how far the job exceeded its request, then adjust the memory request in the script.

A job ended as FAILED (F) or NODE_FAIL (NF)

FAILED means the job terminated with a non-zero exit code or another failure condition, so the ExitCode field and your program’s own output files are the place to look. NODE_FAIL means the job terminated due to the failure of one or more allocated nodes. The documentation lists REQUEUED (RQ) as a separate state for a job that was completing and then requeued.

Commands behave differently than you expect

Remember that command-line options override #SBATCH lines in the script, that options are case sensitive, and that Slurm does not copy your files to the compute nodes. Run the command with --help or read its man page for the exact options on your installed version.

Free and open source options

Slurm is itself open source, as the overview states, so there is nothing to purchase to use it on a cluster you administer. Whether another scheduler such as PBS or LSF is a better fit usually depends on what your institution already runs, because researchers rarely choose the scheduler. For a side-by-side of the three, see Slurm vs LSF vs PBS.

Frequently asked questions

What is Slurm Workload Manager?

It is an open source cluster management and job scheduling system for Linux clusters. It allocates nodes to users, provides a framework for starting and monitoring jobs on them, and manages the queue of pending work.

Is Slurm the same as sbatch?

No. Slurm is the whole system; sbatch is one of its commands, used to submit a script for later execution. The others include srun, squeue, scancel, sinfo, scontrol and sacct.

Do I have to install Slurm to use it?

Not as a user. Access normally comes through an account on a cluster someone else administers. Administrators install it from the SchedMD download page using the administrator quick start.

What does PD mean in squeue?

PD is the compact code for PENDING: the job is waiting. The reason column explains why, for example Priority or Resources.

Why can I not find my job in sacct?

By default sacct shows jobs from the start of the current day. Give a start time with -S or a job ID with -j to see earlier jobs, and note that the cluster must have accounting storage enabled.

Does Slurm copy my data to the compute nodes?

No. Files must be on local disk of the allocated nodes or on a shared file system such as NFS or Lustre. The sbcast command can copy a file to local storage on allocated nodes.

Sources

  • SchedMD, Slurm Workload Manager overview, https://slurm.schedmd.com/overview.html (read 2026-11-15).
  • SchedMD, Slurm quick start user guide, https://slurm.schedmd.com/quickstart.html (read 2026-11-15).
  • SchedMD, squeue reference, https://slurm.schedmd.com/squeue.html (read 2026-11-15).
  • SchedMD, sacct reference, https://slurm.schedmd.com/sacct.html (read 2026-11-15).

Data: Slurm squeue job state codes (compact and extended names with meanings)

Compact and extended job state codes reported by the Slurm squeue command, with the meaning of each as stated in the SchedMD squeue reference, compiled into one lookup table for researchers reading queue output.

Slurm squeue job state codes (compact and extended names with meanings)
Compact codeExtended stateMeaning
CACANCELLEDJob was explicitly cancelled by the user or system administrator; it may or may not have been initiated.
CDCOMPLETEDJob has terminated all processes on all nodes with an exit code of zero.
CGCOMPLETINGJob is in the process of completing; some processes on some nodes may still be active.
CFCONFIGURINGJob has been allocated resources but is waiting for them to become ready for use (for example booting).
DLDEADLINEThe job terminated on its deadline.
FFAILEDThe job terminated with a non-zero exit code or other failure condition.
NFNODE_FAILThe job terminated due to failure of one or more allocated nodes.
OOMOUT_OF_MEMORYThe job experienced an out of memory error.
PDPENDINGThe job is awaiting resource allocation.
PRPREEMPTEDThe job terminated due to preemption.
RRUNNINGThe job currently has an allocation.
SSUSPENDEDJob has an allocation, but execution has been suspended and CPUs have been released for other jobs.
STSTOPPEDJob has an allocation, but execution has been stopped with a SIGSTOP signal; CPUs have been retained by the job.
TOTIMEOUTThe job terminated upon reaching its time limit.
BFBOOT_FAILJob terminated due to launch failure, typically due to a hardware failure.
RQREQUEUEDThe job was completing and then requeued.

Sources

Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about Slurm Workload Manager: What It Does and the Commands Researchers Use

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →