Cancelling Slurm Jobs with scancel

Every Slurm cluster eventually needs a job stopped before it finishes on its own: a script with a bug that’s burning core-hours, a job array that was submitted with the wrong index range, or a job that’s been sitting in the queue for hours because a dependency will never resolve. The tool for all of these is scancel, Slurm’s job-cancellation command. This guide covers the exact syntax for cancelling by job ID, by array (whole array or a single task), and by username, plus what actually changes depending on whether the job is still pending or already running.

Basic syntax: cancelling a single job

The simplest form takes a job ID:

scancel 1234

This cancels job 1234 and all of its job steps, whether the job is currently pending in the queue or already running on compute nodes. You can pass more than one job ID in a single call:

scancel 1234 1235 1236

By default, scancel sends a signal that terminates the job outright. If you want to send a specific signal instead — for example, to trigger an application’s own checkpoint-and-exit handler — use -s or --signal:

scancel --signal=TERM 1234

This is also how you target an individual job step rather than the whole job: scancel --signal=TERM 1234.1 signals only step 1 of job 1234. If you want a script to write a partial results file before Slurm actually kills the job, sending SIGTERM or SIGUSR1 a few minutes ahead of the job’s time limit (Slurm can do this automatically via --signal on the sbatch submission itself) is the standard pattern — a bare scancel with no signal option does not give the application any chance to clean up.

Cancelling a job array: whole array vs. one task

Job arrays add one wrinkle to the syntax. A Slurm job array is submitted once but expands into many individual tasks, each identified by an index (Slurm reports these as <job_id>_<array_index>, for example 1237_4). scancel distinguishes between cancelling the entire array and cancelling one task within it:

• Cancel the whole array — use the base job ID by itself:

  scancel 1237

  This cancels every pending and running task belonging to array job 1237.

• Cancel one specific array task — append the underscore and the task index:

  scancel 1237_4

  This cancels only task 4 of array 1237. The other tasks in the array continue running or waiting unaffected.

This distinction matters most when you’ve caught a problem partway through a large array. If your array has 500 tasks and task index 217 is the only one that crashed and needs resubmitting after a code fix, cancel just that task (scancel 1237_217) rather than the whole array — otherwise you throw away the 216 tasks that already completed correctly, or interrupt the ones still running cleanly.

Cancelling by username

You can cancel every job belonging to a user with -u or --user:

scancel --user=jsmith

Used alone, this is a blunt instrument — it targets every job that user owns across every partition and every state. In practice it’s almost always combined with other filters to narrow the blast radius. Two of the most useful:

scancel --user=jsmith --partition=debug --state=PENDING

This cancels only jsmith‘s jobs that are (a) in the debug partition and (b) still pending — running jobs and jobs in other partitions are left alone. --state (or -t) accepts PENDING, RUNNING, or SUSPENDED.

Permissions and shared-cluster etiquette: Slurm’s own access control is straightforward — “a job or job step can only be signaled by the owner of that job or user root.” An ordinary account cannot cancel another user’s jobs; --user is normally used by a user cancelling their own jobs by name (useful when you don’t want to look up individual job IDs first), or by a cluster administrator or account coordinator with elevated privileges. If you’re a PI or lab manager who does have that kind of access on a shared cluster, treat scancel --user= on someone else’s account as an action of last resort, not routine cleanup — coordinate with the affected user first wherever possible, since a broad user-scoped cancel with no other filters will stop every job that person has in flight, including ones unrelated to whatever problem prompted the cancellation.

Pending vs. running: what actually happens

scancel‘s effect looks the same from the command line either way — the job disappears from squeue — but what happens underneath differs:

• Cancelling a pending job simply removes it from the scheduling queue. It never gets a resource allocation, consumes no compute time, and there’s nothing to clean up on a node.
• Cancelling a running job sends a termination signal to the job’s processes on every node it’s allocated. Slurm then reclaims the allocation once the processes exit (or are forcibly killed if they don’t exit promptly) and the job’s completion accounting reflects a CANCELLED state rather than COMPLETED — the job will still show up in accounting records (sacct) with whatever wall-clock and resource usage it accumulated before cancellation.

This is why the --state filter matters when you’re cleaning up in bulk: scancel --user=jsmith --state=PENDING clears a backlog of queued jobs without touching anything already consuming allocated node-hours, which is usually the safer first move before deciding whether running jobs also need to go.

Common situation: a runaway job array

A frequent real-world case is a job array submitted with a bug — a script that enters an infinite loop, an out-of-memory condition that spawns retries, or an off-by-one in the array’s index range that creates far more tasks than intended. The fix is almost always the whole-array form:

scancel 1237

This immediately stops every task in the array, both the ones still queued (preventing them from ever starting and wasting further allocation) and the ones already running. If only a subset of indices are actually misbehaving — for example, a subset of inputs that trigger the bug while the rest process fine — target those specific tasks instead, either one at a time (scancel 1237_4) or by combining scancel calls in a small loop over the known-bad indices, so the healthy tasks aren’t interrupted.

Common situation: a job stuck on a bad dependency

Slurm lets you chain jobs with --dependency so that one job doesn’t start until another reaches a given state (commonly afterok, afterany, or similar). When the job it’s waiting on fails, gets cancelled, or was never submitted correctly, the dependent job doesn’t error out on its own — it simply sits in the queue indefinitely. Running squeue on it shows the reason in the REASON column: Dependency means the job is still waiting on an unsatisfied dependency, and DependencyNeverSatisfied means Slurm has determined that dependency can no longer be met (for instance, the job it depended on already failed under an afterok condition). Either way, the stuck job is still pending and holds no resources, but it also isn’t going to run on its own — scancel is the way out:

scancel 1240

Since a dependency-stuck job is by definition still pending (it never gets far enough to be allocated resources), cancelling it is the lightweight case described above — nothing to clean up on a compute node, just a queue entry removed. After fixing whatever broke the dependency chain, resubmit rather than trying to modify the dependency on the cancelled job.

Confirming a cancellation took effect

scancel does not print confirmation on success — a clean cancel returns silently. Check that a job is actually gone with squeue:

squeue --job=1234

An empty result means the job is no longer pending or running. For a completed record of the cancellation (useful when you need to confirm a running job’s resource usage at the moment it was stopped, or when documenting cleanup for a lab notebook), check accounting history with sacct; a cancelled job shows a CANCELLED state there rather than COMPLETED or FAILED.

FAQ

Does scancel work on a job that’s already finished?

No — once a job has completed, failed, or timed out, there’s nothing left to cancel. scancel only acts on jobs still in the pending, running, or suspended states.

What’s the difference between scancel 1237 and scancel 1237_4 on an array job?

The base job ID (1237) cancels the entire array — every task, pending or running. Appending an underscore and an index (1237_4) cancels only that one task, leaving the rest of the array untouched.

Can I cancel someone else’s job?

Only if you’re that job’s owner or you have root/administrator privileges on the scheduler. Slurm enforces this at the daemon level — an ordinary user account cannot signal a job it doesn’t own, regardless of what --user value is passed.

Will scancel stop a job stuck waiting on an unmet dependency?

Yes. A dependency-blocked job is still in the pending state — squeue shows the reason as Dependency or DependencyNeverSatisfied — and scancel <job_id> removes it from the queue the same as any other pending job.

What signal does scancel send by default?

If you don’t specify -s/--signal, the job or step is terminated outright. Use --signal=<name> when you want a specific signal (such as SIGTERM or SIGUSR1) delivered instead, typically so the application can catch it and exit cleanly or write a checkpoint.

Related reading

For getting an array submitted correctly in the first place, see the guide to Slurm job arrays. For the submission script itself, see how to write an sbatch job script. If you’re weighing whether a workload should be split into an array at all, embarrassingly parallel jobs covers when that pattern fits. For institutional context on where the allocation you’re cancelling jobs against actually comes from, see NSF ACCESS national HPC allocation vs. campus recharge models, and for the broader reproducibility picture around HPC workflows, reproducibility infrastructure: workflows, containers, and code sharing.