Written and maintained by CASRAI Editorial Board
Last updated
Submitting a separate sbatch job for every sample in a batch works — right up until the batch is 200 samples deep, at which point you’ve handed the scheduler 200 individual jobs to queue, prioritize, and dispatch, and yourself 200 near-identical submissions to keep track of. A Slurm job array replaces that submit loop with a single sbatch call that creates many nearly-identical tasks, indexed so each one can select its own slice of the input. This guide covers the --array directive syntax, how to use SLURM_ARRAY_TASK_ID inside a script to pick per-task input, why arrays are lighter on the scheduler than a loop of individual submissions, and the array-specific controls for throttling concurrency and cancelling tasks.
What a Slurm Job Array Actually Is
A job array is one sbatch submission that produces a set of tasks sharing the same script, the same resource request (CPUs, memory, time limit, partition), and the same job name — but each task runs with a different numeric index available to it as an environment variable. Slurm represents the whole array as a single job internally, with individual tasks distinguished by that index, rather than as N unrelated job records. That’s the structural reason arrays scale better than a submission loop, covered in more detail below.
Arrays are the standard pattern anywhere you have “the same computation, run once per item in a list” — aligning 200 sequencing samples against a reference, running a parameter sweep across 50 configuration values, or bootstrapping 1,000 replicate simulations. If you’re instead orchestrating a multi-step pipeline with dependencies between stages, a workflow manager like the one compared in Snakemake vs Nextflow often generates array-style submissions for you under the hood; a job array on its own is the right tool for a flat, embarrassingly-parallel batch of independent tasks.
The –array Directive Syntax
Per Slurm’s own documentation, sbatch‘s --array (short form -a) option accepts several index formats:
- A simple range:
--array=0-31creates 32 tasks, indexed 0 through 31. - A range with a step:
--array=1-7:2creates tasks 1, 3, 5, 7 — every second index in the range. - A comma-separated list:
--array=1,3,5,7creates exactly those four indices, no others. - A throttle on concurrent tasks:
--array=0-15%4creates 16 tasks (0 through 15) but limits Slurm to running at most 4 of them at once — the rest queue and start as running tasks finish. This is covered in its own section below.
The smallest valid index is 0. The largest index a cluster will accept is governed by the MaxArraySize parameter in slurm.conf; Slurm’s documentation lists a default of 1001, but this is a site-configurable value that individual clusters commonly raise or lower. Don’t assume any specific ceiling — check your own cluster with scontrol show config | grep MaxArraySize before sizing an array, rather than discovering the limit from a rejected submission.
The directive can be set in the script header (#SBATCH --array=0-199) or passed on the command line at submission time (sbatch --array=0-199 script.sh), the same as any other sbatch option — a command-line flag overrides one set in the script.
Indexing Into Input Data With SLURM_ARRAY_TASK_ID
Each task in the array runs the identical script, but Slurm exports SLURM_ARRAY_TASK_ID into that task’s environment, set to its own index. The script reads this variable to decide what it’s actually supposed to work on for that particular task — most commonly by using it to pull one line out of a list file, one element from a bash array, or one row from a manifest.
The most common pattern, and the one used in the worked example below, is a plain-text list with one item per line, read with sed:
SAMPLE=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" samples.txt)sed -n "Np" prints line N, and sed lines are 1-indexed while SLURM_ARRAY_TASK_ID starts at 0 by default — hence the + 1. (You can sidestep that arithmetic entirely by submitting --array=1-200 instead of 0-199, so the task index matches the line number directly.) The same indexing idea works with a native bash array:
SAMPLES=(sample_A sample_B sample_C)
SAMPLE=${SAMPLES[$SLURM_ARRAY_TASK_ID]}— in which case bash’s own zero-based array indexing lines up with Slurm’s zero-based task indexing without any offset needed.
Worked Example: 200 Samples, One Submission
Given samples.txt with one sample ID per line (200 lines total) and paired FASTQ files named <sample>_R1.fastq.gz / <sample>_R2.fastq.gz, this script aligns every sample against a reference genome, submitted once:
#!/bin/bash
#SBATCH --job-name=align_samples
#SBATCH --array=0-199%20
#SBATCH --output=logs/align_%A_%a.out
#SBATCH --error=logs/align_%A_%a.err
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=02:00:00
SAMPLE=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" samples.txt)
echo "Task ${SLURM_ARRAY_TASK_ID} of ${SLURM_ARRAY_TASK_COUNT}: processing ${SAMPLE}"
bwa mem reference.fa "${SAMPLE}_R1.fastq.gz" "${SAMPLE}_R2.fastq.gz"
> "aligned/${SAMPLE}.sam"Submitted once, from the shell, with no loop at all:
sbatch align_samples.shTwo details worth pointing out. First, --output=logs/align_%A_%a.out uses Slurm’s array-specific filename substitutions: %A expands to the array job ID (shared by every task) and %a expands to that task’s own array index — so each task writes to its own log file (align_38512_0.out, align_38512_1.out, and so on) instead of every task fighting over the same slurm-%j.out. Second, %20 on the --array line caps this array at 20 tasks running simultaneously, which is a submission-time choice to be a reasonable neighbor on a shared cluster, not a Slurm default — size it to what your allocation and your cluster’s fair-use norms actually support. Nothing here required writing 200 separate sbatch invocations or hand-generating 200 near-identical scripts.
Why This Is Kinder to the Scheduler Than a Submit Loop
A shell loop that calls sbatch once per sample creates 200 completely independent job records for the controller to track, prioritize against every other job in the queue, and schedule individually — each carrying its own overhead in the scheduler’s bookkeeping and its own entry in squeue‘s output. A job array submits once and is represented internally as a single job with many tasks, which is materially less bookkeeping for the controller on a busy shared cluster with many users’ jobs competing for the same nodes. It’s also easier to reason about and administer as a unit: one job ID identifies the whole batch (with each task addressable as jobid_taskid, covered below), your priority is computed once rather than 200 separate times, and — as the throttling section below covers — you get a single, simple lever to control how much of the cluster your batch claims at once, instead of manually pacing 200 individual sbatch calls to avoid hammering the queue. On a shared, allocation-based system like the ones described in NSF ACCESS HPC allocations, that difference in how considerately a batch uses the scheduler is not just a courtesy — it directly affects how your jobs are treated relative to everyone else’s.
Array Environment Variables Reference
Slurm exports five array-specific variables into every task’s environment. Only SLURM_ARRAY_TASK_ID is genuinely different per task — the other four describe the array as a whole and are identical across every task in it:
| Variable | Meaning |
|---|---|
SLURM_ARRAY_JOB_ID |
The array’s own job ID — the first job ID assigned to the array, shared by every task in it. |
SLURM_ARRAY_TASK_ID |
This task’s own index value — the one piece of per-task information the array exists to provide. |
SLURM_ARRAY_TASK_COUNT |
The total number of tasks in the array. |
SLURM_ARRAY_TASK_MIN |
The lowest index value in the array. |
SLURM_ARRAY_TASK_MAX |
The highest index value in the array. |
SLURM_JOB_ID (without ARRAY in the name — the ordinary variable every Slurm job gets) is set to that specific task’s own unique job ID, which differs from SLURM_ARRAY_JOB_ID; the combination is what appears in the jobid_taskid notation Slurm uses to address one array element, e.g. 36_1 for task 1 of array job 36.
Limiting Concurrent Tasks With %N
The %N suffix on --array is the throttle covered briefly above: --array=0-99%10 creates 100 tasks but tells Slurm to never run more than 10 of them at the same time, regardless of how many nodes are otherwise free. Tasks beyond the limit queue in a pending state and start automatically as running tasks complete — no manual resubmission or babysitting required.
This matters for two distinct reasons. On a shared cluster, it’s straightforward etiquette — a 500-task array with no cap can flood every idle node the moment it’s dispatched, starving other users’ work (and your own other jobs) until it finishes. And even on a cluster you have to yourself, an unthrottled array can exceed a downstream resource that has nothing to do with Slurm’s own scheduling — a database connection limit, a shared filesystem’s I/O ceiling, a license server with a fixed seat count, or a third-party API’s rate limit. Setting %N to something the actual bottleneck can sustain is often the real fix in that situation, not requesting more nodes.
Cancelling Array Jobs
Because Slurm treats an array as one job with addressable tasks, scancel can target the whole thing or just a piece of it:
scancel 20— cancels every task in array job 20, running or still pending.scancel 20_4— cancels only task 4 of array job 20, leaving the rest running.scancel 20_4 20_5— cancels multiple specific tasks in one call by listing severaljobid_taskidpairs.scancel 20_[1-3]— cancels a contiguous range of tasks (1 through 3) using Slurm’s bracket range syntax.
This is the practical payoff of the array structure showing up again: if a bug surfaces partway through a 200-task run — say tasks 150 onward are failing on a bad input file — you can cancel exactly the affected range with one scancel call, fix the input, and resubmit just that slice as its own array, rather than hunting down and cancelling 50 individually-tracked job IDs.
Checking on a Running Array
squeue -j <jobid> shows the status of an entire array (Slurm’s display may collapse many pending tasks with the same state into a single summarized line); appending the task index — squeue -j 20_4 — filters to one specific task. Once the array has finished, sacct -j <jobid> reports per-task exit codes and resource usage, which is generally the fastest way to find the handful of tasks that failed out of a few hundred, rather than scrolling through log files one at a time.
Frequently Asked Questions
Do all tasks in an array request the same resources?
Yes — the #SBATCH resource directives (CPUs, memory, time limit, partition, GPUs) apply identically to every task in the array. If different items in your batch genuinely need different resource requests, a job array isn’t the right tool for that batch; either split it into resource-homogeneous sub-batches or have your script request a generous-enough ceiling for the largest case.
What happens if one task in the array fails?
By default, the other tasks are unaffected — each task in an array is an independent job as far as success or failure is concerned, so one failing task does not stop, cancel, or pause the rest. This is different from a workflow manager’s dependency graph (see Snakemake vs Nextflow for that comparison), where a failed step can legitimately block downstream steps. Check sacct after the array finishes to find which specific task indices failed.
Is there a limit to how large a job array can be?
Yes, governed by MaxArraySize in slurm.conf. This is a cluster-configurable value, so treat any specific number you’ve seen quoted as that cluster’s setting, not a universal ceiling — confirm your own cluster’s value with scontrol show config | grep MaxArraySize before assuming an array of a given size will be accepted.
Can I run each array task inside a container?
Yes — a job array’s script can invoke a container runtime the same way any other Slurm job script can, with SLURM_ARRAY_TASK_ID used exactly as shown above to select the per-task input passed into the container. See Apptainer vs Docker for the tradeoffs between runtimes in an HPC context specifically, since Docker’s daemon model is a poor fit for most shared academic clusters.
Does SLURM_ARRAY_TASK_ID start at 0 or 1?
Whatever you specify in --array — there’s no fixed starting point. --array=0-199 starts at 0; --array=1-200 starts at 1. Pick whichever aligns more naturally with how your input is indexed (a 0-indexed bash array vs. a 1-indexed line-numbered file), rather than always defaulting to one and adding offset arithmetic in the script.
Why would I use %N instead of just requesting fewer nodes?
%N throttles how many array tasks run concurrently, not how many resources each task requests — it’s the right tool when the constraint is a task count (database connections, license seats, API rate limits, cluster courtesy) rather than a per-task resource size. If the actual problem is that each task itself is too large, that’s a --cpus-per-task/--mem adjustment, a separate concern from %N.
