Requesting a GPU in a Slurm Batch Job

Requesting a GPU in Slurm looks simple until a job sits in PENDING for an explainable reason, or runs to completion on a $3/hour A100 node without ever touching the card. The directive you choose (--gres=gpu:N vs. --gpus=N), whether you pin a specific GPU model, and whether your environment actually loads a CUDA toolkit all matter before the science ever starts. This guide covers the request syntax, the differences between the older and newer directive families, how to target a specific GPU type, the two mistakes that burn the most GPU-hours for nothing, and how to confirm — from inside the running job — that the GPU you asked for is the GPU you got.

If you haven’t written a Slurm batch script before, start with how to write an sbatch job script for the directive basics (partitions, time limits, memory) this guide builds on.

–gres=gpu:N vs. –gpus=N: what’s actually different

Slurm has accumulated two overlapping ways to ask for a GPU, and they are not interchangeable in how they count.

--gres=gpu:N is the older, general-purpose “generic resource” syntax, shared with any consumable resource a site defines (not just GPUs). The full form is --gres=gpu[:type]:count — for example --gres=gpu:2 requests 2 GPUs, and it is a per-node request: on a multi-node job, each allocated node gets that many GPUs, not the job as a whole.

--gpus=N (short form -G) is the newer, GPU-specific option, added specifically to remove the per-node ambiguity. --gpus=[type:]count specifies the total GPUs the job needs across all allocated nodes, and Slurm’s scheduler works out the per-node split itself. For a single-node job the two are equivalent in effect; for a multi-node job they are not, and mixing up “per node” and “total” is a common source of jobs that allocate far more (or fewer) GPUs than intended.

A related pair of directives constrains how much CPU and memory rides along with each GPU: --cpus-per-gpu=N and --mem-per-gpu=size. Both are mutually exclusive with their non-GPU equivalents (--cpus-per-task and --mem/--mem-per-cpu respectively) — pick one style per resource, not both.

The newer per-node and per-task forms

Two further directives split the same total differently, and both accept an optional GPU type ahead of the count:

• --gpus-per-node=[type:]count — GPUs per allocated node, useful when you want an explicit, guaranteed-even split across nodes rather than letting the scheduler decide. It is mutually exclusive with --gres=gpu — use one or the other, never both in the same submission.
• --gpus-per-task=[type:]count — GPUs per spawned task, for jobs launched with multiple tasks (typically via srun inside the script). This requires an explicit task count, set with --ntasks or by pairing it with --gpus. It also implicitly binds each task to its own GPU (setting Slurm’s task-to-GPU binding) unless you override that behavior — the practical effect is that rank 0 sees one GPU, rank 1 sees another, without you writing that binding logic yourself. This is the directive to reach for on a multi-GPU, multi-process job — the kind of workload covered in embarrassingly parallel jobs when each task is independent, or a distributed-training job when the tasks cooperate.

A less common sibling, --gpus-per-socket=[type:]count, ties the request to CPU sockets and requires --sockets-per-node to also be set — most users won’t need it outside NUMA-sensitive placement tuning.

If you’re running the same GPU job many times over a parameter sweep rather than manually resubmitting, a Slurm job array applies the same GPU request line across every array index without duplicating the script.

Requesting a specific GPU type or model

Clusters with mixed GPU generations — A100s in one partition, V100s or older cards in another — let you name the type inside the same directives rather than adding a separate flag. The type is the first field before the count: --gres=gpu:a100:2 requests two GPUs of the a100 type, and --gpus=a100:2 does the same via the newer syntax. The exact type string is site-defined, not standardized by Slurm itself — one cluster’s a100 might be another’s nvidia_a100 or gpu:a100_80gb. Run sinfo -o "%P %G" (or check your cluster’s documentation) before assuming a type name; guessing wrong produces an allocation that either fails outright or silently lands you on the wrong hardware.

Some sites instead expose GPU model as a node feature rather than a gres type, selected with --constraint=a100 alongside a plain --gpus=2. Which convention a given cluster uses is a local decision — this is exactly the kind of directive that has no standard naming across institutions, so a script that works on one cluster can fail on another without a rewrite.

Two mistakes that waste GPU-hours

Both of the following produce a job that runs, exits 0, and consumes a full GPU allocation while doing nothing useful with it — the failure mode that’s hardest to catch because nothing errors.

Requesting a GPU your code never calls

--gres=gpu:1 makes a GPU visible to the job. It does not make your program use it. A script running stock CPU-only NumPy, a PyTorch build installed without CUDA support, or an R session with no GPU-aware package loaded will run to completion on the CPU alone while the GPU sits idle for the full walltime — and the scheduler still bills the job for GPU-hours, and still holds the node against everyone else’s queue priority. Before submitting at scale, confirm the library you’re calling is actually the CUDA-enabled build: for a conda environment, that usually means the CUDA-toolkit-linked package variant, not the default channel build — see conda, pip, and mamba in one environment for how build variants collide when mixed installers are involved. Wasted GPU-hours are also a real sustainability line item on shared HPC infrastructure — see sustainable HPC — and most allocation programs meter and charge for them the same as used ones, per how HPC allocations get charged.

Forgetting to load a CUDA module

Requesting hardware and loading the software stack to drive it are separate steps. Most clusters expose CUDA through an environment module system, and the job script has to load it explicitly — a line like module load cuda/12.4 (the exact name and version are site-specific; module avail cuda lists what’s installed) — before any CUDA-linked binary or Python import will find the driver and runtime libraries it needs. Skip that line and a program built against CUDA typically fails immediately with a library-not-found or “no CUDA-capable device” error, even though the GPU was correctly allocated and is sitting right there. Container-based workflows sidestep this by bundling the CUDA runtime inside the image itself — see Apptainer vs Docker for how GPU passthrough differs between the two on shared HPC systems — which is one reason GPU-heavy pipelines lean on containers more than CPU-only ones do.

Verifying the GPU is actually visible inside the job

The one check that catches both mistakes above, and any allocation problem besides, is running nvidia-smi as the first real command in the job — before the expensive part of the script starts, not after:

#!/bin/bash
#SBATCH --job-name=gpu-check
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-gpu=4
#SBATCH --mem-per-gpu=16G
#SBATCH --time=00:10:00

module load cuda/12.4

echo "Allocated GPU(s): $SLURM_JOB_GPUS"
nvidia-smi

python train.py

nvidia-smi with no arguments prints the driver version, CUDA version the driver supports, and a table of every GPU visible to the job with its memory and current utilization — if that table is empty or the command isn’t found, the allocation or the module load failed before your program ever ran, and you’ve found the problem in ten seconds of walltime instead of after a multi-hour job silently ran on CPU. The $SLURM_JOB_GPUS environment variable (and, inside a step launched with srun, $CUDA_VISIBLE_DEVICES) confirms which physical device index or indices Slurm actually bound to the job — worth echoing whenever a script behaves differently across otherwise-identical submissions, since it’s the fastest way to catch a job that landed on a different GPU type than intended. For an interactive check outside a batch script, srun --partition=gpu --gres=gpu:1 --pty nvidia-smi gives the same confirmation without writing a script at all.

FAQ

Should I use –gres=gpu or –gpus for a single-node job?

For a single-node job the two produce the same allocation, so either works. --gpus is the more current, purpose-built option and reads more clearly in a script someone else has to maintain; --gres=gpu remains the more portable choice on older Slurm versions or clusters whose GPU documentation is still written entirely in gres terms.

Why is my job stuck pending after I requested a GPU?

Most commonly either the requested count exceeds what any single node (or the whole partition) actually has, the GPU type string doesn’t match what the cluster defines (check with sinfo -o "%P %G"), or you combined --gres=gpu with --gpus-per-node in the same submission — Slurm rejects that combination as mutually exclusive rather than picking one.

Do I need to request CPUs and memory separately from the GPU?

Yes. A GPU request does not implicitly reserve CPU cores or system memory for feeding it — set --cpus-per-gpu and --mem-per-gpu (or the job-level --cpus-per-task/--mem equivalents) explicitly, since an under-provisioned host side is a common cause of a GPU sitting starved for data between batches.

How do I request more than one GPU type in the same job?

--gpus-per-node accepts a comma-separated list, e.g. --gpus-per-node=v100:1,a100:1, if your cluster genuinely mixes types on the same node. This is uncommon — most sites separate GPU generations into distinct partitions instead, so check partition layout with sinfo before assuming a mixed request is supported.

Does nvidia-smi running successfully mean my code is using the GPU correctly?

It confirms the GPU is visible and the driver stack loaded — not that your specific program is using it. Watch the utilization column in nvidia-smi (or run it a second time, or in a loop, mid-job) while the workload is actually executing; a GPU that stays at 0% utilization throughout a run that’s supposedly GPU-accelerated points back at the “requested but never called” mistake above, not a request-syntax problem.