sacct and seff: Checking Slurm Job Resource Usage

A Slurm batch job finishes and tells you almost nothing about how it actually ran. Did it use the memory you requested, or a fraction of it? Did every core stay busy, or did most of the allocation sit idle? Two commands answer this directly: sacct, which queries Slurm’s accounting database for the full history of a job — state, elapsed time, exit code, memory, CPU time — and seff, a companion tool that turns that same accounting data into a one-screen efficiency summary. Together they’re how you find out whether a job’s resource request matched its resource use, which matters for more than curiosity: on a shared cluster, requests that consistently overshoot actual use waste allocation that fairshare scheduling accounts for, and can quietly push your future jobs down the priority queue.

This guide covers the sacct syntax that actually surfaces memory and CPU data (it’s easy to run a query that comes back blank), how to read seff‘s efficiency percentages, and how to use both to right-size the #SBATCH requests in your job script going forward.

Checking job history with sacct

sacct queries the Slurm accounting database (slurmdbd) rather than the live scheduler, so it works for jobs that finished hours, days, or months ago — not just what’s currently running. The basic form:

sacct -j 4821553

By default this returns a compact table — JobID, JobName, Partition, Account, AllocCPUS, State, ExitCode — but notably not memory usage, even though you might expect it. That’s the single most common point of confusion with sacct: Slurm records a job as two or more accounting entries — a parent allocation record plus one entry per job step (typically a .batch step for the script itself, and an .extern step for anything running outside a step). Resource-usage fields like MaxRSS and MaxVMSize are populated on the step records, captured via the wait3()/getrusage() system calls when a step exits — the parent allocation line reports them as zero because the allocation itself doesn’t consume memory, only the steps running inside it do.

To actually see memory usage, query the batch step directly, and choose your own fields with --format:

sacct -j 4821553.batch --format=JobID,MaxRSS,MaxVMSize,Elapsed,TotalCPU,State

Or pull everything — parent and steps — in one query and scan for the step that has the numbers you want:

sacct -j 4821553 --format=JobID,JobName,Elapsed,AllocCPUS,TotalCPU,MaxRSS,ReqMem,State,ExitCode

Fields worth knowing

Field	What it reports
JobID	Job identifier; a step is shown as JobID.stepname (e.g. 4821553.batch)
Elapsed	Wall-clock run time, formatted [DD-[HH:]]MM:SS
TotalCPU	Sum of user + system CPU time across all tasks — for a job using multiple cores, this routinely exceeds Elapsed, because CPU-seconds accumulate independently on every core
AllocCPUS	CPU cores actually allocated to the job
MaxRSS	Peak resident-set (physical) memory used by any single task in the step
ReqMem	Memory requested (a trailing c means per-core, n means per-node)
State	COMPLETED, FAILED, TIMEOUT, OUT_OF_MEMORY, CANCELLED, etc.
ExitCode	The program’s exit status, as code:signal

A few options that make repeated use practical:

• -u USERNAME — every job for a given user, instead of one job ID
• -S 2026-08-01 -E 2026-08-31 (--starttime/--endtime) — restrict to a date range, essential once you’re querying your own job history rather than one job you already know the ID of
• -X, --allocations — collapse output to one line per job allocation instead of one line per step (trades away the per-step memory detail above for a quick overview across many jobs)
• -p/-P, --parsable/--parsable2 — pipe-delimited output, meant for scripting rather than reading at a terminal
• -l, --long — a wide default field set, useful for a first look at what’s available before narrowing with --format

Run sacct --helpformat on your own cluster to see the full field list; Slurm accounting can carry site-specific TRES (trackable resource) fields beyond the core set above, particularly on clusters that track GPUs or licenses as accounted resources.

Getting a quick read with seff

Once you know a job’s ID, seff converts the same accounting data sacct exposes into a single efficiency summary, without you having to assemble a --format string or do the arithmetic yourself:

seff 4821553

Typical output looks like this (an illustrative example — the real numbers come from your job):

Job ID: 4821553
Cluster: cluster
User/Group: researcher/researcher
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 16
CPU Utilized: 01:12:40
CPU Efficiency: 27.29% of 04:26:40 core-walltime
Job Wall-clock time: 00:16:40
Memory Utilized: 1.85 GB
Memory Efficiency: 7.71% of 24.00 GB

CPU Efficiency is TotalCPU divided by the core-time you were allocated (Elapsed × AllocCPUS) — in the example above, the job used about 27% of the CPU-seconds it was given across its 16 requested cores, meaning most of those cores sat mostly idle for the job’s 16-minute runtime. Memory Efficiency is peak MaxRSS divided by ReqMem — here, under 8% of the requested 24 GB.

A handful of caveats matter for reading seff correctly rather than at face value:

• Running jobs report incomplete numbers. seff can be run against a still-executing job, but it’s reading whatever accounting data exists so far (via sstat), not the job’s final usage — re-run it after the job finishes for the real figures.
• Multi-node memory efficiency can mislead. MaxRSS reflects the peak on a single task/node, not summed across the whole allocation — a multi-node job that’s memory-efficient on the node seff happened to sample can still be starving a different node in the same allocation. Cross-check with per-task sacct output for jobs spread across many tasks or array indices, where a single aggregate figure hides real variance between tasks.
• seff doesn’t cover GPUs. Its efficiency figures are CPU and memory only; Slurm’s default TRES accounting doesn’t give seff GPU utilization data, so a job that requested a GPU and never used it will show as CPU/memory-efficient regardless. GPU utilization needs a separate tool (nvidia-smi, DCGM, or whatever your center’s GPU accounting layer provides).
• seff is a companion script, not a core Slurm binary. It’s bundled and widely deployed, but it isn’t guaranteed present on every cluster the way sacct is — if it’s missing, the same numbers are still derivable from sacct directly using the field list above.

Right-sizing future resource requests

The reason to look at any of this isn’t retrospective curiosity — it’s that Slurm’s fairshare algorithm factors in resource consumption when it computes scheduling priority for your account, and a habit of requesting far more than you use has a real, cumulative cost. A job that ties up 24 GB and uses 1.85 GB isn’t just wasteful in an abstract sense: on a shared cluster, that memory is unavailable to every other queued job for the job’s entire runtime, whether or not it’s actually being touched. Centers that track fairshare usage by requested (not consumed) allocation will reflect that overshoot in your account’s future priority, and even centers that bill or weight by actual use still lose the scheduling flexibility that memory would have bought elsewhere. Treat a job that used 8% of requested memory the same way you’d treat over-provisioned cloud infrastructure: a real, recurring cost, not a rounding error.

A practical workflow:

1. Request conservatively generous on the first run of a new job type — you don’t have data yet, so erring high avoids an OOM kill or a TIMEOUT mid-run, both of which cost more wall-clock time than an oversized request does.
2. Run seff (or the equivalent sacct query) as soon as the job completes, before you submit the next batch of similar jobs. Some sites automatically email seff output on job completion via #SBATCH --mail-type=END; check local documentation for whether yours does.
3. Narrow the request toward the observed peak, with headroom — not down to the exact MaxRSS figure, since memory use can vary run-to-run with input size, and a request that’s too tight risks an out-of-memory kill on a slightly larger input. A common rule of thumb is padding 20–30% above the highest observed peak across a few representative runs, not just one.
4. Re-check periodically, not just once. Input data, library versions, and compiler flags all change memory and CPU behavior over the life of a project; a request that was well-tuned six months ago may no longer match what the job actually does today.
5. For array and highly parallel jobs, right-size the whole family, not one instance. Look at the spread across array task indices or parallel job instances rather than tuning to a single representative task — if usage varies a lot between tasks, size to the tail, not the median, or you’ll trade wasted allocation for OOM kills on the heavier tasks.

This same logic extends past memory and CPU cores: the sustainable-HPC case for right-sizing is the same waste, viewed as energy and carbon rather than allocation-share — an idle, over-provisioned job burns power for capacity nothing is using, independent of who’s paying the scheduling-priority cost.

Common pitfalls

• Querying the parent job ID and concluding memory usage is zero. As covered above, this is almost always a step-selection problem, not a job that genuinely used no memory — add .batch or query without -X to see step-level figures.
• Comparing TotalCPU to Elapsed and concluding a job is CPU-inefficient. For any job using more than one core, TotalCPU exceeding Elapsed is expected and correct — it’s the sum across all cores, not a single core’s time. Compare against Elapsed × AllocCPUS (which is exactly what seff‘s CPU Efficiency does), not Elapsed alone.
• Requesting resources per the job’s submission command rather than the script’s actual directives. If you’re mixing srun, sbatch, and salloc for different stages of a workflow, the resource request that sacct/seff measure against is whichever one actually allocated the job — double check you’re reading the accounting for the right submission if a workflow chains several.
• Treating one run’s efficiency numbers as permanent. A job that used 90% of requested memory on a small test dataset can use a fraction of that, or far more, once pointed at production-scale input — re-verify after any meaningful change in input size or job parameters.

FAQ

Why does sacct show blank memory fields for my job?

You’re almost certainly querying the parent allocation record rather than a step. Add .batch to the job ID (e.g. sacct -j 4821553.batch) or drop the -X/--allocations flag to see per-step usage, which is where MaxRSS and related fields are actually recorded.

What’s a “good” CPU or memory efficiency percentage?

There’s no universal number, and Slurm itself doesn’t define one — it depends on the workload. A tightly-tuned, compute-bound job might reasonably run in the 80–95% CPU-efficiency range; an I/O-bound job that spends real time waiting on disk or network will legitimately run lower without anything being wrong. The useful comparison is your own job’s history over time and against similar jobs, not an absolute threshold.

Can I check sacct/seff data for a job that finished weeks ago?

sacct queries the accounting database, not the live queue, so historical jobs are available as long as your cluster’s retention policy hasn’t purged them — ask your center’s documentation or administrators for the retention window, since it’s site-configured and varies widely.

Does using more memory or CPU than requested cause a problem?

CPU usage above the requested core count generally isn’t possible — Slurm enforces the allocation via cgroups on most clusters, so the job is confined to the cores it was given. Memory is the more common failure mode: exceeding ReqMem typically ends the job with an OUT_OF_MEMORY state and a non-zero exit code, which is exactly why the conservative-first, narrow-later workflow above avoids over-correcting too aggressively on the down side.

Does seff account for GPU utilization?

No — seff‘s efficiency figures cover CPU and memory only. A job can show high CPU/memory efficiency while its requested GPU sits completely idle; GPU utilization needs a separate check, typically nvidia-smi during the run or a center-provided GPU accounting tool.