How to Calculate AUC: Worked Example and Code

Q: What is the difference between auc and roc_auc_score in scikit-learn?

roc_auc_score computes ROC AUC from true labels and prediction scores. auc computes the area under any curve you give it with the trapezoidal rule, so it needs x values that are monotonic, such as the FPR from roc_curve .

To calculate AUC, rank every case by its model score and count how often a randomly chosen positive case scores higher than a randomly chosen negative case (a tied pair counts one half), or equivalently, plot the ROC curve and add up the trapezoids under it. Both routes give the same number. This page works one four-case example both ways, then shows the same calculation in Excel, R and Python and how to report it. It covers the ROC AUC used to judge classifiers and diagnostic tests; for what the number means and where it misleads, see ROC curves and AUC. The area under a concentration-time curve in pharmacokinetics is a different calculation, covered in trapezoidal AUC: linear vs log-linear.

What AUC is, in one paragraph

AUC is the area under the receiver operating characteristic (ROC) curve, which plots the true positive rate against the false positive rate as the decision threshold moves. Hanley and McNeil (1982) showed that this area is the probability that a randomly chosen diseased subject is ranked as more suspicious than a randomly chosen non-diseased one, and that it is the same quantity estimated by the nonparametric Wilcoxon statistic. Google’s Machine Learning Crash Course gives the same reading for classifiers: the probability that the model ranks a randomly chosen positive example above a randomly chosen negative one. That is why AUC does not depend on any single threshold.

The AUC formula, with every symbol explained

Route 1: the pairwise ranking count (rank-sum form)

AUC = (1 / (n₁ × n₀)) × Σ over all positive-negative pairs of [ 1 if s₊ > s₋; 0.5 if s₊ = s₋; 0 if s₊ < s₋ ]

n₁: the number of positive cases (the event you are trying to detect).
n₀: the number of negative cases.
s₊, s₋: the model score of one positive case and one negative case. Higher must mean “more likely positive”.
n₁ × n₀: the number of positive-negative pairs, so the result is a proportion between 0 and 1.

The same number comes from ranks. Rank all n₁ + n₀ scores from lowest (1) to highest, giving tied scores the average of their ranks. If R₁ is the sum of the ranks of the positive cases, then AUC = (R₁ − n₁(n₁ + 1)/2) / (n₁ × n₀). This is the Wilcoxon (Mann-Whitney) form that Hanley and McNeil connect to the area.

Route 2: the trapezoidal rule on the ROC points

AUC = Σ over k of (FPRₖ − FPRₖ₋₁) × (TPRₖ + TPRₖ₋₁) / 2

FPRₖ, TPRₖ: the false positive rate and true positive rate at the k-th threshold, with the points ordered along the horizontal axis starting at (0, 0) and ending at (1, 1).
(FPRₖ − FPRₖ₋₁): the width of one slice.
(TPRₖ + TPRₖ₋₁) / 2: the average height of that slice, which is what makes it a trapezoid rather than a rectangle.

scikit-learn’s sklearn.metrics.auc and the R package pROC’s auc() both document this trapezoidal rule.

Worked example: four cases, calculated two ways

Four patients are scored by a model. Two truly have the condition (positive) and two do not (negative). The scores are the same ones used in the scikit-learn documentation example for sklearn.metrics.auc.

Case	True class	Model score
A	Negative	0.10
B	Negative	0.40
C	Positive	0.35
D	Positive	0.80

By counting pairs

There are n₁ × n₀ = 2 × 2 = 4 positive-negative pairs.

C (0.35) vs A (0.10): positive scores higher, count 1.
C (0.35) vs B (0.40): positive scores lower, count 0.
D (0.80) vs A (0.10): positive scores higher, count 1.
D (0.80) vs B (0.40): positive scores higher, count 1.

AUC = (1 + 0 + 1 + 1) / 4 = 3/4 = 0.75.

By the trapezoidal rule

Use each score as a threshold, from highest to lowest, calling a case positive when its score is at or above the threshold. Each threshold gives one (FPR, TPR) point, where TPR = positives caught / 2 and FPR = negatives wrongly flagged / 2.

Threshold	Cases flagged	TPR	FPR
above 0.80	none	0	0
0.80	D	1/2 = 0.5	0
0.40	D, B	0.5	1/2 = 0.5
0.35	D, B, C	1	0.5
0.10	D, B, C, A	1	1

The ROC points are (0, 0), (0, 0.5), (0.5, 0.5), (0.5, 1) and (1, 1). Slice by slice:

(0, 0) to (0, 0.5): width 0, area 0.
(0, 0.5) to (0.5, 0.5): width 0.5, average height 0.5, area 0.25.
(0.5, 0.5) to (0.5, 1): width 0, area 0.
(0.5, 1) to (1, 1): width 0.5, average height 1, area 0.5.

AUC = 0 + 0.25 + 0 + 0.5 = 0.75, the same answer as the pair count, and the value scikit-learn’s documentation shows for this data.

By ranks

The ranks of the scores 0.10, 0.35, 0.40, 0.80 are 1, 2, 3, 4. The positives are C (rank 2) and D (rank 4), so R₁ = 6. AUC = (6 − 2 × 3 / 2) / (2 × 2) = (6 − 3) / 4 = 0.75.

How to interpret the result

AUC	Reading
1.0	Perfect ranking: every positive scores above every negative.
0.5	No better than random: a randomly chosen positive outranks a randomly chosen negative half the time.
below 0.5	Worse than chance, which usually means the scores or labels are pointing the wrong way (see mistakes below).
0.75 (the example)	In 3 of the 4 positive-negative pairs the positive case was ranked higher.

AUC summarises ranking quality across all thresholds, so it says nothing about which threshold to use. For the trade-off at a chosen cut-off, see sensitivity vs. specificity.

Assumptions and common mistakes

Reversed direction. The score must rise with the likelihood of the positive class. If a lower score means disease, flip the sign or set the direction. In pROC this is the direction argument (“<” or “>”), and in scikit-learn it is the pos_label of roc_curve.
Feeding hard 0/1 predictions. roc_auc_score expects probability estimates or non-thresholded decision values. Passing already-thresholded class labels collapses the ROC curve to a single point.
Wrong probability column. For a binary problem scikit-learn takes the scores of the class with the greater label, which for a fitted classifier is predict_proba(X)[:, 1].
Non-monotonic x in the trapezoid function. sklearn.metrics.auc(x, y) is a general trapezoidal-rule function and requires x to be monotonic increasing or decreasing. Sort the ROC points first if you build them yourself.
Strong class imbalance. Google’s Machine Learning Crash Course says ROC and AUC suit roughly balanced data and suggests precision-recall curves when positives are rare. See precision and recall.
Treating AUC as calibration. A high AUC does not mean the predicted probabilities are accurate. The Brier score measures that separately.

How to report AUC (APA example)

State the scored outcome, the number of positive and negative cases, how AUC was computed (rank-sum or trapezoidal) and the software, and cite the method. A sentence template:

The model’s discrimination was summarised with the area under the ROC curve, calculated as the area under the empirical ROC curve, following Hanley and McNeil (1982), from [n₁] positive and [n₀] negative cases; AUC = [value].

In APA 7th edition a journal article reference follows the pattern Author, A. A., & Author, B. B. (Year). Title of the article in sentence case. Journal Name, volume(issue), pages. https://doi.org/… The reference for this method:

Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29-36. https://doi.org/10.1148/radiology.143.1.7063747

The journal-article pattern above follows the APA Style 7th edition reference examples listed under Sources. For diagnostic-accuracy and prediction-model studies, also see the reporting checklists in STARD and TRIPOD+AI.

How to calculate AUC in Excel, R or Python

All three snippets use the four-case example above and should return 0.75.

Excel (rank-sum method)

Put the scores in A2:A5 and the labels (1 = positive, 0 = negative) in B2:B5. Excel has no built-in AUC function, so this uses the rank formula from the formula section.

C2 (fill down to C5):  =RANK.AVG(A2, $A$2:$A$5, 1)
E1 (positives, n1):    =COUNTIF(B2:B5, 1)
E2 (negatives, n0):    =COUNTIF(B2:B5, 0)
E3 (AUC):              =(SUMIF(B2:B5, 1, C2:C5) - E1*(E1+1)/2) / (E1*E2)

Ascending ranks (the final argument 1) give the lowest score rank 1, and RANK.AVG gives tied scores the average rank, so ties are handled as half-credit pairs.

R (pROC)

library(pROC)
y <- c(0, 0, 1, 1)
s <- c(0.10, 0.40, 0.35, 0.80)
r <- roc(y, s, levels = c(0, 1), direction = "<")
auc(r)   # 0.75, trapezoidal rule

# Base R, no package: trapezoids on ordered ROC points
fpr <- c(0, 0, 0.5, 0.5, 1)
tpr <- c(0, 0.5, 0.5, 1, 1)
sum(diff(fpr) * (head(tpr, -1) + tail(tpr, -1)) / 2)   # 0.75

pROC’s documentation describes auc() as computing the area under the ROC curve with the trapezoidal rule. Its partial.auc option restricts the calculation to part of the curve, and with partial.auc.correct = TRUE applies the McClish correction so that 0.5 is non-discriminant and 1.0 is maximal.

Python (scikit-learn)

from sklearn.metrics import roc_auc_score, roc_curve, auc

y_true  = [0, 0, 1, 1]
y_score = [0.10, 0.40, 0.35, 0.80]

print(roc_auc_score(y_true, y_score))   # 0.75, straight from scores

fpr, tpr, thresholds = roc_curve(y_true, y_score)
print(auc(fpr, tpr))                    # 0.75, trapezoidal rule on the ROC points

roc_auc_score takes the scores directly; auc is the generic trapezoid function that you feed with the ROC points yourself. For a fitted classifier, pass model.predict_proba(X)[:, 1] as the scores.

Frequently asked questions

How do you calculate AUC?

Score every case, then either count the share of positive-negative pairs in which the positive scores higher (ties count one half), or build the ROC points and sum the trapezoids under them. For the four-case example both give 0.75.

Is the trapezoidal AUC the same as the Wilcoxon (rank-sum) AUC?

For an empirical ROC curve built from every score, yes. Hanley and McNeil describe the area as the same quantity the Wilcoxon statistic estimates, and the worked example above returns 0.75 by pairs, by ranks and by trapezoids.

What is the difference between `auc` and `roc_auc_score` in scikit-learn?

roc_auc_score computes ROC AUC from true labels and prediction scores. auc computes the area under any curve you give it with the trapezoidal rule, so it needs x values that are monotonic, such as the FPR from roc_curve.