Written and maintained by CASRAI Editorial Board
Published
On this page
In scikit-learn, AUROC is computed with sklearn.metrics.roc_auc_score(y_true, y_score), which returns the area under the receiver operating characteristic curve as a float. For a binary classifier you pass the true 0/1 labels and the predicted probability of the positive class, for example clf.predict_proba(X)[:, 1], not the hard class predictions. The scikit-learn documentation (version 1.9.1, read 2026-11-18) also supports multiclass and multilabel targets, with restrictions covered below.
What roc_auc_score is used for in research
The scikit-learn user guide describes roc_auc_score as the function that computes the area under the ROC curve, abbreviated ROC-AUC or AUROC. In research it is the usual summary of how well a model’s scores rank positive cases above negative ones, for example in diagnostic or prognostic prediction studies, because it does not require choosing a classification threshold for each label. That is also why it behaves differently from accuracy or F1: the input is a score, not a decision.
This page covers the function itself: its arguments, the input shapes it accepts, the errors it raises, and how to read its result. For what the curve plots and where AUC misleads, see ROC curves and AUC. For a hand, Excel and R walk-through of the same quantity, see how to calculate AUC. For the choice between AUROC and the precision-recall alternative, see AUROC vs. AUPRC.
The signature
The documented signature is:
roc_auc_score(y_true, y_score, *, average='macro', sample_weight=None,
max_fpr=None, multi_class='raise', labels=None)
- y_true: true labels, shape (n_samples,), or a label-indicator array of shape (n_samples, n_classes) for multilabel problems.
- y_score: target scores. What is valid depends on the problem type (see the data table below).
- average: one of ‘micro’, ‘macro’, ‘samples’, ‘weighted’ or None; the default is ‘macro’. Which values are allowed depends on the problem type.
- sample_weight: optional per-sample weights.
- max_fpr: a float greater than 0 and at most 1. When set, the function returns a standardized partial AUC over the false-positive-rate range [0, max_fpr], following McClish (1989).
- multi_class: ‘raise’ (default), ‘ovr’ or ‘ovo’. Only used for multiclass targets.
- labels: only for multiclass; lists the classes in the order of the columns of y_score.
How to install or access roc_auc_score
The function ships with scikit-learn. The official installation page gives pip install -U scikit-learn for an existing environment; pip’s own documentation gives the general form python -m pip install [options] followed by a requirement specifier such as the package name. Run the command inside the virtual environment you use for the project, then import the function:
from sklearn.metrics import roc_auc_score
The examples on this page were executed against scikit-learn 1.9.1. Messages and defaults can differ in other versions, so run python -m pip show scikit-learn to see which one you have before comparing an error text.
How to compute AUROC in scikit-learn step by step
Binary classification with predict_proba
- Fit a classifier and hold out data you did not train on.
- Take the score for the positive class. For a classifier whose
classes_are [0, 1], that is column 1 ofpredict_proba. The documentation states that the binary score is the probability of the class with the greater label, that isestimator.classes_[1]. - Call
roc_auc_score(y_true, scores).
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=0, stratify=y)
clf = LogisticRegression(solver="newton-cholesky", random_state=0).fit(X_train, y_train)
roc_auc_score(y_test, clf.predict_proba(X_test)[:, 1]) # 0.9895
roc_auc_score(y_test, clf.decision_function(X_test)) # 0.9895
Both calls return the same value, 0.9895 on this split, because AUROC depends only on the ranking of the scores. The documentation accepts either probabilities or non-thresholded decision values; decision_function output is not a probability and does not need to be.
A worked example you can check by hand
Illustrative toy data, not a real study: eight cases, four negative and four positive.
y_true = [0, 0, 0, 0, 1, 1, 1, 1]
y_score = [0.1, 0.4, 0.35, 0.8, 0.35, 0.6, 0.7, 0.9]
roc_auc_score(y_true, y_score) # 0.71875
The negatives score 0.1, 0.4, 0.35 and 0.8; the positives score 0.35, 0.6, 0.7 and 0.9. There are 4 × 4 = 16 positive-negative pairs. The positive case outscores the negative case in 11 pairs and ties in 1 pair (0.35 against 0.35), which counts one half. That gives (11 + 0.5) / 16 = 0.71875, matching the function. This pairwise reading is the practical meaning of AUROC: the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case, with ties counted as half.
Multiclass classification
- Pass the full probability matrix from
predict_proba, shape (n_samples, n_classes). Each row must sum to 1. - Choose
multi_classexplicitly. The default ‘raise’ does not compute anything for a multiclass target. - Pick ‘ovr’ (each class against the rest) or ‘ovo’ (average over pairs of classes).
from sklearn.datasets import load_iris
Xi, yi = load_iris(return_X_y=True)
ci = LogisticRegression(max_iter=1000).fit(Xi, yi)
P = ci.predict_proba(Xi)
roc_auc_score(yi, P, multi_class='ovr') # 0.9984
roc_auc_score(yi, P, multi_class='ovo') # 0.9984
roc_auc_score(yi, P, multi_class='ovr', average=None) # [1.0, 0.9976, 0.9976]
On this training-set example the two schemes happen to agree. They differ in principle: the documentation says ‘ovr’ is sensitive to class imbalance even when average='macro', while ‘ovo’ (the pairwise approach of Hand and Till, 2001) is insensitive to class imbalance when average='macro'. If your classes have very different prevalence, that distinction is a reason to report which scheme you used. Column order must match labels; if you do not pass labels, the columns are assumed to follow the numerical or lexicographical order of the labels in y_true.
Partial AUC with max_fpr
Setting max_fpr restricts the area to false-positive rates up to that value and returns the McClish-standardized partial AUC. On the toy data above, roc_auc_score(y_true, y_score, max_fpr=0.5) returns 0.6667. The documentation notes that for multiclass targets max_fpr must be None or 1.0.
Using AUROC as a cross-validation score
The model-evaluation page lists these scoring strings, all backed by roc_auc_score: ‘roc_auc’, ‘roc_auc_ovr’, ‘roc_auc_ovo’, ‘roc_auc_ovr_weighted’ and ‘roc_auc_ovo_weighted’. For binary problems use ‘roc_auc’; for multiclass use one of the explicit variants:
from sklearn.model_selection import cross_val_score
cross_val_score(LogisticRegression(max_iter=5000), X, y, cv=5, scoring='roc_auc')
# array([0.994, 0.993, 0.998, 0.979, 0.998]) (breast cancer data)
cross_val_score(LogisticRegression(max_iter=1000), Xi, yi, cv=5, scoring='roc_auc_ovr')
# array([0.993, 1. , 0.995, 0.993, 1. ]) (iris data)
How to read the output
The result is a number between 0 and 1. A value of 0.5 corresponds to ranking no better than chance, 1.0 to every positive case scoring above every negative case, and values below 0.5 mean the scores rank in the wrong direction. The documentation adds that the Gini coefficient is 2 × AUC − 1, which is 0 for random guessing in expectation.
Three reading points follow from running the function:
- Hard labels give a different number. On the breast-cancer split above, passing
clf.predict(X_test)instead of probabilities returned 0.9251, against 0.9895 for the probabilities. With 0/1 predictions the curve has a single operating point, so the result is not the model’s ranking quality. - A value near 0 usually means the wrong column. Passing
predict_proba(X_test)[:, 0], the probability of class 0, returned 0.0105 on the same split, which is 1 minus the correct value. Check the column againstclasses_. - AUROC says nothing about calibration or a chosen threshold. To pick a cut-off from the curve, see Youden’s J index for ROC threshold selection; to inspect the classifier at one threshold, see sklearn confusion matrix. When positives are rare, the sklearn precision_recall_curve guide covers the companion view.
The data table below sets out what each problem type accepts, compiled from the scikit-learn documentation.
Troubleshooting roc_auc_score
These are the messages produced by scikit-learn 1.9.1 for the most common input mistakes, each reproduced by running the call.
Multiclass target with the default multi_class
Calling roc_auc_score(yi, P) on the three-class data raises ValueError: multi_class must be in ('ovo', 'ovr'). The documented default, ‘raise’, exists to force the choice. Pass multi_class='ovr' or 'ovo'.
Rows of y_score do not sum to 1
Passing decision_function output, or a probability matrix with a missing column, to a multiclass call raises ValueError: Target scores need to be probabilities for multiclass roc_auc, i.e. they should sum up to 1.0 over classes. Use predict_proba with every class column.
The full probability matrix passed for a binary problem
Passing predict_proba(X_test) with both columns for a binary target raised ValueError: y should be a 1d array, got an array of shape (171, 2) instead. Select the positive-class column with [:, 1].
Only one class in y_true
With y_true = [1, 1, 1, 1] the function returns nan and emits UndefinedMetricWarning: Only one class is present in y_true. ROC AUC score is not defined in that case. In cross-validation this shows up as a nan fold score, so check that each fold contains both classes (stratified splitting helps).
Scoring string on a multiclass problem
Using scoring='roc_auc' with cross_val_score on the three-class iris data returned an array of nan with a “Scoring failed” warning, because the plain string leaves multi_class at ‘raise’. Use ‘roc_auc_ovr’ or ‘roc_auc_ovo’ instead.
Unsupported argument combinations
multi_class='ovo'withaverage=NoneraisesNotImplementedError: average=None is not implemented for multi_class='ovo'.max_fpr=0.5on a multiclass target raisesValueError: Partial AUC computation not available in multiclass setting, 'max_fpr' must be set to `None`, received `max_fpr=0.5` instead.- Scores containing NaN raise
ValueError: Input contains NaN., and mismatched lengths raiseValueError: Found input variables with inconsistent numbers of samples.
Free alternatives to roc_auc_score
If you need the curve itself or a different route to the same area, these are the scikit-learn tools the documentation lists alongside the function. All are part of scikit-learn, which is open source.
| Tool | What it does | Source |
|---|---|---|
sklearn.metrics.roc_curve |
Returns false positive rates, true positive rates and thresholds for a binary task; the area can then be computed from the curve points. | roc_auc_score “See also” |
RocCurveDisplay.from_estimator / from_predictions |
Plots the ROC curve from a fitted estimator or from scores. | roc_auc_score “See also” |
sklearn.metrics.average_precision_score |
Summarises the precision-recall curve, the usual companion when classes are imbalanced. | roc_auc_score “See also” |
For the roc_curve function, the scikit-learn documentation example uses y = [1, 1, 2, 2] with scores [0.1, 0.4, 0.35, 0.8] and pos_label=2, which returns thresholds [inf, 0.8, 0.4, 0.35, 0.1].
Sources
- scikit-learn: roc_auc_score, checked 2026-11-18
- scikit-learn: Metrics and scoring: quantifying the quality of predictions, checked 2026-11-18
- scikit-learn: roc_curve, checked 2026-11-18
- scikit-learn: Installing scikit-learn, checked 2026-11-18
- pip documentation: pip install, checked 2026-11-18
Data: scikit-learn roc_auc_score: accepted inputs and options by problem type
Compiled from the scikit-learn 1.9.1 roc_auc_score reference page and the model-evaluation user guide (ROC section), read 2026-11-18. Each row states, for one problem type, what y_true and y_score must contain and which average and max_fpr values the documentation allows.
| Problem type | y_true | y_score | average values documented | max_fpr |
|---|---|---|---|---|
| Binary | Shape (n_samples,) | Probability of the class with the greater label (classes_[1]), or non-thresholded decision values; shape (n_samples,) | 'samples' is ignored when y_true is binary | Float above 0 and up to 1 gives standardized partial AUC |
| Multiclass, multi_class='ovr' | Shape (n_samples,) | predict_proba matrix, shape (n_samples, n_classes), rows sum to 1 | 'macro' and 'weighted'; None and 'micro' only with 'ovr' | None or 1.0 only |
| Multiclass, multi_class='ovo' | Shape (n_samples,) | predict_proba matrix, shape (n_samples, n_classes), rows sum to 1 | 'macro' and 'weighted' only | None or 1.0 only |
| Multilabel | Label-indicator array, shape (n_samples, n_classes) | Shape (n_samples, n_classes) | 'micro', 'macro', 'samples', 'weighted' or None | Restriction (None or 1.0) is documented for multiclass only |
Sources
- scikit-learn: roc_auc_score (accessed 2026-11-18)
- scikit-learn: Metrics and scoring (accessed 2026-11-18)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







