Skip to main content
v2026.11,986 entries · CC-BY 4.0

sklearn roc_auc_score (AUROC): Syntax, Multiclass and Errors

How to compute AUROC with scikit-learn’s roc_auc_score: arguments, binary and multiclass inputs, max_fpr, and the exact errors wrong inputs raise.

Written and maintained by CASRAI Editorial Board

Published

On this page

In scikit-learn, AUROC is computed with sklearn.metrics.roc_auc_score(y_true, y_score), which returns the area under the receiver operating characteristic curve as a float. For a binary classifier you pass the true 0/1 labels and the predicted probability of the positive class, for example clf.predict_proba(X)[:, 1], not the hard class predictions. The scikit-learn documentation (version 1.9.1, read 2026-11-18) also supports multiclass and multilabel targets, with restrictions covered below.

What roc_auc_score is used for in research

The scikit-learn user guide describes roc_auc_score as the function that computes the area under the ROC curve, abbreviated ROC-AUC or AUROC. In research it is the usual summary of how well a model’s scores rank positive cases above negative ones, for example in diagnostic or prognostic prediction studies, because it does not require choosing a classification threshold for each label. That is also why it behaves differently from accuracy or F1: the input is a score, not a decision.

This page covers the function itself: its arguments, the input shapes it accepts, the errors it raises, and how to read its result. For what the curve plots and where AUC misleads, see ROC curves and AUC. For a hand, Excel and R walk-through of the same quantity, see how to calculate AUC. For the choice between AUROC and the precision-recall alternative, see AUROC vs. AUPRC.

The signature

The documented signature is:

roc_auc_score(y_true, y_score, *, average='macro', sample_weight=None,
              max_fpr=None, multi_class='raise', labels=None)
  • y_true: true labels, shape (n_samples,), or a label-indicator array of shape (n_samples, n_classes) for multilabel problems.
  • y_score: target scores. What is valid depends on the problem type (see the data table below).
  • average: one of ‘micro’, ‘macro’, ‘samples’, ‘weighted’ or None; the default is ‘macro’. Which values are allowed depends on the problem type.
  • sample_weight: optional per-sample weights.
  • max_fpr: a float greater than 0 and at most 1. When set, the function returns a standardized partial AUC over the false-positive-rate range [0, max_fpr], following McClish (1989).
  • multi_class: ‘raise’ (default), ‘ovr’ or ‘ovo’. Only used for multiclass targets.
  • labels: only for multiclass; lists the classes in the order of the columns of y_score.

How to install or access roc_auc_score

The function ships with scikit-learn. The official installation page gives pip install -U scikit-learn for an existing environment; pip’s own documentation gives the general form python -m pip install [options] followed by a requirement specifier such as the package name. Run the command inside the virtual environment you use for the project, then import the function:

from sklearn.metrics import roc_auc_score

The examples on this page were executed against scikit-learn 1.9.1. Messages and defaults can differ in other versions, so run python -m pip show scikit-learn to see which one you have before comparing an error text.

How to compute AUROC in scikit-learn step by step

Binary classification with predict_proba

  1. Fit a classifier and hold out data you did not train on.
  2. Take the score for the positive class. For a classifier whose classes_ are [0, 1], that is column 1 of predict_proba. The documentation states that the binary score is the probability of the class with the greater label, that is estimator.classes_[1].
  3. Call roc_auc_score(y_true, scores).
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0, stratify=y)
clf = LogisticRegression(solver="newton-cholesky", random_state=0).fit(X_train, y_train)

roc_auc_score(y_test, clf.predict_proba(X_test)[:, 1])   # 0.9895
roc_auc_score(y_test, clf.decision_function(X_test))     # 0.9895

Both calls return the same value, 0.9895 on this split, because AUROC depends only on the ranking of the scores. The documentation accepts either probabilities or non-thresholded decision values; decision_function output is not a probability and does not need to be.

A worked example you can check by hand

Illustrative toy data, not a real study: eight cases, four negative and four positive.

y_true  = [0, 0, 0, 0, 1, 1, 1, 1]
y_score = [0.1, 0.4, 0.35, 0.8, 0.35, 0.6, 0.7, 0.9]
roc_auc_score(y_true, y_score)   # 0.71875

The negatives score 0.1, 0.4, 0.35 and 0.8; the positives score 0.35, 0.6, 0.7 and 0.9. There are 4 × 4 = 16 positive-negative pairs. The positive case outscores the negative case in 11 pairs and ties in 1 pair (0.35 against 0.35), which counts one half. That gives (11 + 0.5) / 16 = 0.71875, matching the function. This pairwise reading is the practical meaning of AUROC: the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case, with ties counted as half.

Multiclass classification

  1. Pass the full probability matrix from predict_proba, shape (n_samples, n_classes). Each row must sum to 1.
  2. Choose multi_class explicitly. The default ‘raise’ does not compute anything for a multiclass target.
  3. Pick ‘ovr’ (each class against the rest) or ‘ovo’ (average over pairs of classes).
from sklearn.datasets import load_iris
Xi, yi = load_iris(return_X_y=True)
ci = LogisticRegression(max_iter=1000).fit(Xi, yi)
P = ci.predict_proba(Xi)

roc_auc_score(yi, P, multi_class='ovr')                  # 0.9984
roc_auc_score(yi, P, multi_class='ovo')                  # 0.9984
roc_auc_score(yi, P, multi_class='ovr', average=None)    # [1.0, 0.9976, 0.9976]

On this training-set example the two schemes happen to agree. They differ in principle: the documentation says ‘ovr’ is sensitive to class imbalance even when average='macro', while ‘ovo’ (the pairwise approach of Hand and Till, 2001) is insensitive to class imbalance when average='macro'. If your classes have very different prevalence, that distinction is a reason to report which scheme you used. Column order must match labels; if you do not pass labels, the columns are assumed to follow the numerical or lexicographical order of the labels in y_true.

Partial AUC with max_fpr

Setting max_fpr restricts the area to false-positive rates up to that value and returns the McClish-standardized partial AUC. On the toy data above, roc_auc_score(y_true, y_score, max_fpr=0.5) returns 0.6667. The documentation notes that for multiclass targets max_fpr must be None or 1.0.

Using AUROC as a cross-validation score

The model-evaluation page lists these scoring strings, all backed by roc_auc_score: ‘roc_auc’, ‘roc_auc_ovr’, ‘roc_auc_ovo’, ‘roc_auc_ovr_weighted’ and ‘roc_auc_ovo_weighted’. For binary problems use ‘roc_auc’; for multiclass use one of the explicit variants:

from sklearn.model_selection import cross_val_score
cross_val_score(LogisticRegression(max_iter=5000), X, y, cv=5, scoring='roc_auc')
# array([0.994, 0.993, 0.998, 0.979, 0.998])   (breast cancer data)
cross_val_score(LogisticRegression(max_iter=1000), Xi, yi, cv=5, scoring='roc_auc_ovr')
# array([0.993, 1.   , 0.995, 0.993, 1.   ])   (iris data)

How to read the output

The result is a number between 0 and 1. A value of 0.5 corresponds to ranking no better than chance, 1.0 to every positive case scoring above every negative case, and values below 0.5 mean the scores rank in the wrong direction. The documentation adds that the Gini coefficient is 2 × AUC − 1, which is 0 for random guessing in expectation.

Three reading points follow from running the function:

  • Hard labels give a different number. On the breast-cancer split above, passing clf.predict(X_test) instead of probabilities returned 0.9251, against 0.9895 for the probabilities. With 0/1 predictions the curve has a single operating point, so the result is not the model’s ranking quality.
  • A value near 0 usually means the wrong column. Passing predict_proba(X_test)[:, 0], the probability of class 0, returned 0.0105 on the same split, which is 1 minus the correct value. Check the column against classes_.
  • AUROC says nothing about calibration or a chosen threshold. To pick a cut-off from the curve, see Youden’s J index for ROC threshold selection; to inspect the classifier at one threshold, see sklearn confusion matrix. When positives are rare, the sklearn precision_recall_curve guide covers the companion view.

The data table below sets out what each problem type accepts, compiled from the scikit-learn documentation.

Troubleshooting roc_auc_score

These are the messages produced by scikit-learn 1.9.1 for the most common input mistakes, each reproduced by running the call.

Multiclass target with the default multi_class

Calling roc_auc_score(yi, P) on the three-class data raises ValueError: multi_class must be in ('ovo', 'ovr'). The documented default, ‘raise’, exists to force the choice. Pass multi_class='ovr' or 'ovo'.

Rows of y_score do not sum to 1

Passing decision_function output, or a probability matrix with a missing column, to a multiclass call raises ValueError: Target scores need to be probabilities for multiclass roc_auc, i.e. they should sum up to 1.0 over classes. Use predict_proba with every class column.

The full probability matrix passed for a binary problem

Passing predict_proba(X_test) with both columns for a binary target raised ValueError: y should be a 1d array, got an array of shape (171, 2) instead. Select the positive-class column with [:, 1].

Only one class in y_true

With y_true = [1, 1, 1, 1] the function returns nan and emits UndefinedMetricWarning: Only one class is present in y_true. ROC AUC score is not defined in that case. In cross-validation this shows up as a nan fold score, so check that each fold contains both classes (stratified splitting helps).

Scoring string on a multiclass problem

Using scoring='roc_auc' with cross_val_score on the three-class iris data returned an array of nan with a “Scoring failed” warning, because the plain string leaves multi_class at ‘raise’. Use ‘roc_auc_ovr’ or ‘roc_auc_ovo’ instead.

Unsupported argument combinations

  • multi_class='ovo' with average=None raises NotImplementedError: average=None is not implemented for multi_class='ovo'.
  • max_fpr=0.5 on a multiclass target raises ValueError: Partial AUC computation not available in multiclass setting, 'max_fpr' must be set to `None`, received `max_fpr=0.5` instead.
  • Scores containing NaN raise ValueError: Input contains NaN., and mismatched lengths raise ValueError: Found input variables with inconsistent numbers of samples.

Free alternatives to roc_auc_score

If you need the curve itself or a different route to the same area, these are the scikit-learn tools the documentation lists alongside the function. All are part of scikit-learn, which is open source.

Tool What it does Source
sklearn.metrics.roc_curve Returns false positive rates, true positive rates and thresholds for a binary task; the area can then be computed from the curve points. roc_auc_score “See also”
RocCurveDisplay.from_estimator / from_predictions Plots the ROC curve from a fitted estimator or from scores. roc_auc_score “See also”
sklearn.metrics.average_precision_score Summarises the precision-recall curve, the usual companion when classes are imbalanced. roc_auc_score “See also”

For the roc_curve function, the scikit-learn documentation example uses y = [1, 1, 2, 2] with scores [0.1, 0.4, 0.35, 0.8] and pos_label=2, which returns thresholds [inf, 0.8, 0.4, 0.35, 0.1].

Sources

Data: scikit-learn roc_auc_score: accepted inputs and options by problem type

Compiled from the scikit-learn 1.9.1 roc_auc_score reference page and the model-evaluation user guide (ROC section), read 2026-11-18. Each row states, for one problem type, what y_true and y_score must contain and which average and max_fpr values the documentation allows.

scikit-learn roc_auc_score: accepted inputs and options by problem type
Problem typey_truey_scoreaverage values documentedmax_fpr
BinaryShape (n_samples,)Probability of the class with the greater label (classes_[1]), or non-thresholded decision values; shape (n_samples,)'samples' is ignored when y_true is binaryFloat above 0 and up to 1 gives standardized partial AUC
Multiclass, multi_class='ovr'Shape (n_samples,)predict_proba matrix, shape (n_samples, n_classes), rows sum to 1'macro' and 'weighted'; None and 'micro' only with 'ovr'None or 1.0 only
Multiclass, multi_class='ovo'Shape (n_samples,)predict_proba matrix, shape (n_samples, n_classes), rows sum to 1'macro' and 'weighted' onlyNone or 1.0 only
MultilabelLabel-indicator array, shape (n_samples, n_classes)Shape (n_samples, n_classes)'micro', 'macro', 'samples', 'weighted' or NoneRestriction (None or 1.0) is documented for multiclass only

Sources

Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about sklearn roc_auc_score (AUROC): Syntax, Multiclass and Errors

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →