Calibration Plots for Prediction Models: Reading Slope, Intercept, and the Calibration Hierarchy

A calibration plot answers a question that discrimination metrics like AUC cannot: when a prediction model says a patient, applicant, or unit has a 20% probability of an outcome, do roughly 20% of the people who actually get that prediction go on to have the outcome? Discrimination and calibration are both real, necessary properties of a prediction model, and they are not the same property — a model can rank cases well while its actual predicted probabilities are systematically too high, too low, or too extreme. That combination is common enough in published prediction models that calibration reporting is now an explicit requirement in TRIPOD+AI, not an optional extra. This guide covers what calibration measures, how to read a calibration plot, what the calibration slope and intercept mean, and why good discrimination is not evidence of good calibration.

nn

Calibration vs. discrimination: two different questions

n

Discrimination asks whether a model separates people who will have the outcome from people who will not — whether it assigns higher predicted probabilities to cases than to non-cases, on average. The area under the ROC curve (AUC, also called the c-statistic) is the standard summary of discrimination: a model with AUC 0.85 correctly ranks a randomly chosen case above a randomly chosen non-case 85% of the time. See Sensitivity vs. Specificity for the underlying 2×2 mechanics ROC curves are built from.

n

Calibration asks a different question entirely: among all the people a model assigns a given predicted probability, what fraction actually have the outcome? A model can be told “you predicted 200 people at 30% risk” and calibration checks whether close to 60 of them actually had the event. Discrimination never asks this — it only cares about relative ranking, not the actual numeric value of the prediction.

n

That difference is not academic. AUC is invariant to any monotonic transformation of the predicted probabilities: if you took every prediction from a model and squared it, halved it, or ran it through some other rank-preserving distortion, the AUC would not change at all, because the ordering of cases relative to non-cases is untouched. But the actual numbers would now be wrong — systematically too low, too high, or compressed toward the middle — and anyone using those numbers as real probabilities (to counsel a patient, size a risk-adjustment reserve, or set a screening threshold) would be acting on miscalibrated information from a model whose discrimination statistic looks fine.

nn

Why a well-discriminating model can still be poorly calibrated

n

The most common real-world source of this gap is overfitting during model development. A model fit and evaluated on the same data tends to produce predicted probabilities that are too extreme — high risks predicted too high, low risks predicted too low — because the model has partly learned noise specific to that sample, not just the true underlying relationship. The ranking of cases relative to non-cases can still be largely correct (so AUC on that same data looks good, and often looks good on a validation sample too), while the absolute predicted values are systematically off. A second common source is case-mix shift: a model developed in one population (say, a single hospital’s patient mix) applied to a different population with a different outcome prevalence will often still rank patients sensibly — sicker patients still get higher scores than healthier ones — while the absolute predicted probabilities no longer match the new population’s actual event rate. Both failure modes are invisible to AUC by construction and visible immediately on a calibration plot.

nn

Reading a calibration plot

n

A calibration plot puts predicted probability on the x-axis and observed proportion with the outcome on the y-axis, for the same set of predictions. Two common ways to build the observed values:

n

n
• Grouped (decile) calibration — sort cases into 10 (or another fixed number of) bins by predicted probability, and plot each bin’s mean predicted probability against its actual observed event rate. Simple to build and explain, but the choice of bin count and boundaries is somewhat arbitrary and can visually smooth over real miscalibration within a bin.

n

• Smoothed (flexible) calibration curve — fit a flexible nonparametric smoother (commonly a restricted cubic spline or LOESS) of observed outcome against predicted probability, avoiding the binning choice entirely. This is the form increasingly recommended in the methodological literature because it doesn’t hide local miscalibration inside arbitrary groups.

n

n

Either way, a 45° reference line from (0,0) to (1,1) represents perfect calibration — predicted probability exactly equal to observed proportion at every point on the curve. A calibration curve that sits above the diagonal at a given predicted probability means the model is under-predicting risk there (more people actually had the outcome than the model said would); a curve below the diagonal means the model is over-predicting. A curve that’s flatter than the diagonal — rising too slowly across the predicted-probability range — is the classic overfitting signature: predictions are too extreme at both ends, over-predicting the highest-risk group and under-predicting the lowest-risk group simultaneously.

nn

Calibration-in-the-large: the intercept

n

Calibration-in-the-large (also called mean calibration) is the simplest calibration summary: does the model’s average predicted probability across the whole sample match the sample’s actual observed event rate? It’s estimated by fitting a logistic regression of the observed outcome on the logit of the predicted probability, fixing the slope at exactly 1 and estimating only the intercept. An intercept of 0 means the average prediction matches the average observed rate; a positive intercept means the model systematically under-predicts risk overall, and a negative intercept means it systematically over-predicts. Calibration-in-the-large is a single number and can look fine even when a model badly over-predicts for one subgroup and under-predicts for another in a way that cancels out on average — it’s a necessary check, not a sufficient one, which is exactly why the full calibration plot and the slope below both matter alongside it.

nn

The calibration slope

n

The calibration slope comes from the same logistic regression, this time letting both the intercept and slope vary freely: outcome regressed on the logit of the predicted probability. A slope of exactly 1 means the model’s predictions are, on average, exactly as spread out as they should be. A slope below 1 — the most common finding in practice, especially on a model’s first external validation — means predictions are too extreme: the model’s high-risk predictions are higher than warranted and its low-risk predictions are lower than warranted, the signature of overfitting on the development sample. A slope above 1 is less common and indicates the opposite: predictions are too conservative, understating how much risk actually varies across the population. A slope well below 1 is one of the standard justifications for shrinkage (uniformly pulling regression coefficients toward zero before predictions are generated) or full recalibration on independent data before a model is used outside the sample it was built on.

nn

The calibration hierarchy: four levels, not one

n

Van Calster and colleagues formalized calibration into a four-level hierarchy, from weakest to strongest claim a model can support:

n

n
• Mean calibration — the weakest level: average predicted risk matches average observed risk across the whole sample (calibration-in-the-large, above).

n

• Weak calibration — the calibration intercept and slope, taken together, both meet target (intercept 0, slope 1) when predictions are summarized through a single logistic regression — a stronger claim than mean calibration alone, but still a two-number summary.

n

• Moderate calibration — the full calibration curve (grouped or smoothed) lies close to the diagonal across the whole range of predicted probabilities, not just on average — this is what a calibration plot is actually built to check, and is the level most methodologists treat as the practical minimum bar for a model intended for individual-level decisions.

n

• Strong calibration — calibration holds within every covariate subgroup and combination, not just marginally across predicted-probability values — a much stronger, largely unverifiable-in-practice standard that mainly serves as a theoretical ceiling explaining why “the model is calibrated” is always a claim about a specific population and a specific level, not an absolute property of the model itself.

n

n

Source: Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. “A calibration hierarchy for risk models was defined: from utopia to empirical data.” Journal of Clinical Epidemiology. 2016;74:167–176.

nn

A note on the Hosmer-Lemeshow test

n

Older prediction-model papers often report a single Hosmer-Lemeshow goodness-of-fit p-value as their calibration evidence. It has two well-documented problems a calibration plot doesn’t share: the result depends on an arbitrary choice of group count, and a non-significant result (conventionally read as “calibration is fine”) mainly reflects low statistical power rather than genuinely good calibration, especially in small or moderate samples. A calibration plot with the intercept and slope reported alongside it shows the same information a Hosmer-Lemeshow test is trying to summarize, plus the direction and shape of any miscalibration the test’s single p-value cannot show at all. Van Calster and colleagues, writing for the STRATOS initiative, make this same case at length.

n

Source: Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW; Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative. “Calibration: the Achilles heel of predictive analytics.” BMC Medicine. 2019;17(1):230.

nn

Reporting calibration: what TRIPOD+AI expects

n

TRIPOD+AI, the current reporting guideline for multivariable prediction-model studies, requires authors to report both discrimination and calibration for every model, describe how calibration was assessed (plot, slope, intercept, or a named test), and disclose whether calibration was checked in the same data the model was developed on or in an independent validation sample — internal calibration on development data is expected to look better than calibration on truly external data, and reporting only the former without saying so is exactly the kind of incomplete reporting the guideline exists to close off. See STARD and TRIPOD+AI: Reporting Diagnostic and Prediction Model Studies for the full checklist context.

n

Calibration failures also distort model-comparison metrics that get layered on top of a base model, not just the base model’s own reported performance: a poorly calibrated model can still show an apparently favorable Net Reclassification Improvement purely because of the miscalibration itself, independent of whether it adds any genuine predictive information. See Net Reclassification Improvement for that specific failure mode.

nn

Frequently asked questions

n

Can a model have a high AUC and still be poorly calibrated?

n

Yes, and this is common in practice, not a rare edge case. AUC only measures whether a model ranks cases above non-cases correctly; it is unaffected by any rank-preserving distortion of the actual predicted values. A model can discriminate well while its predicted probabilities are systematically too high, too low, or too extreme — that gap is exactly what a calibration plot is designed to catch and AUC cannot.

n

What’s a good calibration slope?

n

The target is exactly 1. In practice, a slope noticeably below 1 (predictions too extreme, usually from overfitting) is the far more common finding, especially the first time a model is applied outside its development sample; a slope above 1 (predictions too conservative) is less common. There’s no single numeric cutoff for “acceptable” — a slope of 0.9 in a large, well-behaved validation sample may be unremarkable, while the same slope in a small or unstable sample deserves more scrutiny.

n

What does it mean if a calibration curve is flat?

n

A calibration curve that’s flatter than the 45° diagonal — not rising as fast as observed risk actually does across the predicted-probability range — means predictions are too extreme at both ends: the model over-predicts for its highest-risk group and under-predicts for its lowest-risk group at the same time. This is the visual signature of a calibration slope below 1, usually traced back to overfitting during model development.

n

How do you fix a poorly calibrated model?

n

The standard fixes scale with how badly the model is miscalibrated. If only the intercept is off (mean calibration fails but the slope is close to 1), a simple intercept update — re-centering predictions to the new population’s event rate — is often enough. If the slope itself is off, logistic recalibration (re-estimating both intercept and slope on new data) or shrinking the original model’s coefficients toward zero before recalibrating are standard next steps; more flexible approaches like isotonic regression are also used, particularly for machine-learning models, though they require a reasonably large recalibration sample to avoid introducing new instability.

n

Is calibration or discrimination more important?

n

They answer different questions and neither substitutes for the other. Discrimination determines whether a model is capable of separating cases from non-cases at all; calibration determines whether its stated probabilities can be trusted at face value for an individual decision. A model needs both to be treated as reliable for decisions that depend on the actual predicted number, not just the ranking — risk communication to a patient, a threshold set at a specific probability, or a resource-allocation formula that multiplies by predicted risk directly.

nn

Related CASRAI guides

n

For the ranking-based metric calibration is distinct from, see Sensitivity vs. Specificity. For the regression model most often generating the predicted probabilities being calibrated, see Logistic Regression (the Logit Model). For how miscalibration distorts a common model-comparison statistic, see Net Reclassification Improvement. For the reporting checklist that requires calibration disclosure, see STARD and TRIPOD+AI. Browse the full Research Methods & Statistics hub for related quantitative-analysis guides.

n