Calibration Slope, Calibration-in-the-Large (CITL), and ICI in Clinical Prediction Models

What Is the Calibration Slope? Definition and Role vs. Discrimination

The calibration slope (denoted β) is a parametric summary statistic in logistic recalibration that measures whether the spread (extremeness) of a prediction model’s estimated probabilities matches the observed outcome frequencies on the log-odds scale, with an ideal target value of 1.0. Whereas discrimination (quantified by the concordance statistic or area under the ROC curve [AUC]) evaluates whether a model assigns higher scores to individuals who experience the event than to those who do not, calibration evaluates whether the absolute predicted probabilities can be trusted for clinical decision-making—such as whether a predicted 20% surgical mortality risk genuinely corresponds to 20 events per 100 patients with that prediction.

In clinical prediction model validation and regulatory evaluations of machine learning / software as a medical device (SaMD), reporting discrimination without calibration is insufficient: a model whose probabilities are uniformly doubled or compressed can retain an identical rank-order AUC while systematically misleading clinical threshold decisions in decision curve analysis.

Formulas: Linear Predictor, Calibration-in-the-Large (CITL), Calibration Slope, and ICI

Following Cox’s (1958) logistic recalibration framework and the STRATOS guidance by Van Calster et al. (2019), consider a binary outcome Y_i ∈ {0, 1} and a model’s predicted event probability p̂_i ∈ (0, 1) for validation subject i = 1, …, N.

1. Linear Predictor (Log-Odds Transformation)

First, convert each predicted probability p̂_i into its log-odds linear predictor L_i:

L_i = logit(p̂_i) = ln( p̂_i / (1 − p̂_i) )

2. Calibration-in-the-Large (CITL, α)

Calibration-in-the-large (CITL) compares the mean predicted risk across the validation cohort to the overall observed event proportion. On the log-odds scale, CITL is the intercept α estimated by fitting an intercept-only logistic regression model with L_i entered as a fixed offset (constraining the slope to exactly 1):

logit( P(Y_i = 1) ) = α + 1 · L_i

3. Calibration Slope (β)

The calibration slope β is estimated by fitting an unconstrained univariate logistic regression of the observed binary outcome Y_i on the linear predictor L_i:

logit( P(Y_i = 1) ) = α_recal + β · L_i

4. Integrated Calibration Index (ICI), E₅₀, and E₉₀

Proposed by Austin and Steyerberg (2019) to overcome the binning artifacts of the Hosmer-Lemeshow test and capture non-linear local miscalibration, the Integrated Calibration Index (ICI) fits a flexible non-parametric smoother (such as LOESS or restricted cubic splines) of Y_i on p̂_i to obtain a smoothed observed probability p̂_i^c for each subject, then computes the mean absolute difference across the empirical distribution of predictions:

ICI = (1 / N) · Σ_{i=1..N} | p̂_i^c − p̂_i |
E_50 = median( | p̂_i^c − p̂_i | )
E_90 = 90th percentile( | p̂_i^c − p̂_i | )

Symbol Reference

• Y_i: Observed binary outcome for subject i (1 = event occurred, 0 = no event).
• p̂_i: Predicted event probability generated by the prediction model for subject i.
• L_i: Logit-transformed predicted probability, ln(p̂_i / (1 − p̂_i)).
• α (CITL): Calibration-in-the-large intercept when the slope is fixed at 1 (ideal = 0).
• β: Calibration slope measuring prediction spread on the log-odds scale (ideal = 1).
• p̂_i^c: Smoothed calibrated probability at p̂_i estimated from the flexible calibration curve.

Worked Example With Arithmetic Shown

Suppose an external validation study evaluates a risk model across N = 1,000 patients divided into four risk strata of 250 patients each. The model predicts probabilities of 0.05, 0.15, 0.35, and 0.75 across the four strata, while the observed event counts in the validation cohort are 22, 42, 70, and 140 (total observed events = 274; observed prevalence = 27.4%).

Stratum (nk = 250)	Predicted Risk (p̂k)	Linear Predictor Lk = ln(p̂k/(1−p̂k))	Observed Events (Ok)	Observed Proportion (p̂kc)	Observed Log-Odds ln(p̂kc/(1−p̂kc))	Absolute Error |p̂kc − p̂k|
Stratum 1 (Low)	0.050	ln(0.05 / 0.95) = −2.9444	22	22 / 250 = 0.088	ln(0.088 / 0.912) = −2.3383	|0.088 − 0.050| = 0.038
Stratum 2 (Low-Mid)	0.150	ln(0.15 / 0.85) = −1.7346	42	42 / 250 = 0.168	ln(0.168 / 0.832) = −1.5999	|0.168 − 0.150| = 0.018
Stratum 3 (High-Mid)	0.350	ln(0.35 / 0.65) = −0.6190	70	70 / 250 = 0.280	ln(0.280 / 0.720) = −0.9445	|0.280 − 0.350| = 0.070
Stratum 4 (High)	0.750	ln(0.75 / 0.25) = +1.0986	140	140 / 250 = 0.560	ln(0.560 / 0.440) = +0.2412	|0.560 − 0.750| = 0.190

1. Mean Predicted Risk vs. Observed Event Rate: Mean predicted risk = (0.05 + 0.15 + 0.35 + 0.75) / 4 = 0.325 (32.5%), whereas the overall observed event rate is 274 / 1,000 = 0.274 (27.4%). Because mean predicted risk exceeds observed prevalence, the model globally overestimates risk.
2. Calibration-in-the-Large (CITL, α): Fitting the offset logistic model logit(P(Y=1)) = α + offset(L) across the 1,000 observations yields α = −0.311 (95% CI: −0.468 to −0.154). Since α < 0, predicted risks are systematically too high on average.
3. Calibration Slope (β): Fitting the logistic recalibration model logit(P(Y=1)) = α_recal + β · L yields β = 0.631 (95% CI: 0.525 to 0.737) with recalibration intercept α_recal = −0.511. Because β = 0.631 < 1.0, the model exhibits classic overfitting: low predicted risks (0.05) underestimate the true risk (0.088), while high predicted risks (0.75) substantially overestimate the true risk (0.560).
4. Integrated Calibration Index (ICI): Across the four equal-sized strata (250 patients each), the weighted average absolute difference between smoothed observed risk and predicted risk is:
   ICI = (250×0.038 + 250×0.018 + 250×0.070 + 250×0.190) / 1000 = 0.0790 (7.9 percentage points).

How to Interpret CITL, Calibration Slope, and the Four-Level Van Calster Hierarchy

Use the two tables below to interpret empirical calibration estimates and classify a validation study within the four-level STRATOS calibration hierarchy (Van Calster et al., 2019).

Table 1: Diagnostic Interpretation of CITL (α), Calibration Slope (β), and ICI

Metric & Value Range	Statistical Diagnosis	Clinical Effect on Predicted Risks	Typical Root Cause in Model Development
CITL (α) = 0	Calibrated-in-the-large	Average predicted risk matches observed cohort incidence	Baseline hazard/intercept matches target population
CITL (α) < 0	Systematic risk overestimation	Predicted probabilities are too high on average	Validation cohort has lower event incidence (e.g., primary care vs. tertiary hospital)
CITL (α) > 0	Systematic risk underestimation	Predicted probabilities are too low on average	Validation cohort has higher event incidence or more severe case-mix
Slope (β) = 1.0	Ideal spread on logit scale	Predicted risk differences reflect true log-odds gradients	Appropriate shrinkage/penalization and matching predictor effects
Slope (β) < 1.0	Predictions are too extreme	Low risks are underestimated; high risks are overestimated	Overfitting during development (too many parameters per event, unpenalized selection)
Slope (β) > 1.0	Predictions are too narrow	Low risks are overestimated; high risks are underestimated	Underfitting, excessive L1/L2 shrinkage, or broader case-mix heterogeneity
ICI → 0 (e.g., < 0.02)	High moderate calibration	Smoothed calibration curve closely tracks the 45° diagonal	Both global parameters and local functional forms are well specified

Table 2: The Four Levels of the Van Calster (2019) Calibration Hierarchy

Hierarchy Level	Formal Mathematical Condition	What It Guarantees	What It Can Still Miss
1. Mean Calibration (CITL)	E[p̂] = E[Y] (or offset intercept α = 0)	Average predicted risk equals overall event rate	Severe overfitting (β ≪ 1) where low-risk under-prediction cancels high-risk over-prediction
2. Weak Calibration	α = 0 and β = 1 simultaneously in logistic recalibration	No systematic global over/under-estimation and correct average spread	Non-linear local miscalibration (e.g., S-shaped or U-shaped deviations captured by ICI)
3. Moderate Calibration	P(Y = 1 | p̂ = p) = p for all p ∈ (0, 1)	Among all patients given predicted risk p, the observed event rate is p (smooth curve on 45° line)	Subgroup miscalibration where two covariate profiles share risk p but have opposite errors
4. Strong Calibration	P(Y = 1 | X = x) = p̂(x) for every covariate vector x	Exact calibration inside every possible subgroup and covariate pattern	Unattainable in finite samples with continuous predictors unless the true data-generating model is known

Assumptions and Common Mistakes

• Mistaking internal apparent slope for external validation: In an unpenalized standard maximum-likelihood logistic regression evaluated on its own development dataset, CITL is algebraically 0.0 and the calibration slope is algebraically 1.0. Informative calibration slope evaluation requires bootstrap optimism correction, cross-validation, or an independent external validation cohort.
• Estimating CITL and slope from the same regression equation: In logit(Y) = α_recal + β · L, the intercept α_recal is conditioned on β and is not Calibration-in-the-Large. CITL (α) must be estimated from a separate model where the slope on L is constrained to 1 via an offset(L) term.
• Boundary values (p̂ = 0 or p̂ = 1) in machine learning models: Random forests or uncalibrated decision trees can output exact 0 or 1 probabilities, for which ln(p̂ / (1 − p̂)) is undefined (−∞ or +∞). Clip probabilities to [1e-6, 1 - 1e-6] or apply Platt / isotonic post-hoc calibration before computing L_i.

How to Report Calibration Slope and ICI (TRIPOD+AI and APA Style)

Under the 2024 TRIPOD+AI reporting guideline (Collins et al., 2024), validation reports must present point estimates and 95% confidence intervals for both CITL and calibration slope alongside a smoothed calibration plot and non-parametric summary metrics:

“In the external validation cohort (N = 1,000; 274 events, 27.4%), the model demonstrated moderate risk overestimation with a calibration-in-the-large (CITL) of −0.31 (95% CI [−0.47, −0.15]) and evidence of overfitting with a calibration slope of 0.63 (95% CI [0.53, 0.74]). Flexible LOESS calibration analysis yielded an Integrated Calibration Index (ICI) of 0.079, a median absolute calibration error (E₅₀) of 0.054, and a 90th-percentile error (E₉₀) of 0.190.”

How to Calculate Calibration Slope, CITL, and ICI in R, Python, and Excel

1. R (Using base glm() and loess())

For general model fitting diagnostics in R, see our guide to Logistic Regression in R. To compute CITL, calibration slope, and LOESS-based ICI:

# Given vectors: y (0/1 binary outcomes) and p_hat (predicted probabilities in (0,1))
L <- qlogis(p_hat)  # log(p_hat / (1 - p_hat))

# 1. Calibration-in-the-Large (CITL, slope fixed at 1 via offset)
fit_citl <- glm(y ~ 1, offset = L, family = binomial(link = 'logit'))
citl <- coef(fit_citl)[1]
citl_ci <- confint.default(fit_citl)

# 2. Calibration Slope (unconstrained slope on L)
fit_slope <- glm(y ~ L, family = binomial(link = 'logit'))
cal_slope <- coef(fit_slope)['L']
slope_ci <- confint.default(fit_slope)['L', ]

# 3. Integrated Calibration Index (ICI), E50, E90 via LOESS smoother
loess_cal <- loess(y ~ p_hat, degree = 2, span = 0.75)
p_cal <- predict(loess_cal, newdata = p_hat)
abs_diff <- abs(p_cal - p_hat)
ICI <- mean(abs_diff, na.rm = TRUE)
E50 <- quantile(abs_diff, 0.50, na.rm = TRUE)
E90 <- quantile(abs_diff, 0.90, na.rm = TRUE)

2. Python (Using statsmodels and numpy)

import numpy as np
import statsmodels.api as sm
from statsmodels.nonparametric.smoothers_lowess import lowess

# y: 1D numpy array of 0/1 outcomes; p_hat: 1D array of predicted probabilities
p_clipped = np.clip(p_hat, 1e-6, 1 - 1e-6)
L = np.log(p_clipped / (1.0 - p_clipped))

# 1. Calibration-in-the-Large (CITL) with offset=L
citl_model = sm.GLM(y, np.ones((len(y), 1)), family=sm.families.Binomial(), offset=L).fit()
citl = float(citl_model.params[0])

# 2. Calibration Slope
X_slope = sm.add_constant(L)
slope_model = sm.GLM(y, X_slope, family=sm.families.Binomial()).fit()
cal_slope = float(slope_model.params[1])

# 3. Integrated Calibration Index (ICI), E50, E90 via LOWESS
p_cal = lowess(endog=y, exog=p_clipped, frac=0.75, return_sorted=False)
abs_err = np.abs(p_cal - p_clipped)
ici = float(np.mean(abs_err))
e50 = float(np.percentile(abs_err, 50))
e90 = float(np.percentile(abs_err, 90))

3. Excel (Grouped Log-Odds Approximation)

• In Column A (A2:A11), enter decile mean predicted probabilities p̂_k; in Column B (B2:B11), enter decile observed event proportions O_k.
• In Column C (Predicted Log-Odds), enter =LN(A2/(1-A2)); in Column D (Observed Log-Odds), enter =LN(B2/(1-B2)).
• Compute the approximate grouped calibration slope using =SLOPE(D2:D11, C2:C11) and the grouped absolute calibration error using =AVERAGE(ABS(B2:B11 - A2:A11)).

Frequently Asked Questions

How do you interpret Calibration-in-the-Large (CITL) and the Calibration Slope in a clinical prediction model?

CITL (α) measures whether average predicted risk matches the overall observed event rate on the log-odds scale (α = 0 is ideal; α < 0 means systematic risk overestimation; α > 0 means systematic underestimation). The calibration slope (β) measures the spread of predicted risks (β = 1 is ideal; β < 1 indicates overfitting with predictions that are too extreme; β > 1 indicates underfitting with predictions that are too narrow).

What are the four levels of the Van Calster calibration hierarchy (mean, weak, moderate, strong)?

Level 1 (Mean calibration) requires average predicted risk to equal the overall observed event rate. Level 2 (Weak calibration) requires CITL α = 0 and calibration slope β = 1 simultaneously. Level 3 (Moderate calibration) requires the smoothed calibration curve of observed risk against predicted risk to lie on the 45-degree identity line across the entire prediction range. Level 4 (Strong calibration) requires accurate calibration inside every possible covariate pattern.

How do smoothed calibration curves and the Integrated Calibration Index (ICI) reveal risk over- or under-estimation?

A smoothed calibration curve uses LOESS or restricted cubic splines to show local regions where observed risk lies below the 45-degree diagonal (risk overestimation) or above the diagonal (risk underestimation). The Integrated Calibration Index (ICI) summarizes this curve by calculating the empirical-density-weighted mean absolute vertical distance between the smoothed calibration curve and the diagonal identity line.

Sources

• Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW; Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17(1):230. https://pmc.ncbi.nlm.nih.gov/articles/PMC6912996/
• Austin PC, Steyerberg EW. The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Statistics in Medicine. 2019;38(21):4051–4065. https://pmc.ncbi.nlm.nih.gov/articles/PMC6771557/
• Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. https://www.bmj.com/content/385/bmj-2023-078378
• Cox DR. Two further applications of a model for binary regression. Biometrika. 1958;45(3/4):562–565.