Written and maintained by CASRAI Editorial Board
Last updated
The Brier score is the mean squared difference between a predicted probability and what actually happened. It runs from 0 to 1 for a yes/no outcome, and lower is better. The number only means something next to a reference line: a model that gives every case the observed event rate already earns a score of p(1-p), so a “low” Brier score on a rare event can still be no better than ignoring your predictors.
What is the Brier score?
The Brier score measures the accuracy of probability forecasts for events that either happen or do not. scikit-learn describes it as “the mean squared difference between the predicted probability and the actual outcome”, and as a strictly proper scoring rule, meaning a forecaster minimises the expected score only by reporting their true belief. Brier (1950, Monthly Weather Review) is the original reference. It is also used to evaluate clinical prediction models and machine-learning classifiers.
It is “calculated like MSE” (mean squared error), but the forecast is a probability between 0 and 1 and the observation is 0 or 1. A perfect forecaster scores 0; the worst possible binary forecaster, who says 100% for everything that does not happen and 0% for everything that does, scores 1.
Brier score formula
For a binary outcome, with N cases:
BS = (1/N) × Σ (pi − yi)²
- N: the number of forecasts (cases).
- pi: the predicted probability that case i has the event, between 0 and 1.
- yi: the observed outcome, 1 if the event occurred and 0 if not.
- Σ: sum over all N cases.
Multiclass Brier score
With more than two classes the squared error is summed over classes as well: BS = (1/N) × Σi Σc (yic − pic)², where yic is 1 if case i belongs to class c. In scikit-learn this version ranges from 0 to 2, not 0 to 1, and the library only rescales by one half for the binary case by default. Never compare a multiclass value to a binary one.
Worked example with the arithmetic shown
Four patients, a model that outputs a risk, and the observed outcomes (this is the same four-case example used in the scikit-learn documentation):
| Case | Predicted p | Observed y | (p − y) | (p − y)² |
|---|---|---|---|---|
| 1 | 0.1 | 0 | 0.1 | 0.01 |
| 2 | 0.9 | 1 | −0.1 | 0.01 |
| 3 | 0.8 | 1 | −0.2 | 0.04 |
| 4 | 0.3 | 0 | 0.3 | 0.09 |
Sum of squared errors = 0.01 + 0.01 + 0.04 + 0.09 = 0.15. Divide by N = 4: Brier score = 0.15 / 4 = 0.0375.
Now turn each forecast into a hard yes/no at a 0.5 threshold (cases 2 and 3 become 1, cases 1 and 4 become 0). Every call is correct, so the score is 0. The same predictions score 0.0375 as probabilities and 0 as labels: the Brier score penalises confidence that is not fully certain, which is why it must be computed on the probabilities and never on thresholded classes.
What is a good Brier score?
There is no universal cut-off. A value is only good relative to the event rate in your data. Two reference forecasts bound the problem:
- Predicting 0.5 for every case scores 1/4 = 0.25 whatever the event rate.
- Predicting the overall prevalence p for every case scores p − p² = p(1 − p) (Hoessly, 2025). At a prevalence of 10% that is 0.09.
| Event rate (prevalence) | Score of the “predict the prevalence for everyone” forecast, p(1 − p) |
|---|---|
| 1% | 0.0099 |
| 5% | 0.0475 |
| 10% | 0.0900 |
| 20% | 0.1600 |
| 30% | 0.2100 |
| 50% | 0.2500 |
(Each row is computed as p × (1 − p).) A model with a Brier score of 0.09 on data with a 10% event rate has done no better than a constant forecast that uses no predictors at all. The same 0.09 on a 30% event rate beats the 0.21 benchmark clearly.
To express this as one number, divide by the benchmark: 1 − BS / [p(1 − p)]. In the worked example the event rate is 2 of 4 = 0.5, the benchmark is 0.25, and 1 − 0.0375 / 0.25 = 0.85. A value of 0 means no better than the benchmark, and 1 means perfect.
Reading guidance
| Result | How to read it |
|---|---|
| Lower than the p(1 − p) benchmark | The predictions carry information beyond the overall event rate. |
| Equal to or above the benchmark | No gain over giving everyone the average risk. |
| Lower on a different dataset | Not comparable unless the event rates match; the benchmark moves with prevalence. |
| Exactly 0 | Reachable only if every true risk is 0 or 1; it is not a realistic target for most clinical models. |
Assumptions and common mistakes
Hoessly (2025, arXiv preprint) lists several misconceptions about the Brier score:
- “A score of 0 means a perfect model.” The score is 0 under perfect predictions only if every true risk is 0 or 1. If outcomes are genuinely uncertain, even the ideal model scores above 0.
- “Lower is always better across datasets.” Lower is better on the same data. Across datasets the event rate changes the benchmark.
- “A low score means good calibration.” In the paper’s words, “A low Brier score does not necessarily indicate good calibration of a model.” Check calibration separately, for example with calibration-in-the-large and a calibration curve.
Other mistakes: scoring thresholded class labels instead of probabilities, and mixing binary and multiclass scales (0 to 1 versus 0 to 2).
Brier score vs calibration, AUC and NRI
Discrimination (can the model rank cases that have the event above those that do not) and calibration (do predicted probabilities match observed frequencies) are different properties, and the Brier score blends both into one squared-error number. The ROC curve and AUC look only at ranking, so a model can have a strong AUC and a poor Brier score if its probabilities are systematically too high or too low. The Brier score is also one of the measures recommended instead of the net reclassification index: see net reclassification improvement for why poor calibration can make a model look better under that metric. If the model is used to flag patients, the positive predictive value and spectrum bias in your sample matter for how the probabilities generalise.
How to report a Brier score
State the number, the sample size, the event rate and the benchmark it should be read against. Example wording:
The Brier score of the model was 0.0375 (N = 4, event rate = 50%; reference score for a constant forecast = 0.25).
When you can resample the data, add a confidence interval for the score, and report calibration alongside it rather than relying on the Brier score alone. Cite Brier (1950) for the score itself.
How to calculate the Brier score in Excel, R or Python
All three give 0.0375 on the worked example.
Excel
Put the observed outcomes in B2:B5 and the predicted probabilities in C2:C5, then:
=SUMXMY2(B2:B5, C2:C5) / COUNT(B2:B5)R
y <- c(0, 1, 1, 0)
p <- c(0.1, 0.9, 0.8, 0.3)
mean((p - y)^2) # 0.0375Python
from sklearn.metrics import brier_score_loss
y_true = [0, 1, 1, 0]
y_prob = [0.1, 0.9, 0.8, 0.3]
brier_score_loss(y_true, y_prob) # 0.0375If you pass hard labels instead of probabilities to brier_score_loss you get the thresholded score (0.0 in this example), not the probability score.
FAQ
Is a lower Brier score better?
Yes. Lower values mean the predicted probabilities were closer to the outcomes. Compare it to the p(1 − p) benchmark for your event rate before calling it good.
What is the range of the Brier score?
0 to 1 for a binary outcome. For the multiclass sum in scikit-learn the range is 0 to 2.
Is the Brier score the same as mean squared error?
It is calculated like mean squared error, applied to probability forecasts and 0/1 outcomes.
Does a low Brier score mean the model is well calibrated?
Not necessarily. Assess calibration separately with a calibration curve and calibration-in-the-large.
Why is the Brier score called a proper scoring rule?
Because its expected value is optimised by reporting your honest probability, so it cannot be gamed by shading forecasts. Gneiting and Raftery (2007) treat strictly proper scoring rules formally.
Sources
Checked 2026-10-29.
- scikit-learn documentation, sklearn.metrics.brier_score_loss: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html (checked 2026-10-29).
- scores package (NCI) tutorial, Brier Score, v2.8.0: https://scores.readthedocs.io/en/latest/tutorials/Brier_Score.html (checked 2026-10-29).
- Hoessly, L. (2025). On misconceptions about the Brier score in binary prediction models. arXiv:2504.04906v4 (preprint): https://arxiv.org/html/2504.04906v4 (checked 2026-10-29).
- Brier, G. W. (1950). Monthly Weather Review. Reference listed in the scores tutorial above.
- Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association. Reference listed in the scores tutorial above.
