Written and maintained by CASRAI Editorial Board
Last updated
Cross-validation is a resampling method that estimates how well a model will perform on data it has not seen, by repeatedly holding part of the data out, fitting on the rest, and averaging the held-out scores. This guide is for researchers and analysts who need to run it correctly and report it in a methods section: it covers the definition, the formula, a worked 5-fold example with the arithmetic, how to read the result, leakage and nested cross-validation, and code in Excel, R and Python.
What is cross-validation and how does it work?
Cross-validation splits the data into parts, trains the model on some of them, scores it on the part left out, and repeats until every part has served as the held-out set. In the scikit-learn user guide’s description of k-fold cross-validation, the training set is split into k smaller sets; a model is trained on k-1 of the folds and validated on the remaining fold, and the performance measure reported is the average of the values computed in the loop.
The reason to do this is that a model scored on the same data it was fitted to looks better than it will on new data. A single train/test split gives one held-out estimate that depends on which rows landed in the test set; cross-validation uses every row for testing exactly once and gives several estimates instead of one. It is the standard tool for spotting overfitting and for comparing candidate models or tuning settings.
Cross-validation formula
For k folds, the cross-validation estimate is the mean of the fold scores:
CV = (1/k) × Σ si for i = 1 to k
- k: the number of folds (5 and 10 are the usual choices).
- si: the score of the model trained on all folds except fold i, measured on fold i. The score can be accuracy, RMSE, AUC or any other metric you choose in advance.
- Σ: the sum over all k folds.
With equal fold sizes this is the same as pooling all held-out predictions. With unequal fold sizes, state whether you averaged fold scores or pooled predictions. The spread across folds is summarised with the usual sample standard deviation, SD = √[ Σ (si − CV)² / (k − 1) ].
What is 5-fold cross-validation? A worked example
5-fold cross-validation sets k = 5: the data are divided into five folds, and the model is fitted five times, each time holding out a different fold. Suppose a classifier is evaluated on n = 100 observations, so each fold holds 20. The numbers below are illustrative, chosen to make the arithmetic easy to follow.
| Fold held out | Correct of 20 | Accuracy si | si − mean | (si − mean)² |
|---|---|---|---|---|
| 1 | 17 | 0.85 | 0.00 | 0.0000 |
| 2 | 18 | 0.90 | +0.05 | 0.0025 |
| 3 | 16 | 0.80 | −0.05 | 0.0025 |
| 4 | 17 | 0.85 | 0.00 | 0.0000 |
| 5 | 17 | 0.85 | 0.00 | 0.0000 |
- Mean: (0.85 + 0.90 + 0.80 + 0.85 + 0.85) / 5 = 4.25 / 5 = 0.85. Pooled, 85 of 100 predictions are correct, the same value because the folds are equal in size.
- Sum of squared deviations: 0.0025 + 0.0025 = 0.0050.
- Sample variance: 0.0050 / (5 − 1) = 0.00125. Sample SD: √0.00125 = 0.035 (0.0354 before rounding).
- Fold-to-fold standard error: 0.0354 / √5 = 0.0158. Treat this as descriptive only: the training sets of different folds overlap, so the fold scores are not independent and this figure is not a valid confidence interval for true accuracy.
Interpretation: what the result means
| What you see | Reading |
|---|---|
| Mean of fold scores | Your estimate of out-of-sample performance for the modelling procedure, given the metric and the data. |
| Small SD across folds | Performance was similar whichever rows were held out. |
| Large SD across folds | Performance depends heavily on which rows are held out; the mean is less trustworthy, especially with small n. |
| Cross-validated score far below the training score | The usual signature of overfitting. |
| Tuned with the same folds you report | The score is optimistically biased; use nested cross-validation (below). |
k-fold, leave-one-out, repeated and nested cross-validation
k-fold
The default choice. In scikit-learn, cross_val_score with an integer cv uses KFold, or StratifiedKFold when the estimator is a classifier. By default no shuffling occurs, so if rows are ordered (by time, site or class) shuffle with a fixed seed.
Leave-one-out (LOO)
LOO is k-fold with k = n: it builds n models, each tested on one observation. The scikit-learn guide notes that LOO often results in high variance as an estimator of the test error, and that most authors and empirical evidence suggest 5- or 10-fold cross-validation should be preferred.
Repeated k-fold
RepeatedKFold and RepeatedStratifiedKFold repeat k-fold with different random splits, which reduces dependence on one particular partition.
Grouped cross-validation
When several rows come from the same subject, patient or experiment, use GroupKFold so that the same group never appears in both the training and test sets (StratifiedGroupKFold when you also need class balance). Random splits on clustered data put near-duplicates on both sides and inflate the score.
Nested cross-validation
If you tune hyperparameters and report performance with the same folds, information leaks from the test folds into model selection and the score is overly optimistic; scikit-learn’s nested cross-validation example puts it this way and notes the size of the effect depends on dataset size and model stability. Nested cross-validation puts the tuning (for example GridSearchCV) in an inner loop and estimates generalisation error by averaging test scores over the outer splits. In scikit-learn’s own iris demonstration the non-nested score was only slightly higher (average difference 0.0076 over 30 trials), so the gap can be small, but you cannot know that in advance.
Is 99% accuracy overfitting? Is 80% accuracy good?
Neither number means anything alone. A cross-validated 99% is not by itself evidence of overfitting, because overfitting is a gap between training and held-out performance. But 99% should prompt two checks: whether the classes are imbalanced (a model that always predicts the majority class scores 99% when 99% of cases are in it), and whether information has leaked into the test folds. Likewise, 80% is good or poor only relative to a baseline: compare it with the majority-class rate, a null model or a simple published benchmark for the same data, and report a metric suited to the question (for imbalanced outcomes, sensitivity, specificity or AUC rather than accuracy alone).
Assumptions and common mistakes
- Preprocessing outside the loop. Standardisation, imputation and feature selection must be learned from the training folds only and applied to the held-out fold. The scikit-learn guide recommends a Pipeline so this happens automatically inside each fold. See data leakage.
- Tuning and reporting on the same folds: see nested cross-validation above.
- Ignoring grouping or time order in clustered or temporal data.
- Using
cross_val_predictas the performance estimate. The scikit-learn guide says its output is not an appropriate measure of generalisation error; usecross_val_scorefor that. - Treating the SD or fold-to-fold standard error as a confidence interval. Folds share training data (see the worked example). For an interval around a statistic, consider bootstrapping.
- Choosing the metric or k after seeing results. Fix them first.
How to report cross-validation (APA-style example)
Report enough to let a reader repeat the procedure: k, repeats, the metric, the mean and SD, the seed, stratification or grouping, and what sat inside the pipeline. Example wording:
Model performance was estimated with stratified 5-fold cross-validation, repeated 3 times (random seed = 42). Standardisation and model fitting were performed within each training fold. Mean accuracy across folds was 0.85 (SD = 0.04). Hyperparameters were tuned in an inner 5-fold loop (nested cross-validation).
Use the numbers from your own analysis; the sentence above only shows the elements to include. For related reporting of error metrics, see root mean square error.
Cross-validation in Python, R and Excel
Python (scikit-learn)
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=cv, scoring="accuracy")
print(scores.mean(), scores.std(ddof=1))The pipeline refits the scaler inside each training fold. Note ddof=1, which gives the sample SD used above.
R (base R, manual loop)
set.seed(42)
# df has a 0/1 outcome column y and predictor columns
k = 5
folds = sample(rep(1:k, length.out = nrow(df)))
acc = sapply(1:k, function(i) {
train = df[folds != i, ]
test = df[folds == i, ]
fit = glm(y ~ ., data = train, family = binomial)
mean(round(predict(fit, test, type = "response")) == test$y)
})
mean(acc); sd(acc)Excel
With observations in rows 2 to 101: put =RAND() in column B, =RANK(B2,$B$2:$B$101) in column C, and the fold number in column D with =MOD(C2-1,5)+1. Copy column B as values before fitting so the folds stop changing. For each fold, fit on rows where D is not that fold, score on rows where D equals it, enter the five scores in a column, and use =AVERAGE() and =STDEV.S().
Frequently asked questions
What is cross-validation in statistics?
A resampling method that estimates out-of-sample performance by repeatedly fitting on part of the data and scoring on the held-out remainder, then averaging the scores.
How many folds should I use?
The scikit-learn guide states that most authors and empirical evidence suggest 5- or 10-fold cross-validation should be preferred over leave-one-out.
Is cross-validation the same as a train/test split?
No. A split gives one estimate from one partition; cross-validation rotates the held-out portion so every observation is tested once. See train/test split in Python.
Can I use cross-validation to choose a regularisation strength?
Yes, but report performance with nested cross-validation or a separate untouched test set. Elastic net regression is a common case.
Sources
- scikit-learn developers. 3.1 Cross-validation: evaluating estimator performance, scikit-learn user guide. https://scikit-learn.org/stable/modules/cross_validation.html (read 2026-11-07).
- scikit-learn developers. Nested versus non-nested cross-validation. https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html (read 2026-11-07). The example cites Cawley and Talbot (2010) on selection bias in performance evaluation.
- Hastie, T., Tibshirani, R., and Friedman, J. The Elements of Statistical Learning (2nd ed.), chapter 7, and James, G., Witten, D., Hastie, T., and Tibshirani, R. An Introduction to Statistical Learning, chapter 5: standard textbook treatments of model assessment and resampling.
