Written and maintained by CASRAI Editorial Board
Last updated
Overfitting is when a model matches its training data so closely that it fails to make correct predictions on new data. The model has learned the noise and quirks of one sample instead of the pattern that generalises. In scikit-learn, the check is mechanical: compare the training score with the cross-validated validation score. If the training score is high and the validation score is low, the estimator is overfitting.
This guide gives that rule, the two scikit-learn functions that apply it (validation_curve and learning_curve), and a worked polynomial example where the cross-validated error grows by orders of magnitude as the model becomes too flexible.
What overfitting means in research modelling
Google’s Machine Learning Crash Course defines overfitting as creating a model that matches the training set so closely that it fails to make correct predictions on new data. It names two causes: the training set does not adequately represent real-life data, or the model is too complex. A model that generalises well, by contrast, makes good predictions on new data.
For a researcher this matters whenever a model is fitted and then reported: a prediction model for a clinical outcome, a classifier on imaging features, a regression with many candidate predictors. Performance measured on the same data that fitted the model describes that sample, not the population. The general background on machine learning study design is in What Is Machine Learning?, and the related idea of keeping a model no more complex than the data justify is covered in Parsimonious Model.
Overfitting versus underfitting
Underfitting is the opposite failure. scikit-learn’s own example shows a linear function (a polynomial of degree 1) that is not sufficient to fit the training samples, and calls this underfitting. Its rule for telling the two apart is: if the training score and the validation score are both low, the estimator is underfitting; if the training score is high and the validation score is low, the estimator is overfitting; otherwise it is working very well. A low training score with a high validation score is usually not possible.
| Training score | Validation score | Reading (scikit-learn documentation) |
|---|---|---|
| Low | Low | Underfitting |
| High | Low | Overfitting |
| High | High | Working very well |
| Low | High | Usually not possible |
Bias and variance in the same picture
The scikit-learn user guide describes bias as the average error for different training sets and variance as how sensitive the estimator is to varying training sets. An underfitting estimator is one dominated by bias error; an overfitting one is dominated by variance error.
Install or access (official source only)
The examples use scikit-learn, installed from the Python Package Index with the command given in the official scikit-learn installation instructions:
pip install -U scikit-learnEnvironment set-up, data loading and a first model are covered step by step in the scikit-learn tutorial. Both curve functions below live in sklearn.model_selection.
Step by step: detect overfitting with scikit-learn
- Hold out a test set first. Split the data before any tuning, as shown in Train Test Split in Python. The learning-curve documentation warns that if you optimise hyperparameters on a validation score, that score becomes biased and no longer a good estimate of generalisation; to get a proper estimate you compute the score on another test set.
- Score with cross-validation, not on the training data. The scikit-learn polynomial example uses
cross_val_score(pipeline, X[:, np.newaxis], y, scoring="neg_mean_squared_error", cv=10)and reports the error as-scores.mean(). Cross-validation gives each model a validation score on data it was not fitted to. - Sweep one complexity setting with
validation_curve. The documented call isvalidation_curve(SVC(kernel="linear"), X, y, param_name="C", param_range=np.logspace(-7, 3, 3)). It returnstrain_scoresandvalid_scores, one row per parameter value. Plot them withValidationCurveDisplay.from_estimator. - Vary the amount of training data with
learning_curve. The documented call islearning_curve(SVC(kernel="linear"), X, y, train_sizes=[50, 80, 110], cv=5). It returnstrain_sizes,train_scoresandvalid_scores. Plot them withLearningCurveDisplay.from_estimator. - Apply the rule from the table above to each setting or training size, then choose the setting where the validation score is best, and confirm it once on the untouched test set.
Output and how to read it
A worked example: polynomial degree 1, 4 and 15
The scikit-learn “Underfitting vs. Overfitting” example fits polynomials of degree 1, 4 and 15 to samples of a true function, using degrees = [1, 4, 15] and ten-fold cross-validated mean squared error (MSE). It reports that a degree-4 polynomial approximates the true function almost perfectly, while for higher degrees the model overfits the training data, that is, it learns the noise of the training data. The validation MSE is the quantitative check: the higher it is, the less likely the model generalises correctly.
| Polynomial degree | Cross-validated MSE in the example’s figure titles | Reading |
|---|---|---|
| 1 | 4.08e-01 (+/- 4.25e-01) | Underfits: a straight line cannot follow the function |
| 4 | 4.32e-02 (+/- 7.08e-02) | Close to the true function |
| 15 | 1.83e+08 (+/- 5.48e+08) | Overfits: error explodes on held-out folds |
The degree-15 figure is not a modest rise but a collapse of roughly nine orders of magnitude compared with degree 4, which is why the training fit alone is a poor guide. The exact figures are those printed in the documentation’s figures and can differ between scikit-learn versions and random seeds; the pattern (best in the middle, far worse at both ends, catastrophic at high complexity) is the point.
Reading a validation curve
Move along the complexity axis. Where both scores are low, the model is underfitting. Where the training score keeps rising but the validation score falls away, the model is overfitting. The best setting is the one where the validation score peaks.
Reading a learning curve
A learning curve is, in the documentation’s words, a tool to find out how much you benefit from adding more training data and whether the estimator suffers more from a variance error or a bias error. In the documented SVM example, for small amounts of data the training score is much greater than the validation score, and adding more training samples will most likely increase generalisation. More data helps only if the true function is too complex to be approximated by an estimator with a lower variance; if the estimator is already a poor fit at every training size, the fix is a more flexible model, not more rows.
Troubleshooting
- The validation score is much worse than the training score. That is the overfitting signature. Reduce model complexity (for the polynomial example, lower the degree), or, per the learning-curve documentation, add training data if the curve shows the gap closing as the sample grows. Penalised regression such as ridge and LASSO is one way to constrain complexity; see Ridge vs LASSO Regression.
- Both scores are low. That is underfitting, not overfitting. Use a more flexible model or better features.
- The tuned model looks excellent on the validation folds but disappoints later. The validation score was used to choose the settings, so it is biased. Report performance on a separate test set that played no part in tuning.
- The training set does not look like the population. Google names an unrepresentative training set as a cause of overfitting in its own right. No complexity setting repairs that; the sample has to change.
- Few outcome events relative to predictors. Sample size drives how much a model can be trusted. See Events per Variable and the Minimum Sample Size for Regression.
Free alternatives: scikit-learn tools for the same check
All of these ship with scikit-learn and answer the same question from a different angle.
| Tool | What it varies | What it tells you (per the scikit-learn documentation) |
|---|---|---|
cross_val_score |
Nothing; scores one model on held-out folds | A validation estimate such as MSE, as in the degree 1, 4, 15 example |
validation_curve / ValidationCurveDisplay |
One hyperparameter, for example C |
Training and validation scores across the parameter range, so under- and overfitting regions show |
learning_curve / LearningCurveDisplay |
Number of training samples | Whether more data helps, and whether variance or bias error dominates |
FAQ
What is the difference between overfitting and underfitting?
Underfitting means the model is too simple to capture the pattern, so both training and validation scores are low. Overfitting means the model fits the training data, noise included, so the training score is high and the validation score is low.
How do I know if my model is overfitting?
Compare the training score with a cross-validated validation score. A high training score with a low validation score is overfitting under the scikit-learn rule. Google’s crash course describes the same pattern in loss curves: training loss declines or holds steady while validation loss increases.
How do I prevent overfitting?
Address the two causes Google names. If the model is too complex, reduce its complexity. If the training set does not represent real-life data, collect a more representative sample. Use a validation curve to pick the complexity, a learning curve to judge whether more data would help, and a held-out test set to report the final result.
Does more data fix overfitting?
Sometimes. The learning-curve documentation says collecting more data pays off if the true function is too complex to be approximated by an estimator with a lower variance. A learning curve shows whether the validation score is still improving as the sample grows.
Sources
- scikit-learn documentation, Underfitting vs. Overfitting (checked 2026-10-29).
- scikit-learn documentation, Validation curves: plotting scores to evaluate models (checked 2026-10-29).
- Google for Developers, Machine Learning Crash Course, Overfitting (checked 2026-10-29).
- scikit-learn documentation, Installing scikit-learn (installation command, checked 2026-10-29).
