Written and maintained by CASRAI Editorial Board
Last updated
The linearity assumption of linear regression says that the mean of the outcome is a straight-line (linear-in-parameters) function of the predictors you put in the model. You check it by plotting residuals against the fitted values and against each predictor, and you can back the picture with Ramsey’s RESET test. This guide works one example end to end: a line fitted to curved data produces R² = 0.949, yet the residual pattern and a RESET F of 8,776 show the model form is wrong.
What the linearity assumption means
Ordinary least squares fits y = β0 + β1x + ε. The assumption is that the expected value of the error ε is zero at every value of x, so the average of y really does follow that line. If the true relationship curves, the fitted line is systematically too high in some ranges of x and too low in others, and the coefficient estimates describe a relationship that does not exist.
“Linear” refers to the model being linear in its coefficients. A model with x and x² as predictors is still a linear regression; it just no longer assumes a straight line in x.
How to check linearity with residual plots
The NIST/SEMATECH e-Handbook of Statistical Methods names scatter plots of the residuals against the predictor variables in the model as a primary tool, and also recommends plotting residuals against potential predictors that are not yet in the model. Plots with no systematic structure indicate the model fits well; residuals with systematic structure indicate the form of the function can be improved. The handbook also states that no single residual plot can completely justify adopting a particular model.
Residuals versus fitted values
Plot residuals on the vertical axis and fitted values on the horizontal axis, and add a smooth conditional-mean line. The question, as the University of Wisconsin SSCC regression-diagnostics guide frames it, is whether the conditional mean stays close to zero across all fitted values. A funnel shape is a different problem (unequal variance); see how to read a residual plot for homoscedasticity and the Breusch-Pagan test.
Residuals versus each predictor
In a model with several predictors, a single fitted-value plot can hide which variable is the problem. Draw one residual plot per predictor. For the underlying bivariate shape, scatterplot interpretation shows how to judge form before you fit anything.
Ramsey RESET test for linearity
The Ramsey RESET test (Ramsey, 1969, Journal of the Royal Statistical Society, Series B, 31, 350-371) tests for functional-form misspecification. It refits the model with powers of the fitted values added as extra predictors and asks whether they improve the fit.
Restricted model: y = β0 + β1x + ε. Augmented model: y = β0 + β1x + γ2ŷ² + … + γpŷp + ε. The null hypothesis is that every γ equals zero. In F form:
F = [(RSSR − RSSU) / q] / [RSSU / (n − k − q)]
- RSSR: residual sum of squares of the original (restricted) model.
- RSSU: residual sum of squares of the augmented (unrestricted) model.
- q: number of added terms (powers of ŷ).
- n: number of observations. k: number of coefficients in the original model, including the intercept.
- ŷ: the fitted value from the original model.
A software default matters here. R’s lmtest::resettest defaults to power = 2:3 (squared and cubed fitted values) with type = "fitted". In Python, statsmodels’ linear_reset takes power=3, which includes powers 2 through 3, so the defaults line up on terms, but statsmodels reports a chi-square (Wald) statistic unless you set use_f=True. Compare like with like before you compare p-values across programs.
Worked example with real numbers
Ten observations: x = 1, 2, …, 10 and y = 2.1, 3.9, 7.2, 11.8, 17.1, 24.3, 32.8, 42.1, 53.0, 65.4. These are illustrative numbers, not data from a study.
Step 1: fit the line. Mean of x = 5.5; mean of y = 259.7 / 10 = 25.97. Sxx = 82.5 and Sxy = 579.05, so the slope is 579.05 / 82.5 = 7.0188 and the intercept is 25.97 − 7.0188 × 5.5 = −12.633. The fitted line is ŷ = −12.633 + 7.0188x, with R² = 0.949 and RSSR = 219.372 on 8 residual degrees of freedom.
Step 2: look at the residuals.
| x | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Residual | 7.71 | 2.50 | −1.22 | −3.64 | −5.36 | −5.18 | −3.70 | −1.42 | 2.46 | 7.85 |
The signs run + + − − − − − − + +. That is a U shape: the line overshoots the middle and undershoots both ends. Despite an R² near 0.95, the residuals have clear systematic structure. The NIST handbook makes the same warning in general terms: a high R² does not guarantee that the model fits the data well. See how to interpret R-squared.
Step 3: RESET with squared fitted values (q = 1). Adding ŷ² as a predictor gives RSSU = 0.1748 on n − k − q = 10 − 2 − 1 = 7 degrees of freedom.
F = [(219.372 − 0.1748) / 1] / [0.1748 / 7] = 219.197 / 0.024976 = 8,776.2, with 1 and 7 degrees of freedom, p ≈ 4.2 × 10−12. With the default two terms (ŷ² and ŷ³, q = 2, 6 denominator degrees of freedom) the same data give F = 4,183.5, p ≈ 3.7 × 10−10. Both reject linearity.
Interpretation: what each result tells you
| What you see | Reading |
|---|---|
| Residuals show no systematic structure against fitted values and each predictor | Indicates the model form fits the data well (NIST e-Handbook 4.4.4.1). |
| Residuals show systematic structure, such as a U or arch | Indicates the form of the function can be improved (NIST e-Handbook 4.4.4.1). |
| High R² alone | Does not guarantee the model fits well (NIST e-Handbook 4.4.4). |
| RESET not significant | No evidence of neglected nonlinearity from that test; keep checking the plots. |
| RESET significant | The functional form is misspecified. The SSCC guide cautions that it is not a reason to thoughtlessly add several polynomial terms. |
Assumptions and common mistakes
- Relying on one plot. NIST notes no single residual plot can completely justify a model; check against fitted values and every predictor.
- Treating RESET as a diagnosis. It detects misspecification. It does not say whether the cause is a missing curve, a missing variable or a missing interaction.
- Stopping at R². The example above has R² = 0.949 and a clearly wrong model.
- Fixing and not re-checking. The SSCC guide lists changing the formula (adding or dropping variables, interactions) and adding polynomial terms as fixes, and advises re-checking all assumptions afterwards. In the example, adding x² gives ŷ = 1.5417 − 0.0687x + 0.6443x², R² = 0.99996 and RSS = 0.1748.
- Transforming blindly. A transformation of x or y can straighten a relationship, but it changes what the coefficients mean; read log and other data transformations first, and re-run the diagnostics on the transformed model.
Linearity is one of several checks; independence of errors is covered in the Durbin-Watson test guide, and the wider workflow in regression analysis.
How to report it (APA example)
“Residual plots showed a U-shaped pattern against the fitted values. A Ramsey RESET test with squared fitted values rejected the linear specification, F(1, 7) = 8776.24, p < .001. A quadratic term was added, and the revised model, R² = .99996, showed no systematic residual structure.”
Report the powers used, the degrees of freedom and the software, since defaults differ.
How to calculate it in Excel, R or Python
Excel
Put x in A2:A11 and y in C2:C11. In D2 enter =INTERCEPT($C$2:$C$11,$A$2:$A$11)+SLOPE($C$2:$C$11,$A$2:$A$11)*A2 and fill down for the fitted values; in B2 enter =D2^2 and fill down. Then RSSR is =INDEX(LINEST(C2:C11,A2:A11,TRUE,TRUE),5,2) and RSSU is =INDEX(LINEST(C2:C11,A2:B11,TRUE,TRUE),5,2). Compute F = ((RSS_R-RSS_U)/1)/(RSS_U/7) and the p-value with =F.DIST.RT(F,1,7).
R
library(lmtest)
fit <- lm(y ~ x)
resettest(fit, power = 2, type = "fitted") # squared fitted values only
resettest(fit) # default power = 2:3
plot(fit, which = 1) # residuals vs fittedPython
import numpy as np, statsmodels.api as sm
from statsmodels.stats.diagnostic import linear_reset
x = np.arange(1, 11.0)
y = np.array([2.1,3.9,7.2,11.8,17.1,24.3,32.8,42.1,53.0,65.4])
res = sm.OLS(y, sm.add_constant(x)).fit()
print(linear_reset(res, power=2, test_type="fitted", use_f=True))
# F=8776.24, p=4.2e-12, df_denom=7, df_num=1FAQ
What is the linearity assumption in linear regression?
It is the assumption that the outcome’s mean is a linear function of the model’s coefficients and predictors, so residuals average zero at every fitted value.
How do I test the linearity assumption?
Plot residuals against fitted values and each predictor, add a smooth mean line, and look for curvature. Optionally add Ramsey’s RESET test.
What does a curved residual plot mean?
Systematic structure means the model form can be improved, for example with a missing polynomial term, variable or interaction.
What do I do if linearity is violated?
Change the model formula, add polynomial terms, or consider a transformation, then re-check every assumption.
Is a high R² evidence of linearity?
No. In the example above R² = 0.949 with a clearly curved residual pattern.
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods, section 4.4.4.1, “How can I assess the sufficiency of the functional part of the model?” itl.nist.gov/div898/handbook/pmd/section4/pmd441.htm (checked 2026-10-29).
- NIST/SEMATECH e-Handbook, section 4.4.4, model validation overview. itl.nist.gov/div898/handbook/pmd/section4/pmd44.htm (checked 2026-10-29).
- Ramsey, J. B. (1969). Tests for specification errors in classical linear least-squares regression analysis. Journal of the Royal Statistical Society: Series B, 31, 350-371.
- R package lmtest,
resettestmanual. search.r-project.org/CRAN/refmans/lmtest/html/resettest.html (checked 2026-10-29). - statsmodels documentation,
linear_reset. statsmodels.org (checked 2026-10-29). - University of Wisconsin-Madison SSCC, Regression Diagnostics in R: linearity. sscc.wisc.edu/sscc/pubs/RegDiag-R/linearity.html (checked 2026-10-29).
