Written and maintained by CASRAI Editorial Board
Published
On this page
Robust regression is a family of regression methods that give less influence to unusually large residuals than ordinary least squares (OLS) does, so that a few outlying observations cannot pull the fitted line away from the bulk of the data. The most common version is M-estimation with the Huber loss, which treats small residuals like OLS and large ones with a linear penalty, so outliers are downweighted rather than deleted.
What is robust regression?
OLS minimises the sum of squared residuals, so one observation that sits far from the pattern contributes a very large squared term and drags the line toward itself. Robust regression replaces the squared loss with a loss that grows more slowly for large residuals. The result is a fit that follows the majority of the points and reports, through its weights, which observations it treated as unusual. Two ideas from the scikit-learn user guide help to keep the terms straight: an outlier can be in the response (y) or in the predictors (X), and what matters is both how many points are outlying and how far out they lie. The breakdown point is the fraction of the data that can be outlying before the fit starts to miss the inlying data.
This page uses a hand-computed example first and software second. It is written for researchers and analysts who already know simple linear regression and want to know what a robust fit does, which tuning constants the main software packages use by default, and how to report the result. If your concern is the standard errors rather than the coefficients, see robust standard errors (HC0 to HC3) instead: that adjusts the uncertainty of an OLS fit and does not change the fitted line.
Robust regression versus related methods
Robust regression is a different tool from three methods that appear near it in the literature. Weighted least squares also uses weights, but they are fixed in advance from known variances, whereas a robust fit computes its weights from the residuals themselves. Cook’s distance flags influential observations for you to examine; robust regression instead limits their influence automatically. LOESS regression is a local smoother for curved relationships. The general idea that conclusions should survive violations of assumptions is covered by the dictionary entry on robustness.
Huber M-estimation: the formula and every symbol
An M-estimator chooses the coefficients that minimise a sum of a loss function ρ applied to scaled residuals:
minimise over β: Σ ρ(rᵢ / s), where rᵢ = yᵢ − (β₀ + β₁xᵢ)
- yᵢ is the observed response and xᵢ the predictor for observation i.
- β₀, β₁ are the intercept and slope being estimated.
- rᵢ is the residual of observation i.
- s is a scale estimate of the residuals. Dividing by s makes the method invariant to the units of y. The statsmodels documentation lists several scale estimators (MAD, Huber’s proposal 2, an interquartile range and Qn), and R’s MASS::rlm uses a rescaled median absolute deviation (MAD) of the residuals by default.
- ρ is the loss. For Huber’s loss with tuning constant k, ρ(u) = u²/2 when |u| ≤ k and ρ(u) = k|u| − k²/2 when |u| is above k.
Small residuals are penalised quadratically, exactly as in OLS. Beyond k the penalty is linear, so a residual that is ten times larger adds ten times, not one hundred times, as much. scikit-learn’s HuberRegressor writes the same shape as H(z) = z² for |z| below ε and 2ε|z| − ε² otherwise, which is the form above multiplied by two, and its parameter ε plays the role of k. Its documentation says outliers are linearly penalised rather than discarded, and that setting ε to 1.35 achieves 95% statistical efficiency, meaning the estimator is about as precise as OLS when the errors really are normal.
How it is fitted: iteratively reweighted least squares
MASS::rlm documents fitting by iterated re-weighted least squares. Each pass converts the Huber loss into a weight w = min(1, k / |u|) with u = r / s: a point with |u| ≤ k keeps weight 1, and a point beyond k gets a weight that shrinks in proportion to its distance. The next pass refits the line by weighted least squares with those weights, recomputes the residuals and the scale, and repeats until the coefficients stop changing. The Huber loss has a unique solution, so the starting values do not decide the answer; for Hampel and bisquare (Tukey biweight) losses the MASS documentation warns that several local minima can exist, so the starting point matters (MASS starts from a least-squares fit by default).
Worked example with the arithmetic
This example is an illustrative dataset made up for this page and computed with plain arithmetic; it is not from a study. Eight observations lie close to y ≈ 2x, except that the eighth has been recorded as 30.0 instead of about 16.
The data are x = 1, 2, 3, 4, 5, 6, 7, 8 and y = 2.1, 3.9, 6.2, 7.8, 10.1, 12.2, 13.9, 30.0.
Step 1: the OLS fit
The means are x̄ = 36/8 = 4.5 and ȳ = 86.2/8 = 10.775. The sums are Σ(x − x̄)(y − ȳ) = 132.8 and Σ(x − x̄)² = 42. The OLS slope is therefore 132.8 / 42 = 3.162 and the intercept is 10.775 − 3.162 × 4.5 = −3.454. The line y = −3.454 + 3.162x is steeper than the pattern in the first seven points, because the last point has pulled it up. Fitted to those seven points alone, OLS gives y = 0.043 + 1.996x.
Step 2: scale and Huber weights from the OLS residuals
The eight OLS residuals are shown below. Their absolute values in order are 0.168, 1.030, 1.394, 2.256, 2.392, 3.318, 4.780 and 8.158, so the median is (2.256 + 2.392) / 2 = 2.324. In this example the scale is the median absolute residual divided by 0.6745, which gives s = 2.324 / 0.6745 = 3.445. With k = 1.345, the cut-off is k × s = 4.634, so only residuals larger than 4.634 in absolute value are downweighted.
| x | y | OLS residual r | u = r / s | Huber weight (pass 1) |
|---|---|---|---|---|
| 1 | 2.1 | 2.392 | 0.694 | 1.000 |
| 2 | 3.9 | 1.030 | 0.299 | 1.000 |
| 3 | 6.2 | 0.168 | 0.049 | 1.000 |
| 4 | 7.8 | -1.394 | -0.405 | 1.000 |
| 5 | 10.1 | -2.256 | -0.655 | 1.000 |
| 6 | 12.2 | -3.318 | -0.963 | 1.000 |
| 7 | 13.9 | -4.780 | -1.387 | 0.969 |
| 8 | 30.0 | 8.158 | 2.368 | 0.568 |
Observation 7 (residual −4.780) gets weight 4.634 / 4.780 = 0.969, and observation 8 (residual 8.158) gets 4.634 / 8.158 = 0.568. Refitting by weighted least squares with these weights gives y = −2.398 + 2.811x after one pass, already lower than the OLS slope of 3.162.
Step 3: iterate to convergence
Repeating the cycle (recompute residuals, recompute s, recompute weights, refit) moves the slope to 2.534 after pass 2 and 2.337 after pass 3. After 18 passes the coefficients stop changing at four decimals: y = −0.094 + 2.042x, with s = 0.237. Observation 8 now has a residual of 13.758, so its weight is 1.345 × 0.2369 / 13.758 = 0.023: the method has all but ignored it, while the other seven points keep weight 1 and observation 7 stays at 1 (its residual of −0.300 is well inside the cut-off). The robust slope of 2.042 is close to the 1.996 obtained by deleting the outlier by hand, and far from the OLS slope of 3.162.
Note that this example needed 18 passes. MASS::rlm documents a default of 20 iterations (maxit = 20) and a convergence tolerance of 1e-4, so a badly placed outlier can bring a fit close to the iteration limit, and the software’s warning about non-convergence should not be ignored.
Interpretation: reading a robust fit
A robust regression output is read like an OLS output with extra diagnostics. The table gives reading guidance based on the definitions above.
| What you see | How to read it |
|---|---|
| Weight of 1 for an observation | Its scaled residual is within k; it is treated exactly as OLS would treat it. |
| Weight between 0 and 1 | The observation was downweighted in proportion to k divided by its scaled residual. |
| Weight close to 0 (0.023 in the example) | The observation is effectively ignored; inspect it for a recording or data-entry error before deciding what it means. |
| Robust and OLS slopes close together | Few influential points; the OLS result is not being driven by outliers. |
| Robust and OLS slopes far apart (3.162 against 2.042 in the example) | A small number of points are driving the OLS result. Examine them with residual plots and influence measures before reporting either fit. |
| Many observations with low weights | The model form may be wrong (curvature, missing predictor, a subgroup), not the data. |
The residual plot is still the first diagnostic. See how to read a residual plot for the patterns that a robust fit does not cure.
Assumptions and common mistakes
Robust regression relaxes the requirement that the errors be free of extreme values, but it keeps the others: the relationship is linear in the parameters, the observations are independent, and the model includes the right predictors. It is not a substitute for checking the data.
- Using it to hide data errors. A weight near zero tells you where to look; it does not tell you the value is wrong. Trace the observation to its source.
- Treating all outliers alike. scikit-learn’s user guide separates outliers in X from outliers in y, and notes that robust fitting in high-dimensional settings is very hard. Check which kind you have.
- Not stating the estimator. “Robust regression” names a family. The loss (Huber, bisquare, Hampel), the tuning constant, the scale estimator and the software all change the numbers.
- Ignoring the breakdown point. Different estimators tolerate different fractions of contamination. scikit-learn gives a breakdown point of about 29.3% for the Theil-Sen estimator in simple linear regression, and notes its robustness decreases as the number of predictors grows.
- Using case weights with MM-estimation in R. MASS::rlm does not support case weights with method = “MM”.
- Confusing it with robust standard errors. They answer a different question (see above).
How to report robust regression results (APA example)
State the method, the loss and tuning constant, the scale estimator, the software and version, and the comparison with OLS when it matters. An illustrative sentence for the worked example: “Because one observation had a large residual, the association between x and y was estimated with Huber M-estimation (k = 1.345, median absolute deviation scale) in R (MASS::rlm). The slope was b = 2.04, compared with b = 3.16 from ordinary least squares.” Add the standard error and confidence interval from your software output, which the hand calculation above does not produce, and report how many observations received weights below 1. APA 7th reporting conventions apply to the numbers (italicised statistic symbols, a consistent number of decimals); check the manual for the format of your journal.
How to calculate robust regression in R and Python
In R, the MASS package’s rlm function takes a formula and a psi function. The default method is “M” and psi.huber uses k = 1.345; setting method = “MM” uses Tukey’s biweight initialised by an S-estimator.
library(MASS)
d = data.frame(x = 1:8, y = c(2.1, 3.9, 6.2, 7.8, 10.1, 12.2, 13.9, 30.0))
fit = rlm(y ~ x, data = d, psi = psi.huber)
summary(fit)
fit$w # final weights
In Python, statsmodels provides RLM, with the Huber norm HuberT (default t = 1.345):
import statsmodels.api as sm
x = [1, 2, 3, 4, 5, 6, 7, 8]
y = [2.1, 3.9, 6.2, 7.8, 10.1, 12.2, 13.9, 30.0]
X = sm.add_constant(x)
fit = sm.RLM(y, X, M=sm.robust.norms.HuberT()).fit()
print(fit.params)
print(fit.weights)
scikit-learn has HuberRegressor, whose epsilon parameter is the tuning constant; the user guide advises 1.35 for 95% efficiency:
from sklearn.linear_model import HuberRegressor
import numpy as np
X = np.arange(1, 9).reshape(-1, 1)
y = np.array([2.1, 3.9, 6.2, 7.8, 10.1, 12.2, 13.9, 30.0])
model = HuberRegressor(epsilon=1.35).fit(X, y)
print(model.intercept_, model.coef_)
Your output can differ from the hand calculation in the later decimals, because each package makes its own choice of scale estimator, scale convention and stopping rule; the slope should nonetheless be near 2 and not near 3. The data table below compares the tuning constants and breakdown figures that the three packages document. For the alternatives in scikit-learn, its guide describes RANSAC (random subsets, a residual threshold, non-deterministic) and Theil-Sen, and states “when in doubt, use RANSAC”.
Frequently asked questions
When should I use robust regression instead of OLS?
Use it when a few observations have residuals much larger than the rest and you cannot justify removing them, or as a sensitivity check next to OLS. If the robust and OLS slopes agree, the outliers were not driving the result; if they differ, investigate the observations.
Which tuning constant should I use for Huber regression?
The documented defaults are k = 1.345 in R’s psi.huber and statsmodels’ HuberT, and epsilon = 1.35 in scikit-learn, which its user guide advises for 95% statistical efficiency. Smaller values downweight more observations and protect more against outliers; larger values behave more like OLS.
Does robust regression delete outliers?
No. Observations stay in the data and receive weights between 0 and 1. A weight near 0 amounts to near-exclusion, which is why the weights are worth reporting.
Is robust regression the same as robust standard errors?
No. Robust regression changes the coefficient estimates by downweighting large residuals. Heteroscedasticity-robust standard errors keep the OLS coefficients and change only the standard errors.
Sources
- statsmodels, Robust Linear Models (v0.15.0), https://www.statsmodels.org/stable/rlm.html, read 2026-11-15; HuberT norm, https://www.statsmodels.org/stable/generated/statsmodels.robust.norms.HuberT.html, read 2026-11-15.
- R MASS::rlm (version 7.3-65), https://stat.ethz.ch/R-manual/R-devel/library/MASS/html/rlm.html, read 2026-11-15.
- scikit-learn User Guide, 1.1.14 Robustness regression, https://scikit-learn.org/stable/modules/linear_model.html, read 2026-11-15.
- Huber, P. J. (1973). Robust regression: Asymptotics, conjectures and Monte Carlo. Annals of Statistics, 1(5), 799-821; Huber, P. J. (1981). Robust Statistics. Wiley (both as cited by the statsmodels documentation).
Data: Documented default tuning constants and breakdown figures for robust regression in R, Python statsmodels and scikit-learn
A cross-software comparison of the tuning constants, scale estimators and breakdown figures stated in the official documentation of R MASS::rlm (7.3-65), Python statsmodels RLM (0.15.0) and scikit-learn 1.1.14 for Huber, bisquare, Hampel, MM, Theil-Sen estimators. Compiled by reading each manual page; no single page lists all rows together.
| Estimator or setting | Software | Parameter | Documented value | What the documentation says |
|---|---|---|---|---|
| Huber M-estimator | R MASS::rlm | psi.huber k | 1.345 | Default for the Huber psi; the Huber estimator has a unique solution |
| Tukey bisquare M-estimator | R MASS::rlm | psi.bisquare c | 4.685 | Can have multiple local minima, so the starting point matters |
| Hampel M-estimator | R MASS::rlm | psi.hampel a, b, c | 2, 4, 8 | Can have multiple local minima, so the starting point matters |
| MM-estimator | R MASS::rlm | k0 of the initial S-estimator | 1.548 | Tukey's biweight initialised by an S-estimator; case weights not supported |
| Scale estimate | R MASS::rlm | scale.est | MAD (default) | Rescaled median absolute deviation of residuals; Huber proposal 2 is the alternative |
| Iteration control | R MASS::rlm | maxit, acc | 20, 1e-4 | Defaults for the maximum iterations and convergence tolerance |
| Huber M-estimator | Python statsmodels | HuberT t | 1.345 | Default for the HuberT norm used by RLM |
| Huber regressor | Python scikit-learn | epsilon | 1.35 | Advised for 95% statistical efficiency; outliers are downweighted, not discarded |
| Theil-Sen estimator | Python scikit-learn | Breakdown point (simple linear regression) | about 29.3% | Robustness decreases as the number of predictors grows |
Sources
- R MASS::rlm manual page (7.3-65) (accessed 2026-11-15)
- statsmodels HuberT norm (accessed 2026-11-15)
- statsmodels Robust Linear Models (0.15.0) (accessed 2026-11-15)
- scikit-learn User Guide 1.1.14 Robustness regression (accessed 2026-11-15)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







