Weighted Least Squares: Definition & Formula

Q: What is the difference between Weighted Least Squares and Ordinary Least Squares?

Ordinary least squares (OLS) treats every observation equally under homoscedasticity. Weighted least squares (WLS) introduces observation-specific weights inversely proportional to error variance, minimizing a weighted sum of squared residuals. When heteroscedasticity is present, WLS produces more efficient parameter estimates with smaller standard errors than OLS.

Q: When should researchers use Weighted Least Squares instead of robust standard errors?

Researchers choose weighted least squares when parameter efficiency is paramount and error variance can be modeled or is known a priori (e.g., physical instrument precision or sample cell sizes). If the variance structure is unknown, robust standard errors (such as Huber-White estimators) with OLS are often preferred because misspecified weights in WLS can degrade estimation efficiency.

Q: How do you determine weights if error variances are unknown?

When variances are unknown, researchers use Feasible Generalized Least Squares (FGLS): (1) fit an initial OLS model, (2) extract squared residuals, (3) regress log squared residuals on predictor variables to estimate the variance function, and (4) use the inverse of fitted variance estimates as empirical weights in a final WLS regression.

Q: What is the difference between Weighted Least Squares (WLS) and Generalized Least Squares (GLS)?

Weighted least squares is a special case of Generalized Least Squares. WLS uses a diagonal weight matrix to account for unequal error variances among mutually independent observations. GLS uses a full error covariance matrix that accommodates both unequal variances and non-zero covariances (such as autocorrelation or clustering).

Definition: What Is Weighted Least Squares?

Weighted least squares (WLS) is a regression technique that estimates linear model parameters when observations have unequal error variances (heteroscedasticity). While standard ordinary least squares treats every observation equally under the assumption of constant error variance, weighted least squares assigns each observation a non-negative weight inversely proportional to its error variance (w_i = 1 / σ_i²). Observations measured with high precision receive larger weights and exert greater influence on parameter estimates, whereas noisy observations receive smaller weights. According to the NIST/SEMATECH e-Handbook of Statistical Methods (Section 4.1.4.3), weighted least squares accommodates the error structure directly without altering the functional relationship, restoring estimation efficiency and delivering minimum variance linear unbiased estimators.

Weighted Least Squares Formula and Objective Function

In simple linear regression, the relationship between predictor x and continuous response y is expressed as:

y_i = β₀ + β₁x_i + ε_i

where β₀ is the intercept, β₁ is the regression coefficient (slope), and ε_i is random error with E(ε_i) = 0 and Var(ε_i) = σ_i². The weighted least squares objective function minimizes the weighted sum of squared deviations Q between observed y_i and predicted value ŷ_i:

Q = Σ_i=1ⁿ w_i [y_i − (β̂₀ + β̂₁x_i)]²

The mathematical symbols represent:

n: Number of observations.
y_i, x_i: Observed response and predictor values for observation i.
β̂₀, β̂₁: Estimated weighted intercept and slope.
w_i: Weight assigned to observation i (w_i = 1 / σ_i²).
σ_i²: Error variance for observation i.

In matrix notation, let y be an n × 1 response vector, X an n × p design matrix, and W = diag(w₁, …, w_n) an n × n diagonal weighting matrix. The analytical solution is obtained via the weighted normal equations:

β̂ = (X^TWX)⁻¹ X^TWy

When error terms are correlated in addition to having unequal variances, this formulation extends to Generalized Least Squares (GLS), replacing W with the full inverse covariance matrix Ω⁻¹.

Worked Example With Step-by-Step Arithmetic

Consider a calibration experiment where measurement precision decreases as predictor magnitude increases. Table 1 summarizes the observed data, error variances σ_i², calculated weights w_i = 1 / σ_i², and cross-products used to determine parameter estimates.

i	x_i	y_i	σ_i²	w_i	w_ix_i	w_iy_i	w_ix_i²	w_ix_iy_i
1	1.0	2.2	0.25	4.00	4.00	8.80	4.00	8.800
2	2.0	3.9	0.50	2.00	4.00	7.80	8.00	15.600
3	3.0	6.1	1.00	1.00	3.00	6.10	9.00	18.300
4	4.0	7.8	2.00	0.50	2.00	3.90	8.00	15.600
5	5.0	10.5	4.00	0.25	1.25	2.625	6.25	13.125
Σ	15.0	30.5	—	7.75	14.25	29.225	35.25	71.425

The arithmetic proceeds in four steps:

1. Calculate weighted means:
x̄_w = 14.25 / 7.75 ≈ 1.8387
ȳ_w = 29.225 / 7.75 ≈ 3.7710

2. Calculate weighted sums of squares:
SS_xx,w = 35.25 − [(14.25)² / 7.75] = 9.0484
SS_xy,w = 71.425 − [(14.25 × 29.225) / 7.75] = 17.6887

3. Solve for weighted slope (β̂₁):
β̂₁ = 17.6887 / 9.0484 ≈ 1.9549

4. Solve for weighted intercept (β̂₀):
β̂₀ = 3.7710 − (1.9549 × 1.8387) = 0.1765

The fitted line is ŷ = 0.1765 + 1.9549x. For comparison, OLS on this dataset yields ŷ = −0.0500 + 2.0500x. While OLS yields a slope standard error of 0.0957, WLS achieves 0.0828—a 13.5% reduction in parameter uncertainty gained by downweighting imprecise observations at x = 4 and x = 5.

Interpretation and Practical Guidance

Interpreting a weighted least squares model centers on evaluating parameter estimates alongside precision weights. The slope β̂₁ = 1.9549 indicates that for each 1-unit increase in the predictor, the response increases by 1.95 units on average, anchored primarily by high-precision measurements. The intercept β̂₀ = 0.1765 represents expected response at x = 0.

Table 2 summarizes key statistical metrics from the weighted regression analysis.

Metric	Worked Value	Substantive Meaning	Reading Guidance
Weighted Slope (β̂₁)	1.9549	Change in y per unit change in x.	Evaluate sign and 95% confidence intervals against domain theory.
Standard Error (SE)	0.0828	Sampling precision of slope estimate.	Smaller than OLS SE (0.0957), confirming efficiency gain.
t-statistic / p-value	t = 23.60, p < .001	Test of null hypothesis β₁ = 0.	Statistically significant at α = .05.
Weighted R²	0.9946	Proportion of weighted variation explained.	Applies strictly to transformed data; do not compare to OLS R².

Assumptions and Common Mistakes

Weighted least squares requires satisfying classical assumptions of simple regression, with the exception of constant error variance:

Correct specification: The linear model accurately describes the population mean.
Independent errors: Residuals must be uncorrelated across observations. When autocorrelation or clustering occurs, Generalized Least Squares must be used.
Accurate variance weighting: Weights must reflect true relative variances.
Absence of high-weight outliers: Observations with small variances exert substantial leverage.

Literature by Carroll and Ruppert (1988) and NIST guidance (Section 4.1.4.3) highlights common pitfalls:

Estimating weights from small samples: When weights are estimated from small replicate samples (e.g., 2–3 replicates per point), sample variance noise introduces severe instability into regression coefficients.
Ignoring high-weight outliers: In WLS, an outlier that receives a large weight dominates parameter estimates. Researchers should examine standardized residuals to detect anomalous influential points.
Comparing unweighted and weighted R-squared: Comparing weighted R² against unweighted OLS R² is mathematically invalid because total sums of squares differ.
Omitting residual diagnostic checks: After fitting WLS, researchers should calculate residuals, multiply each residual by √w_i, and inspect plots to verify heteroscedasticity was resolved.

How to Report Weighted Least Squares in APA Format

Under APA 7th edition guidelines, reporting weighted regression requires identifying the weighting rationale, parameter estimates with standard errors, test statistics, degrees of freedom, p-values, confidence intervals, and the precision comparison relative to OLS.

Example APA 7th Narrative:

“A weighted least squares (WLS) linear regression was conducted to examine the effect of calibration load on response output across five measurement levels. Observations were weighted by the reciprocal of known measurement error variance (w_i = 1 / σ_i²) to correct for heteroscedasticity. The overall weighted regression model was statistically significant, F(1, 3) = 556.91, p < .001, accounting for 99.46% of the weighted variance (R² = .995). Calibration load significantly predicted response output, b = 1.95, SE = 0.08, t(3) = 23.60, p < .001, 95% CI [1.69, 2.22]. The intercept estimate was b = 0.18, SE = 0.18, t(3) = 1.00, p = .391, 95% CI [−0.39, 0.74]. Compared to an unweighted ordinary least squares model (SE = 0.10), weighting by measurement precision reduced the slope standard error by 13.5%, confirming greater parameter estimation efficiency.”

How to Calculate Weighted Least Squares in Excel, R, and Python

Weighted least squares can be implemented in standard statistical environments using dedicated parameters or matrix operations.

1. Python (statsmodels)

In Python, the statsmodels.regression.linear_model.WLS class performs weighted least squares directly:

import numpy as np
import statsmodels.api as sm

x = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([2.2, 3.9, 6.1, 7.8, 10.5])
variances = np.array([0.25, 0.50, 1.00, 2.00, 4.00])
weights = 1.0 / variances

X = sm.add_constant(x)
model = sm.WLS(y, X, weights=weights)
results = model.fit()

print(results.summary())
print(f"Slope: {results.params[1]:.4f}, SE: {results.bse[1]:.4f}")
print(f"Intercept: {results.params[0]:.4f}, SE: {results.bse[0]:.4f}")

2. R (stats::lm)

In R, base lm() accepts inverse-variance weights via the weights argument:

x <- c(1.0, 2.0, 3.0, 4.0, 5.0)
y <- c(2.2, 3.9, 6.1, 7.8, 10.5)
variances <- c(0.25, 0.50, 1.00, 2.00, 4.00)
w <- 1 / variances

wls_model <- lm(y ~ x, weights = w)
summary(wls_model)
confint(wls_model)

3. Microsoft Excel

In Excel, enter x in A2:A6, y in B2:B6, variances in C2:C6, weights in D2:D6 (=1/C2), and sum in D7 (=SUM(D2:D6)):

Weighted means: x̄_w = =SUMPRODUCT(D2:D6, A2:A6)/D7, ȳ_w = =SUMPRODUCT(D2:D6, B2:B6)/D7.
Weighted slope: =(SUMPRODUCT(D2:D6,A2:A6,B2:B6) - SUMPRODUCT(D2:D6,A2:A6)*SUMPRODUCT(D2:D6,B2:B6)/D7) / (SUMPRODUCT(D2:D6,A2:A6,A2:A6) - (SUMPRODUCT(D2:D6,A2:A6)^2)/D7).
Weighted intercept: =ȳ_w - Slope * x̄_w.

Frequently Asked Questions

What is the difference between Weighted Least Squares and Ordinary Least Squares?