LME in R: Fitting Linear Mixed-Effects Models with nlme::lme()

Who this is for: researchers with grouped or repeated-measures data (children measured several times, samples nested in sites) who need to fit a linear mixed-effects model in R with lme() from the nlme package, run it, and read the output correctly. Everything below is taken from the nlme reference manual and the CRAN package page (read 2026-11-07).

Short answer: lme() fits a Gaussian linear mixed-effects model. You give it a two-sided fixed formula, a data frame and a one-sided random formula with a grouping factor, such as random = ~ age | Subject. By default it estimates by REML. If you leave random out, it uses the right-hand side of your fixed formula, which is the most common surprise for new users.

What it is used for in research

lme() is for continuous outcomes where observations are not independent because they share a group: repeated measures on the same participant, pupils in a class, plots in a field. Fixed effects describe the population-average relationship; random effects let intercepts and slopes vary by group. For the concepts (what a random effect is, when to treat a factor as random) see Mixed-Effects Models: Multilevel, Hierarchical, and What the Random Part Does and choosing random vs. fixed effects for nested data. For binary or count outcomes you need a generalised model: see generalised linear mixed models (GLMM), which is where glmer() from lme4 belongs. lme() itself covers Gaussian responses only.

Install or access (official source only)

The package is nlme, “Linear and Nonlinear Mixed Effects Models” on CRAN (cran.r-project.org/web/packages/nlme). The CRAN page listed version 3.1-172, published 2026-10-31, as a recommended package that depends on R 3.6.0 or later and has binaries for Windows and macOS. Then load it:

install.packages("nlme")   # only if it is not already installed
library(nlme)

Step-by-step task: fit a model with lme()

The manual’s own examples use the Orthodont data (distance measured repeatedly on each Subject, with age and Sex), so you can reproduce every step without your own data.

1. Know the signature. lme(fixed, data, random, correlation, weights, subset, method, na.action, control, contrasts = NULL, keep.data = TRUE).
2. Write the fixed part as a two-sided formula: distance ~ age.
3. Write the random part as a one-sided formula with a bar and the grouping factor: random = ~ age | Subject means a random intercept and a random age slope for each subject. Nested grouping uses ~ x1 + ... + xn | g1/.../gm.
4. Fit it:

   fm1 <- lme(distance ~ age, data = Orthodont, random = ~ age | Subject)

5. Check the default trap. In the manual, fm1 <- lme(distance ~ age, data = Orthodont) is annotated “random is ~ age”: with no random argument, the random formula defaults to the right-hand side of fixed. And random = ~ 1 gives a random intercept only:

   fm2 <- lme(distance ~ age + Sex, data = Orthodont, random = ~ 1)

6. Summarise: summary(fm1). Keep the object (s1 <- summary(fm1)) and use coef(s1) to pull out the fixed-effects table.
7. Compare models with anova(fm1, fm2) (see the output and troubleshooting sections for the REML caveat).

The lme formula arguments at a glance

Argument	What it takes	Default
fixed	Two-sided linear formula	required
random	One-sided formula ~ x1 + ... + xn | g1/.../gm	Right-hand side of fixed
correlation	A corStruct object for within-group correlation	NULL
weights	A varFunc object or one-sided formula for the variance structure	NULL
method	"REML" or "ML"	"REML"
control	List of values replacing the lmeControl defaults	the defaults

Output and how to read it

summary() on an lme fit returns an object containing a tTable, the fixed-effects correlation (corFixed), residuals, AIC and BIC.

Piece of output	What it tells you
tTable: Value, Std. Error	Estimate and standard error of each fixed effect
tTable: DF, t-value, p-value	Degrees of freedom, t statistic and p-value for each fixed effect
corFixed	Correlation among the fixed-effect estimates
AIC, BIC	Information criteria for comparing fitted models
adjustSigma (default TRUE)	When a model was fit by ML, rescales the residual standard error to a REML-like estimate

anova() with a single lme object returns Wald F-tests per term (sequential sums of squares by default; type = "marginal" deletes one term at a time). With several fitted objects it returns degrees of freedom, (restricted) log-likelihood, AIC and BIC for each, plus a likelihood-ratio statistic and p-value when test = TRUE (the default) and consecutive models differ in degrees of freedom. The manual’s example compares a random-slope model with update(fm1, random = pdDiag(~age)). For the logic of that test, see likelihood-ratio tests for nested models.

Troubleshooting

• Likelihood comparisons look wrong when the fixed effects differ. The anova manual notes that likelihood comparisons are not meaningful for objects fit by REML with different fixed effects. Refit with method = "ML" to compare models that differ in fixed terms.
• Convergence trouble or poor optimisation. The manual says lme() does not scale your data and that optimisation works best when the response is scaled so its variance is of the order of one. Rescale the response, or adjust settings through control.
• An error about offset. As of nlme 3.1-157, offset terms in fixed raise an error; earlier versions silently ignored them.
• Unexpected random structure. Check whether you omitted random; it defaults to the right-hand side of fixed.

Free alternatives (table)

These are other free R packages for mixed models. This page does not cover their syntax.

Option	What it is	Note
nlme::lme()	“Fit and compare Gaussian linear and nonlinear mixed-effects models” (CRAN)	Covered on this page; supports correlation and weights structures
lme4::lmer()	“Fitting Linear Mixed-Effects Models using lme4” (Bates, Maechler, Bolker, Walker)	Companion glmer() handles non-Gaussian responses; see the GLMM guide
glmmTMB	Another free R package for mixed models	Named only as an alternative
brms	Another free R package for mixed models, fitted in a Bayesian framework	See Bayesian hierarchical models

FAQ

What does “lme in R” mean?

It usually means the lme() function in the nlme package, which fits linear mixed-effects models. The same words are sometimes used loosely for lme4’s lmer(), a different function with a different formula syntax.

What is the lme formula in R?

Two formulas: fixed (y ~ x1 + x2) and random (~ x | group). Without random, the right-hand side of fixed is used.

Is there an lme example I can run?

Yes: lme(distance ~ age, data = Orthodont, random = ~ age | Subject), from the manual, then summary() on the result.

What is the difference between lme and lmer?

lme() is in nlme; lmer() is in lme4. This page documents only lme(); its manual describes correlation and weights arguments for within-group error structures.

How do I fit a generalised model (glmer)?

Not with lme(), which is Gaussian. See the GLMM guide.

Sources (official docs)

• nlme reference manual, lme, summary.lme and anova.lme, read 2026-11-07.
• CRAN, nlme package page, read 2026-11-07.