Skip to main content
v2026.11,986 entries · CC-BY 4.0

Permutation Test in R: coin, perm and lmPerm Syntax

Run a permutation test in R with coin, perm or lmPerm: syntax, a worked two-group example, reading the output and why packages disagree.

Written and maintained by CASRAI Editorial Board

Published

On this page

A permutation test in R compares groups by shuffling the group labels many times and asking how often a shuffled data set produces a test statistic at least as extreme as the one you observed. R has no single built-in function for it, so you either write a short loop around sample() or use a CRAN package: coin (oneway_test(), independence_test()), perm (permTS()) or lmPerm (lmp(), aovp()). This guide gives the syntax of each route, a worked two-group example, how to read what comes back, and the settings that make two packages disagree.

What a permutation test in R is used for in research

Researchers use a permutation test when they want a p-value that rests on the study’s own randomisation or group assignment rather than on a distributional assumption such as normality. The null hypothesis is that the outcome has the same distribution in every group, so group labels are interchangeable; the test then rebuilds the null distribution of the statistic by reassigning labels. The conceptual side (exchangeability, how many permutations are enough, a hand-worked example) is covered in the permutation tests guide; this page is the R implementation.

The coin package describes itself as providing “Conditional Inference Procedures in a Permutation Test Framework”. Its two-sample and k-sample location tests include oneway_test(), which the manual says provides the Fisher-Pitman permutation test, plus wilcox_test() (Wilcoxon-Mann-Whitney), kruskal_test(), normal_test() (van der Waerden), median_test() (Brown-Mood) and savage_test(). Each tests equality of the distribution of a numeric response across groups against shift alternatives. If you only need the rank-based version, see the Wilcoxon test in R guide; a permutation test is the choice when you want the statistic itself (for example a difference in means) to be the thing whose null distribution is built from the data.

Three things distinguish the routes. coin covers many test families through one formula interface and can compute exact, Monte Carlo or asymptotic p-values. perm is focused on two-sample and k-sample problems and offers several exact algorithms. lmPerm carries the idea into linear models and ANOVA, so you can permute within a regression or factorial design. The data table below compares the three R routes and the Python equivalent on the settings that matter for a result.

How to install or access coin, perm and lmPerm in R

All three packages are distributed on CRAN, R’s package repository, so installation is the standard route described in the R documentation for install.packages(), whose first argument pkgs is a character vector of package names and whose repos argument is the base URL of a repository such as the CRAN mirror https://cloud.r-project.org.

  1. Start R or RStudio.
  2. Run install.packages(c("coin", "perm", "lmPerm"), repos = "https://cloud.r-project.org"). You can install only the package you need.
  3. Load the package for the session with library(coin), library(perm) or library(lmPerm).

The versions recorded on the CRAN reference manuals when this guide was checked were coin 1.4-6 (published 2026-09-30, requires R 3.6.0 or later, licence GPL-2, maintained by Torsten Hothorn), perm 1.0-0.4 (Michael Fay; described in Fay and Shaw 2010, Journal of Statistical Software, doi:10.18637/jss.v036.i02) and lmPerm 2.1.6 (published 2026-05-08, “Permutation Tests for Linear Models”). The coin methodology is described in Hothorn and colleagues, Journal of Statistical Software, doi:10.18637/jss.v028.i08. Cite the package version you ran in the methods section of a paper; citation("coin") prints the recommended reference.

How to run a permutation test in R step by step

The worked example uses the tritiated-water diffusion data from the coin manual (Hollander and Wolfe, 1999, Table 4.1): 15 measurements of diffusion, ten from the “At term” group and five from the “12-26 Weeks” group. The question is whether the two groups differ in location.

Step 1: put the data in a data frame with a numeric response and a factor

Every route needs one numeric outcome and one grouping factor. This is the construction used in the coin manual:

diffusion = data.frame(
  pd  = c(0.80, 0.83, 1.89, 1.04, 1.45, 1.38, 1.91, 1.64, 0.73, 1.46,
          1.15, 0.88, 0.90, 0.74, 1.21),
  age = factor(rep(c("At term", "12-26 Weeks"), c(10, 5)))
)

Step 2: run the test with coin and ask for the permutation distribution explicitly

The syntax is oneway_test(formula, data, subset = NULL, weights = NULL, ...), with a formula of the form y ~ x | block, where y is numeric, x is a factor and the optional block is a stratification factor. The manual states that an asymptotic approximation of the exact distribution is used by default (distribution = "asymptotic"), and that Monte Carlo resampling or an exact computation for univariate two-sample problems is requested by setting distribution to "approximate" or "exact". So the first call below is not yet a permutation-distribution test; the second and third are:

library(coin)
oneway_test(pd ~ age, data = diffusion)                              # asymptotic (default)
set.seed(1)
pvalue(oneway_test(pd ~ age, data = diffusion,
                   distribution = approximate(nresample = 10000)))   # Monte Carlo
ot = oneway_test(pd ~ age, data = diffusion, distribution = "exact")
pvalue(ot)                                                           # exact

These three calls come from the coin manual’s own example, apart from the set.seed(1) line and the assignment style. approximate() takes nresample (default 10000), parallel ("no", "multicore" or "snow"), ncpus and cl; exact() takes algorithm ("auto", "shift" or "split-up"); asymptotic() takes maxpts, abseps and releps. For a one-sided hypothesis add alternative = "less" or "greater".

If you need a statistic other than the one oneway_test() uses, note that since coin 1.1-0 oneway_test() no longer lets you specify the statistic; the manual directs you to independence_test() for more control.

Step 3: the same comparison with perm

The syntax is permTS(x, y, alternative = c("two.sided", "less", "greater"), exact = NULL, method = NULL, methodRule = methodRuleTS1, control = permControl(), ...), with a formula method permTS(formula, data, subset, na.action, ...) whose right-hand side is a two-level factor. The method argument takes "pclt", "exact.network", "exact.ce" or "exact.mc" (Monte Carlo). For permTS the statistic is equivalent to the difference of the group means.

library(perm)
permTS(pd ~ age, data = diffusion, method = "exact.mc",
       control = permControl(nmc = 10000 - 1))

Step 4: permute inside a linear model with lmPerm

When the question is a regression coefficient or an ANOVA effect rather than a two-group difference, lmPerm provides lmp(formula, data, perm = "Exact", seqs = FALSE, center = TRUE, subset, weights, na.action, ...) and aovp(formula, data = NULL, perm = "Exact", seqs = FALSE, ...). The perm argument takes "Exact", "Prob" or "SPR" and all three give permutation probabilities; seqs = TRUE uses sequential sums of squares and FALSE uses unique sums of squares.

library(lmPerm)
summary(lmp(pd ~ age, data = diffusion, perm = "Prob"))

For the one-way ANOVA case beyond two groups, compare with the running ANOVA in R guide; the permutation version changes how the p-value is computed, not the model formula.

Step 5: write the loop yourself in base R when you need full control

The R documentation for sample() gives the usage sample(x, size, replace = FALSE, prob = NULL) and states that sample(x) generates a random permutation of the elements of x. That is all a hand-rolled permutation test needs: shuffle the labels, recompute the statistic, repeat.

set.seed(2026)
means = function(y, g) tapply(y, g, mean)
obs   = diff(means(diffusion$pd, diffusion$age))
B     = 9999
perm_stat = replicate(B, diff(means(diffusion$pd, sample(diffusion$age))))
p_two = (sum(abs(perm_stat) >= abs(obs) - 1e-12) + 1) / (B + 1)
p_two

Here diff() subtracts the first factor level’s mean from the second, which is “12-26 Weeks” and “At term” because R orders levels alphabetically; the sign does not matter for the two-sided count. The + 1 in numerator and denominator counts the observed arrangement as one member of the null distribution, the same convention SciPy’s documentation describes for randomised tests, and it keeps a Monte Carlo p-value from being exactly zero. The small tolerance 1e-12 stops floating-point rounding from excluding permutations that tie with the observed value. For two samples this loop and a package should agree up to Monte Carlo error; use the bootstrap in R guide instead if your goal is a confidence interval rather than a p-value.

How to read the output

Which object comes back depends on the route, and so does what a “p-value” means.

  • coin. oneway_test() returns an object you can query with pvalue() for the p-value and statistic() for the test statistic; the manual also documents expectation() and covariance() for the conditional moments. For an exact two-sample test, support(ot) lists the possible values of the statistic, and dperm(ot, s) and pperm(ot, s) give the null density and distribution function at s. Confidence intervals are not available for oneway_test(); they are for tests such as wilcox_test() when you pass conf.int = TRUE and then call confint().
  • perm. permTS() returns an htest object (an mchtest object when method = "exact.mc") with p.value, a p.values vector (lower, upper, two-sided and p.equal), an estimate, and for Monte Carlo runs a confidence interval for the p-value itself (p.conf.int) and the number of resamples (nmc).
  • lmPerm. lmp() and aovp() report permutation probabilities. With perm = "Prob" the algorithm stops when the estimated standard error of the p-value falls below Ca times the p-value, with maxIter defaulting to 1000 iterations.
  • Your own loop. The result is a proportion. Report it with the number of permutations, and remember that a Monte Carlo p-value carries its own sampling error: a finer p-value needs more permutations.

When reporting, name the statistic (difference of means, Wilcoxon rank sum), whether the p-value is exact or Monte Carlo and how many resamples, the seed, the alternative (one- or two-sided) and the package version. For wording on the p-value itself, see the p-value dictionary entry.

Troubleshooting a permutation test in R

The p-value is not a permutation p-value

In coin the default is distribution = "asymptotic". If you call oneway_test(pd ~ age, data = diffusion) and stop, you have a large-sample approximation to the conditional distribution, not a resampled or exact one. Add distribution = approximate(nresample = 10000) or distribution = "exact".

Two packages give slightly different p-values

Two causes are documented. First, Monte Carlo variation: coin uses 10000 resamples by default and perm‘s permControl() uses nmc = 10^3 - 1 (999), so equal settings and seeds are needed before comparing. Second, the two-sided convention: perm‘s default tsmethod = "central" computes the two-sided p-value as min(1, 2 * min(pless, pgreater)), and the perm manual states that tsmethod = "abs" matches the default two-sided method of coin. If the two disagree on a skewed small sample, set tsmethod = "abs" in permControl() before suspecting an error.

The result changes every time you run it

Monte Carlo resampling draws random permutations. For coin and a hand-written loop, call set.seed() before the test. perm resets the seed to 1234321 each time by default (seed = 1234321, setSEED = TRUE), so it repeats unless you set setSEED = FALSE. Seed behaviour of sample() also changed in R 3.6.0, which the sample() documentation notes; an old script can give a different shuffle under a newer R even with the same seed, and RNGkind(sample.kind = ...) controls it.

Out-of-memory errors with exact methods

The perm manual warns that exact.network and exact.ce “may give errors related to running out of memory” at roughly 15 observations per group for exact.network or 14 per group for exact.ce. Use method = "exact.mc" or "pclt" beyond that. SciPy’s documentation makes the same point for Python: exact tests are feasible only for very small data sets because the number of distinct permutations grows very rapidly with sample size. In lmPerm, maxExact caps the number of observations for perm = "Exact", beyond which "Prob" is used.

Choosing between a permutation test and a rank test

A permutation test of the mean difference and the Wilcoxon-Mann-Whitney test answer related but different questions: one permutes the raw values and uses their mean difference, the other permutes ranks. Both are available through coin, so the choice is a statistical decision about the estimand, made before looking at the data.

Free alternatives to R for permutation tests

R and its packages are free and open. If your analysis pipeline is in Python, SciPy provides scipy.stats.permutation_test; the SciPy stats guide describes the library more broadly. The table lists the entry points of each tool as given in the official documentation; the data table below adds the default settings.

Tool Language Entry function What it permutes
coin R oneway_test(), independence_test(), wilcox_test() Group labels, for location and independence tests
perm R permTS(), permKS(), permTREND() Two-sample, k-sample and trend problems
lmPerm R lmp(), aovp() Linear-model and ANOVA terms
SciPy Python scipy.stats.permutation_test(data, statistic, ...) Any statistic you supply as a function, with permutation_type of "independent", "samples" or "pairings"

The SciPy function’s n_resamples defaults to 9999; when it is greater than or equal to the number of distinct permutations, the exact null distribution is computed, and otherwise the test is randomised. The result object has statistic, pvalue and null_distribution attributes. Its default two-sided p-value is twice the smaller one-sided p-value, the same form as perm‘s default.

Sources

Data: Settings that decide a permutation test result: coin, perm, lmPerm and SciPy

Compares four free permutation-test routes (three R packages and SciPy for Python) on the entry function, how an exact test is requested, the default resampling setting and the default p-value convention. Compiled from the CRAN reference manuals for coin 1.4-6, perm 1.0-0.4 and lmPerm 2.1.6 and the SciPy documentation, read 2026-11-17.

Period covered: 2026.

Settings that decide a permutation test result: coin, perm, lmPerm and SciPy
RouteEntry functionHow exact or Monte Carlo is requestedDefault resampling or stopping settingDefault p-value basis
coin (R)oneway_test(formula, data, ...)distribution = "exact" or approximate(nresample = ...); also "asymptotic"nresample = 10000 for distribution = approximate()Asymptotic approximation of the conditional distribution (distribution = "asymptotic")
perm (R)permTS(x, y, ...) or permTS(formula, data)method = "exact.network", "exact.ce", "exact.mc" or "pclt"nmc = 10^3 - 1 (999) in permControl(); seed 1234321 reset by defaultTwo-sided p-value tsmethod = "central": min(1, 2 * min(pless, pgreater))
lmPerm (R)lmp(formula, data, perm = "Exact") and aovp()perm = "Exact", "Prob" or "SPR"; maxExact caps observations for Exactperm = "Prob" stops when estimated SE of p is below Ca * p; maxIter = 1000Permutation probabilities, as printed by the model summary
SciPy (Python)scipy.stats.permutation_test(data, statistic, ...)Exact when n_resamples is at least the number of distinct permutations; otherwise randomisedn_resamples = 9999Two-sided: twice the smaller one-sided p-value; randomised tests add one to numerator and denominator

Sources

Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about Permutation Test in R: coin, perm and lmPerm Syntax

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →