Written and maintained by CASRAI Editorial Board
Published
On this page
A permutation test in R compares groups by shuffling the group labels many times and asking how often a shuffled data set produces a test statistic at least as extreme as the one you observed. R has no single built-in function for it, so you either write a short loop around sample() or use a CRAN package: coin (oneway_test(), independence_test()), perm (permTS()) or lmPerm (lmp(), aovp()). This guide gives the syntax of each route, a worked two-group example, how to read what comes back, and the settings that make two packages disagree.
What a permutation test in R is used for in research
Researchers use a permutation test when they want a p-value that rests on the study’s own randomisation or group assignment rather than on a distributional assumption such as normality. The null hypothesis is that the outcome has the same distribution in every group, so group labels are interchangeable; the test then rebuilds the null distribution of the statistic by reassigning labels. The conceptual side (exchangeability, how many permutations are enough, a hand-worked example) is covered in the permutation tests guide; this page is the R implementation.
The coin package describes itself as providing “Conditional Inference Procedures in a Permutation Test Framework”. Its two-sample and k-sample location tests include oneway_test(), which the manual says provides the Fisher-Pitman permutation test, plus wilcox_test() (Wilcoxon-Mann-Whitney), kruskal_test(), normal_test() (van der Waerden), median_test() (Brown-Mood) and savage_test(). Each tests equality of the distribution of a numeric response across groups against shift alternatives. If you only need the rank-based version, see the Wilcoxon test in R guide; a permutation test is the choice when you want the statistic itself (for example a difference in means) to be the thing whose null distribution is built from the data.
Three things distinguish the routes. coin covers many test families through one formula interface and can compute exact, Monte Carlo or asymptotic p-values. perm is focused on two-sample and k-sample problems and offers several exact algorithms. lmPerm carries the idea into linear models and ANOVA, so you can permute within a regression or factorial design. The data table below compares the three R routes and the Python equivalent on the settings that matter for a result.
How to install or access coin, perm and lmPerm in R
All three packages are distributed on CRAN, R’s package repository, so installation is the standard route described in the R documentation for install.packages(), whose first argument pkgs is a character vector of package names and whose repos argument is the base URL of a repository such as the CRAN mirror https://cloud.r-project.org.
- Start R or RStudio.
- Run
install.packages(c("coin", "perm", "lmPerm"), repos = "https://cloud.r-project.org"). You can install only the package you need. - Load the package for the session with
library(coin),library(perm)orlibrary(lmPerm).
The versions recorded on the CRAN reference manuals when this guide was checked were coin 1.4-6 (published 2026-09-30, requires R 3.6.0 or later, licence GPL-2, maintained by Torsten Hothorn), perm 1.0-0.4 (Michael Fay; described in Fay and Shaw 2010, Journal of Statistical Software, doi:10.18637/jss.v036.i02) and lmPerm 2.1.6 (published 2026-05-08, “Permutation Tests for Linear Models”). The coin methodology is described in Hothorn and colleagues, Journal of Statistical Software, doi:10.18637/jss.v028.i08. Cite the package version you ran in the methods section of a paper; citation("coin") prints the recommended reference.
How to run a permutation test in R step by step
The worked example uses the tritiated-water diffusion data from the coin manual (Hollander and Wolfe, 1999, Table 4.1): 15 measurements of diffusion, ten from the “At term” group and five from the “12-26 Weeks” group. The question is whether the two groups differ in location.
Step 1: put the data in a data frame with a numeric response and a factor
Every route needs one numeric outcome and one grouping factor. This is the construction used in the coin manual:
diffusion = data.frame(
pd = c(0.80, 0.83, 1.89, 1.04, 1.45, 1.38, 1.91, 1.64, 0.73, 1.46,
1.15, 0.88, 0.90, 0.74, 1.21),
age = factor(rep(c("At term", "12-26 Weeks"), c(10, 5)))
)
Step 2: run the test with coin and ask for the permutation distribution explicitly
The syntax is oneway_test(formula, data, subset = NULL, weights = NULL, ...), with a formula of the form y ~ x | block, where y is numeric, x is a factor and the optional block is a stratification factor. The manual states that an asymptotic approximation of the exact distribution is used by default (distribution = "asymptotic"), and that Monte Carlo resampling or an exact computation for univariate two-sample problems is requested by setting distribution to "approximate" or "exact". So the first call below is not yet a permutation-distribution test; the second and third are:
library(coin)
oneway_test(pd ~ age, data = diffusion) # asymptotic (default)
set.seed(1)
pvalue(oneway_test(pd ~ age, data = diffusion,
distribution = approximate(nresample = 10000))) # Monte Carlo
ot = oneway_test(pd ~ age, data = diffusion, distribution = "exact")
pvalue(ot) # exact
These three calls come from the coin manual’s own example, apart from the set.seed(1) line and the assignment style. approximate() takes nresample (default 10000), parallel ("no", "multicore" or "snow"), ncpus and cl; exact() takes algorithm ("auto", "shift" or "split-up"); asymptotic() takes maxpts, abseps and releps. For a one-sided hypothesis add alternative = "less" or "greater".
If you need a statistic other than the one oneway_test() uses, note that since coin 1.1-0 oneway_test() no longer lets you specify the statistic; the manual directs you to independence_test() for more control.
Step 3: the same comparison with perm
The syntax is permTS(x, y, alternative = c("two.sided", "less", "greater"), exact = NULL, method = NULL, methodRule = methodRuleTS1, control = permControl(), ...), with a formula method permTS(formula, data, subset, na.action, ...) whose right-hand side is a two-level factor. The method argument takes "pclt", "exact.network", "exact.ce" or "exact.mc" (Monte Carlo). For permTS the statistic is equivalent to the difference of the group means.
library(perm)
permTS(pd ~ age, data = diffusion, method = "exact.mc",
control = permControl(nmc = 10000 - 1))
Step 4: permute inside a linear model with lmPerm
When the question is a regression coefficient or an ANOVA effect rather than a two-group difference, lmPerm provides lmp(formula, data, perm = "Exact", seqs = FALSE, center = TRUE, subset, weights, na.action, ...) and aovp(formula, data = NULL, perm = "Exact", seqs = FALSE, ...). The perm argument takes "Exact", "Prob" or "SPR" and all three give permutation probabilities; seqs = TRUE uses sequential sums of squares and FALSE uses unique sums of squares.
library(lmPerm)
summary(lmp(pd ~ age, data = diffusion, perm = "Prob"))
For the one-way ANOVA case beyond two groups, compare with the running ANOVA in R guide; the permutation version changes how the p-value is computed, not the model formula.
Step 5: write the loop yourself in base R when you need full control
The R documentation for sample() gives the usage sample(x, size, replace = FALSE, prob = NULL) and states that sample(x) generates a random permutation of the elements of x. That is all a hand-rolled permutation test needs: shuffle the labels, recompute the statistic, repeat.
set.seed(2026)
means = function(y, g) tapply(y, g, mean)
obs = diff(means(diffusion$pd, diffusion$age))
B = 9999
perm_stat = replicate(B, diff(means(diffusion$pd, sample(diffusion$age))))
p_two = (sum(abs(perm_stat) >= abs(obs) - 1e-12) + 1) / (B + 1)
p_two
Here diff() subtracts the first factor level’s mean from the second, which is “12-26 Weeks” and “At term” because R orders levels alphabetically; the sign does not matter for the two-sided count. The + 1 in numerator and denominator counts the observed arrangement as one member of the null distribution, the same convention SciPy’s documentation describes for randomised tests, and it keeps a Monte Carlo p-value from being exactly zero. The small tolerance 1e-12 stops floating-point rounding from excluding permutations that tie with the observed value. For two samples this loop and a package should agree up to Monte Carlo error; use the bootstrap in R guide instead if your goal is a confidence interval rather than a p-value.
How to read the output
Which object comes back depends on the route, and so does what a “p-value” means.
- coin.
oneway_test()returns an object you can query withpvalue()for the p-value andstatistic()for the test statistic; the manual also documentsexpectation()andcovariance()for the conditional moments. For an exact two-sample test,support(ot)lists the possible values of the statistic, anddperm(ot, s)andpperm(ot, s)give the null density and distribution function ats. Confidence intervals are not available foroneway_test(); they are for tests such aswilcox_test()when you passconf.int = TRUEand then callconfint(). - perm.
permTS()returns anhtestobject (anmchtestobject whenmethod = "exact.mc") withp.value, ap.valuesvector (lower, upper, two-sided andp.equal), anestimate, and for Monte Carlo runs a confidence interval for the p-value itself (p.conf.int) and the number of resamples (nmc). - lmPerm.
lmp()andaovp()report permutation probabilities. Withperm = "Prob"the algorithm stops when the estimated standard error of the p-value falls belowCatimes the p-value, withmaxIterdefaulting to 1000 iterations. - Your own loop. The result is a proportion. Report it with the number of permutations, and remember that a Monte Carlo p-value carries its own sampling error: a finer p-value needs more permutations.
When reporting, name the statistic (difference of means, Wilcoxon rank sum), whether the p-value is exact or Monte Carlo and how many resamples, the seed, the alternative (one- or two-sided) and the package version. For wording on the p-value itself, see the p-value dictionary entry.
Troubleshooting a permutation test in R
The p-value is not a permutation p-value
In coin the default is distribution = "asymptotic". If you call oneway_test(pd ~ age, data = diffusion) and stop, you have a large-sample approximation to the conditional distribution, not a resampled or exact one. Add distribution = approximate(nresample = 10000) or distribution = "exact".
Two packages give slightly different p-values
Two causes are documented. First, Monte Carlo variation: coin uses 10000 resamples by default and perm‘s permControl() uses nmc = 10^3 - 1 (999), so equal settings and seeds are needed before comparing. Second, the two-sided convention: perm‘s default tsmethod = "central" computes the two-sided p-value as min(1, 2 * min(pless, pgreater)), and the perm manual states that tsmethod = "abs" matches the default two-sided method of coin. If the two disagree on a skewed small sample, set tsmethod = "abs" in permControl() before suspecting an error.
The result changes every time you run it
Monte Carlo resampling draws random permutations. For coin and a hand-written loop, call set.seed() before the test. perm resets the seed to 1234321 each time by default (seed = 1234321, setSEED = TRUE), so it repeats unless you set setSEED = FALSE. Seed behaviour of sample() also changed in R 3.6.0, which the sample() documentation notes; an old script can give a different shuffle under a newer R even with the same seed, and RNGkind(sample.kind = ...) controls it.
Out-of-memory errors with exact methods
The perm manual warns that exact.network and exact.ce “may give errors related to running out of memory” at roughly 15 observations per group for exact.network or 14 per group for exact.ce. Use method = "exact.mc" or "pclt" beyond that. SciPy’s documentation makes the same point for Python: exact tests are feasible only for very small data sets because the number of distinct permutations grows very rapidly with sample size. In lmPerm, maxExact caps the number of observations for perm = "Exact", beyond which "Prob" is used.
Choosing between a permutation test and a rank test
A permutation test of the mean difference and the Wilcoxon-Mann-Whitney test answer related but different questions: one permutes the raw values and uses their mean difference, the other permutes ranks. Both are available through coin, so the choice is a statistical decision about the estimand, made before looking at the data.
Free alternatives to R for permutation tests
R and its packages are free and open. If your analysis pipeline is in Python, SciPy provides scipy.stats.permutation_test; the SciPy stats guide describes the library more broadly. The table lists the entry points of each tool as given in the official documentation; the data table below adds the default settings.
| Tool | Language | Entry function | What it permutes |
|---|---|---|---|
| coin | R | oneway_test(), independence_test(), wilcox_test() |
Group labels, for location and independence tests |
| perm | R | permTS(), permKS(), permTREND() |
Two-sample, k-sample and trend problems |
| lmPerm | R | lmp(), aovp() |
Linear-model and ANOVA terms |
| SciPy | Python | scipy.stats.permutation_test(data, statistic, ...) |
Any statistic you supply as a function, with permutation_type of "independent", "samples" or "pairings" |
The SciPy function’s n_resamples defaults to 9999; when it is greater than or equal to the number of distinct permutations, the exact null distribution is computed, and otherwise the test is randomised. The result object has statistic, pvalue and null_distribution attributes. Its default two-sided p-value is twice the smaller one-sided p-value, the same form as perm‘s default.
Sources
- CRAN: coin, Conditional Inference Procedures in a Permutation Test Framework, checked 2026-11-17
- CRAN: coin reference manual (version 1.4-6), checked 2026-11-17
- CRAN: perm reference manual (version 1.0-0.4), checked 2026-11-17
- CRAN: lmPerm reference manual (version 2.1.6), checked 2026-11-17
- R Core Team: sample, Random Samples and Permutations (base R documentation), checked 2026-11-17
- R Core Team: install.packages (utils documentation), checked 2026-11-17
- SciPy: scipy.stats.permutation_test, checked 2026-11-17
Data: Settings that decide a permutation test result: coin, perm, lmPerm and SciPy
Compares four free permutation-test routes (three R packages and SciPy for Python) on the entry function, how an exact test is requested, the default resampling setting and the default p-value convention. Compiled from the CRAN reference manuals for coin 1.4-6, perm 1.0-0.4 and lmPerm 2.1.6 and the SciPy documentation, read 2026-11-17.
Period covered: 2026.
| Route | Entry function | How exact or Monte Carlo is requested | Default resampling or stopping setting | Default p-value basis |
|---|---|---|---|---|
| coin (R) | oneway_test(formula, data, ...) | distribution = "exact" or approximate(nresample = ...); also "asymptotic" | nresample = 10000 for distribution = approximate() | Asymptotic approximation of the conditional distribution (distribution = "asymptotic") |
| perm (R) | permTS(x, y, ...) or permTS(formula, data) | method = "exact.network", "exact.ce", "exact.mc" or "pclt" | nmc = 10^3 - 1 (999) in permControl(); seed 1234321 reset by default | Two-sided p-value tsmethod = "central": min(1, 2 * min(pless, pgreater)) |
| lmPerm (R) | lmp(formula, data, perm = "Exact") and aovp() | perm = "Exact", "Prob" or "SPR"; maxExact caps observations for Exact | perm = "Prob" stops when estimated SE of p is below Ca * p; maxIter = 1000 | Permutation probabilities, as printed by the model summary |
| SciPy (Python) | scipy.stats.permutation_test(data, statistic, ...) | Exact when n_resamples is at least the number of distinct permutations; otherwise randomised | n_resamples = 9999 | Two-sided: twice the smaller one-sided p-value; randomised tests add one to numerator and denominator |
Sources
- CRAN coin reference manual (1.4-6) (accessed 2026-11-17)
- CRAN perm reference manual (1.0-0.4) (accessed 2026-11-17)
- CRAN lmPerm reference manual (2.1.6) (accessed 2026-11-17)
- SciPy scipy.stats.permutation_test (accessed 2026-11-17)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







