Written and maintained by CASRAI Editorial Board
Published
On this page
To run a Tukey Honest Significant Difference (HSD) test in R, fit a one-way analysis of variance using the base stats function aov() and pass the resulting model object directly into TukeyHSD(). The function calculates all pairwise group mean comparisons while controlling the family-wise error rate through the Studentized range distribution, outputting mean differences, adjusted 95% confidence intervals, and adjusted p-values. Researchers commonly execute this post-hoc test following a statistically significant omnibus F-test to pinpoint exactly which experimental treatments differ without inflating Type I error.
What it is used for in research
In empirical research across biological, medical, agricultural, and behavioural sciences, experimental designs frequently compare more than two treatment conditions. When evaluating three or more group means, conducting repeated two-sample t-tests rapidly inflates the family-wise error rate—the probability of committing at least one false positive (Type I error) across the entire family of comparisons. For an experiment with k groups, there are k(k – 1) / 2 pairwise comparisons. In an experiment with four treatment groups, six pairwise tests conducted at an unadjusted α = 0.05 yield a cumulative family-wise error rate of approximately 1 – (1 – 0.05)6 ≈ 26.5%, dramatically undermining statistical validity.
The standard research workflow begins with an omnibus analysis of variance to determine whether any difference exists among the group means. However, the omnibus F-test is non-directional and non-specific: it indicates that at least one group mean deviates from the rest, but cannot identify which specific pairs differ. John Tukey introduced the Honestly Significant Difference (HSD) test to resolve this dilemma by establishing a single critical difference threshold derived from the Studentized range distribution (q). This distribution accounts for the spread between the highest and lowest sample means under the null hypothesis of equal population means.
In research practice, the Tukey HSD procedure is preferred when an investigator wishes to evaluate all possible pairwise comparisons among group means while maintaining the overall family-wise error rate strictly at the nominal level (typically α = 0.05). Compared to more conservative multiplicity adjustments, such as Bonferroni versus Tukey HSD corrections, Tukey’s test retains substantially greater statistical power when testing all pairwise contrasts because it explicitly models the joint distribution of pairwise differences rather than relying on Bonferroni’s crude inequality bound. Understanding the broader statistical theory of Tukey’s HSD test helps analysts select appropriate post-hoc techniques for their specific experimental hypotheses.
Install or access R and required packages
The primary function for computing Tukey’s test in R is TukeyHSD(), which is part of the core stats package. Because the stats package is bundled with the base distribution of R, no additional installation or repository download is required to run standard Tukey tests. Any standard installation of R (version 3.0.0 through current R 4.x releases) provides immediate access to both aov() and TukeyHSD() upon launching R or RStudio.
To verify that base R and the stats environment are active and to inspect your current R version, execute the following command in the R console:
# Check current R session information and stats package availability
sessionInfo()
packageVersion("stats")
While base R provides all necessary calculation tools, modern reproducible research pipelines often benefit from complementary CRAN packages that facilitate tidy output formatting, advanced contrast modeling, and publication-quality data visualization. These optional packages include:
- broom: Converts S3
TukeyHSDmatrices into tidy tibbles suitable for filtering, joining, and automated table generation. - ggplot2: Constructs customizable forest plots of pairwise differences and confidence intervals.
- emmeans: Computes estimated marginal means (least-squares means) and pairwise contrasts for complex, unbalanced, or mixed-effects models.
- multcomp: Implements general linear hypothesis testing (
glht) and compact letter displays (cld).
To install these supplementary packages from the official Comprehensive R Archive Network (CRAN), execute:
# Install recommended supplementary packages from CRAN
install.packages(c("broom", "ggplot2", "emmeans", "multcomp"))
Step-by-step task: running TukeyHSD in R
Executing Tukey’s test in R follows a systematic sequence: formatting the dataset, verifying factor structures, fitting the analysis of variance model with aov(), computing the pairwise contrasts with TukeyHSD(), and extracting or visualizing the results. The following numbered steps illustrate this process using R’s built-in PlantGrowth dataset, which records dried plant yields across a control group and two distinct treatment conditions (30 total observations, 10 per group).
Step 1: Inspect data and verify factor coding
A frequent pitfall in R data analysis occurs when experimental grouping variables are stored as numeric integers or raw character strings rather than explicit factor vectors. aov() and TukeyHSD() require grouping variables to be coded as factors. Inspect the structure and confirm factor levels:
# Load and inspect the built-in PlantGrowth dataset
data("PlantGrowth")
str(PlantGrowth)
# Verify that the grouping variable is an explicit factor
is.factor(PlantGrowth$group)
# If your grouping variable is numeric or character, convert it:
# PlantGrowth$group <- as.factor(PlantGrowth$group)
# View group sample sizes and summary statistics
table(PlantGrowth$group)
tapply(PlantGrowth$weight, PlantGrowth$group, function(x) c(mean = mean(x), sd = sd(x)))
Step 2: Fit the omnibus ANOVA model using aov()
Tukey’s test requires a fitted model object of class aov. While lm() fits linear regressions, TukeyHSD() specifically requires the output of aov(). Fit the one-way ANOVA model using formula notation (response ~ predictor) and inspect the summary:
# Fit the one-way ANOVA model using aov()
plant_aov <- aov(weight ~ group, data = PlantGrowth)
# Review the omnibus ANOVA summary table
summary(plant_aov)
The resulting ANOVA table reports degrees of freedom, sums of squares, mean squares, the F-statistic, and the corresponding p-value. In this dataset, the omnibus F-test yields F(2, 27) = 4.846, p = 0.0159. Because p < 0.05, we reject the null hypothesis of equal group means and proceed with post-hoc pairwise testing.
Step 3: Execute TukeyHSD() on the fitted model
Pass the fitted plant_aov object directly into TukeyHSD(). By default, the function evaluates all factor terms in the model at a 95% family-wise confidence level:
# Run Tukey's Honest Significant Difference test
plant_tukey <- TukeyHSD(plant_aov)
# Print the pairwise comparison results
print(plant_tukey)
The console prints the factor name ($group) followed by a matrix detailing every pairwise comparison (trt1-ctrl, trt2-ctrl, and trt2-trt1). For each pair, the output reports four values: diff, lwr, upr, and p adj.
Step 4: Specify terms and ordering parameters
The TukeyHSD() function provides several arguments to tailor its calculations:
which: Restricts testing to specific factors in multi-factor designs. For example,TukeyHSD(model, which = "group")calculates intervals only for the specified term.ordered = TRUE: Orders factor levels by their observed sample means prior to calculating differences. This ensures that all reporteddiffvalues are positive, simplifying comparisons when evaluating directional superiority.conf.level: Adjusts the family-wise coverage probability. The default is0.95(95% confidence intervals), but researchers may specifyconf.level = 0.99for a stricter 99% coverage threshold.
# Run Tukey HSD with means ordered from smallest to largest at 95% confidence
tukey_ordered <- TukeyHSD(plant_aov, which = "group", ordered = TRUE, conf.level = 0.95)
print(tukey_ordered)
Step 5: Run Tukey HSD for Two-Way ANOVA models and interaction terms
When analyzing factorial experiments with two independent factors, researchers often must determine whether to test main effects or interaction cell means. Consult the guide to one-way vs two-way ANOVA designs for architectural principles. When a significant interaction term is detected, main effects can be misleading, and pairwise testing across interaction cells is necessary:
# Fit a two-way factorial ANOVA using warpbreaks dataset
data("warpbreaks")
warp_aov <- aov(breaks ~ wool * tension, data = warpbreaks)
# Test main effects and interaction simultaneously
warp_tukey_all <- TukeyHSD(warp_aov)
# Test only the two-way interaction term (all 15 pairwise cell comparisons)
warp_tukey_interaction <- TukeyHSD(warp_aov, which = "wool:tension")
head(warp_tukey_interaction$`wool:tension`)
Step 6: Visualize pairwise confidence intervals
R supports both base graphics and modern grammar-of-graphics approaches to visualize Tukey HSD intervals. The simplest diagnostic plot uses base R’s generic plot() method:
# Base R diagnostic plot with horizontal axis labels
par(mar = c(5, 8, 4, 2)) # Expand left margin to accommodate comparison labels
plot(plant_tukey, las = 1, col = "darkblue")
In base plots, horizontal lines represent the 95% confidence intervals of the difference between group means, and a vertical dashed line marks zero difference. Any interval that crosses zero indicates a non-significant comparison, whereas intervals that do not encompass zero represent statistically significant differences.
For publication-ready figures, combine broom::tidy() with ggplot2 to render a customized forest plot:
library(broom)
library(ggplot2)
# Convert TukeyHSD output into a tidy tibble
tidy_tukey <- tidy(plant_tukey)
# Create a customized forest plot
ggplot(tidy_tukey, aes(x = estimate, y = contrast)) +
geom_vline(xintercept = 0, linetype = "dashed", color = "gray50") +
geom_errorbarh(aes(xmin = conf.low, xmax = conf.high), height = 0.2, color = "#2b5c8f", size = 0.8) +
geom_point(size = 3, color = "#2b5c8f") +
labs(
title = "Tukey HSD Pairwise Differences (95% CI)",
x = "Estimated Mean Difference",
y = "Comparison Pair"
) +
theme_minimal(base_size = 13)
Output and how to read it
Interpreting R’s TukeyHSD() output correctly requires understanding the four statistical metrics displayed in each row of the results matrix. Below is the exact output generated from the PlantGrowth one-way ANOVA:
Tukey multiple comparisons of means
95% family-wise confidence level
Fit: aov(formula = weight ~ group, data = PlantGrowth)
$group
diff lwr upr p adj
trt1-ctrl -0.371 -1.0622161 0.3202161 0.3908711
trt2-ctrl 0.494 -0.1972161 1.1852161 0.1979960
trt2-trt1 0.865 0.1737839 1.5562161 0.0120064
Each component of this table provides critical information for evaluating pairwise hypotheses:
- Comparison label (row name): Indicates the contrast direction, formatted as
GroupA-GroupB. The metric calculates Mean(GroupA) – Mean(GroupB). For instance,trt2-trt1subtracts treatment 1’s mean from treatment 2’s mean. - diff: The observed difference between the sample means. For
trt2-trt1, the difference is 0.865 units (dried weight), indicating that plants in treatment 2 were on average 0.865 grams heavier than those in treatment 1. - lwr and upr: The lower and upper bounds of the simultaneous 95% confidence interval for the mean difference. For
trt2-trt1, the interval ranges from 0.174 to 1.556. Because both limits are strictly positive and the interval does not span zero, the difference is statistically significant at the 5% family-wise significance level. In contrast, the interval fortrt1-ctrlspans from -1.062 to +0.320; because it includes zero, we cannot reject the null hypothesis of no difference between treatment 1 and control. - p adj: The adjusted p-value calculated via the Studentized range distribution. This p-value controls for multiplicity across all three tests. For
trt2-trt1, p adj = 0.0120, which is less than the standard α = 0.05 threshold. Fortrt1-ctrl(p adj = 0.3909) andtrt2-ctrl(p adj = 0.1980), adjusted p-values exceed 0.05, confirming no significant divergence from the control group.
Understanding how confidence intervals align with hypothesis testing decisions provides valuable perspective; review the principles of hypothesis testing and confidence intervals to strengthen your interpretation of overlapping boundaries. When reporting Tukey HSD outcomes in scientific papers (following APA 7th guidelines), always report the omnibus ANOVA first, followed by the specific pairwise differences, confidence intervals, and adjusted p-values, alongside appropriate ANOVA effect sizes such as partial eta-squared or Cohen’s d.
An exemplary APA-formatted results sentence for this analysis reads: “A one-way ANOVA demonstrated significant variation in plant dry weight among conditions, F(2, 27) = 4.85, p = .016, η² = .26. Post-hoc comparisons using Tukey’s HSD test revealed that plants in Treatment 2 (M = 5.47, SD = 0.44) produced significantly greater biomass than plants in Treatment 1 (M = 4.66, SD = 0.79), with a mean difference of 0.865 g, 95% CI [0.174, 1.556], p = .012. Neither treatment group differed significantly from the control condition (M = 5.03, SD = 0.58; both p > .19).”
Troubleshooting common errors
When implementing Tukey’s test in R, researchers frequently encounter specific execution errors, warning messages, and data structure discrepancies. The following solutions address the most common obstacles.
1. Error: no applicable method for ‘TukeyHSD’ applied to an object of class ‘lm’
Cause: This error occurs when a user fits a linear model using lm() and attempts to pass it directly into TukeyHSD():
# This code triggers the error:
fit_lm <- lm(weight ~ group, data = PlantGrowth)
TukeyHSD(fit_lm) # Error: no applicable method for 'TukeyHSD' applied to an object of class "lm"
Solution: TukeyHSD() is an S3 method designed exclusively for objects inheriting from class aov. Either refit the model using aov(), wrap the existing linear model in aov(), or use the emmeans package, which accepts lm objects directly:
# Solution A: Refit or convert with aov()
fit_aov <- aov(weight ~ group, data = PlantGrowth)
TukeyHSD(fit_aov)
# Solution B: Use emmeans on the existing lm object
library(emmeans)
emmeans(fit_lm, pairwise ~ group)
2. Unbalanced sample sizes and the Tukey-Kramer adjustment
Cause: In real-world experimental datasets, sample sizes across groups are frequently unequal due to subject attrition, specimen loss, or non-response. Standard Tukey HSD assumes equal sample sizes ($n_1 = n_2 = dots = n_k$). When group sizes differ, base R’s TukeyHSD() automatically applies the Tukey-Kramer approximation, which replaces the constant sample size with the harmonic mean of the sample sizes for each specific pair.
Consideration: While TukeyHSD() computes the Tukey-Kramer adjustment, the official R documentation notes that the family-wise error rate and confidence levels are approximate rather than exact when group sizes are unbalanced. For rigorous post-hoc testing in highly unbalanced factorial designs, analysts should verify findings using emmeans or multcomp::glht(), which calculate exact simultaneous confidence intervals based on the estimated covariance matrix.
3. Predictor variable treated as continuous numeric
Cause: If an experimental condition is coded as numbers (e.g., 1, 2, 3 representing low, medium, and high dosages), R treats the variable as a continuous numeric predictor rather than a categorical factor by default. In an ANOVA, this fits a simple linear regression with 1 degree of freedom instead of a multi-group factor, causing TukeyHSD() to fail or report nonsensical single-difference output.
Solution: Always confirm factor status before running ANOVA by calling is.factor(data$variable). If false, convert the column explicitly using data$variable <- factor(data$variable) prior to fitting aov().
4. Heterogeneity of variance (violation of homoscedasticity)
Cause: Tukey’s HSD test relies on the pooled mean squared error (MSE) from the omnibus ANOVA, assuming equal population variances across all groups. When variances are severely heterogeneous (demonstrated by a significant Levene’s test, p < 0.05), Tukey HSD can exhibit inflated Type I error rates.
Solution: When homoscedasticity is violated, do not rely on standard TukeyHSD(). Instead, implement the Games-Howell post-hoc test, which does not assume equal variances or equal sample sizes. In R, the Games-Howell procedure is available through the rstatix package via rstatix::games_howell_test().
Free alternatives and extended packages
While base R’s TukeyHSD() provides a dependable, zero-dependency solution for standard balanced and moderately unbalanced one-way and two-way ANOVAs, the broader R package ecosystem offers specialized packages tailored for complex experimental designs, mixed-effects models, and publication-ready reporting.
The table below summarizes the leading free and open-source packages available on CRAN for computing Tukey post-hoc tests and pairwise comparisons in R:
| Package / Tool | Primary Function | Supported Model Classes | Key Advantages | Best Applied In |
|---|---|---|---|---|
| stats (Base R) | TukeyHSD() | aov | Pre-installed, zero dependencies, rapid execution | Standard balanced one-way and two-way ANOVAs |
| emmeans | emmeans(…, pairwise) | lm, glm, lmer, glmer, nlme | Estimated marginal means, handles complex interactions, covariates, and repeated measures | Mixed-effects models, unbalanced factorial designs, ANCOVA |
| multcomp | glht(…, mcp()) | lm, glm, lmer, survreg | Simultaneous multivariate inference, flexible custom contrast matrices | Generalized linear models and custom hypothesis testing |
| rstatix | tukey_hsd() | data.frame, aov, lm | Pipe-friendly (tidyverse compatible), returns clean tibbles, integrates with ggpubr | Exploratory data analysis and automated ggplot reporting |
| agricolae | HSD.test() | aov, lm | Native compact letter display (CLD) grouping tables, agricultural trial standards | Agronomy, plant breeding, and field trial reporting |
| DescTools | PostHocTest(…, method=”hsd”) | aov | Unified interface for multiple post-hoc tests (Scheffe, Duncan, Newman-Keuls, Tukey) | Comparing multiple post-hoc procedures simultaneously |
For research projects involving complex linear mixed models (e.g., subjects nested within labs, longitudinal measurements), the emmeans package is widely regarded as the gold standard in modern R programming. Rather than relying on simple observed cell means, emmeans evaluates estimated marginal means that adjust for covariates and random-effects structures. The data table below compares the technical specifications, input requirements, and mathematical adjustments of each package in detail.
Frequently asked questions
Does Tukey’s test in R require a significant omnibus ANOVA first?
Statistically, the Studentized range distribution controls the family-wise error rate across all pairwise comparisons regardless of the omnibus F-test outcome. However, in standard scientific practice and academic publishing guidelines, researchers generally require a statistically significant omnibus ANOVA (p < 0.05) before conducting and interpreting post-hoc pairwise comparisons to guard against exploratory data fishing.
Why does TukeyHSD() fail when passed a model created with lm()?
The TukeyHSD() function in base R is an S3 generic method written specifically for objects of class aov. While aov() calls lm() internally, it assigns the specific class attribute c("aov", "lm") that TukeyHSD() checks. To fix this error, either fit the model using aov() instead of lm(), or use the emmeans package, which accepts standard lm model objects.
How does R handle unbalanced sample sizes in TukeyHSD()?
When group sizes are unequal, base R’s TukeyHSD() automatically implements the Tukey-Kramer adjustment, replacing the equal sample size parameter with the harmonic mean of the sample sizes for each specific pair. While this adjustment controls Type I error reasonably well, the resulting confidence intervals and p-values are conservative approximations. For exact inference in unbalanced designs, analysts should use emmeans::emmeans() or multcomp::glht().
How do you test two-way interaction effects using TukeyHSD in R?
In a factorial ANOVA fitted with aov(response ~ factorA * factorB, data = my_data), you test the interaction term by passing the interaction argument into the which parameter: TukeyHSD(model, which = "factorA:factorB"). This generates all pairwise comparisons among the combined cell means.
What is the difference between diff and p adj in R’s Tukey output?
In the TukeyHSD() output matrix, diff represents the observed point estimate of the difference between the two sample means (Mean A minus Mean B). p adj is the p-value for that comparison after adjusting for the multiplicity of testing all pairwise pairs, calculated from the Studentized range distribution (q). If p adj is below your significance threshold (typically 0.05), the difference is considered statistically significant.
How do you interpret a negative difference in Tukey’s HSD output?
A negative value in the diff column simply reflects the subtraction order indicated by the row label (Group1-Group2). If the label is A-B and diff is -3.5, it means Group A’s mean is 3.5 units lower than Group B’s mean. To display all differences as positive values, add the argument ordered = TRUE to the TukeyHSD() call.
Sources
- R Core Team. (2026-11-15). TukeyHSD: Compute Tukey Honest Significant Differences. R Documentation (stats package). https://stat.ethz.ch/R-manual/R-devel/library/stats/html/TukeyHSD.html
- R Core Team. (2026-11-15). aov: Fit an Analysis of Variance Model. R Documentation (stats package). https://stat.ethz.ch/R-manual/R-devel/library/stats/html/aov.html
- Hothorn, T., Bretz, F., and Westfall, P. (2026-11-15). multcomp: Simultaneous Inference in General Parametric Models. CRAN Reference Manual. https://cran.r-project.org/web/packages/multcomp/multcomp.pdf
- Lenth, R. V. (2026-11-15). Comparisons and Contrasts in emmeans. CRAN Package Vignette. https://cran.r-project.org/web/packages/emmeans/vignettes/comparisons.html
- Robinson, D., Hayes, A., and Couch, S. (2026-11-15). tidy.TukeyHSD: Tidy a(n) TukeyHSD object. Broom Documentation. https://broom.tidymodels.org/reference/tidy.TukeyHSD.html
- Mendiburu, F. (2026-11-15). agricolae: Statistical Procedures for Agricultural Research. CRAN Documentation. https://cran.r-project.org/package=agricolae
Data: R Multiple Comparison Packages Benchmark
Comparative specifications of R packages and functions for Tukey post-hoc multiple comparisons, compiled from CRAN manual pages and package documentation.
| Package and Function | Model Class Input | Unbalanced Design Support | Output Format | Built-in Plotting Method | Compact Letter Display |
|---|---|---|---|---|---|
| stats::TukeyHSD | aov only | Tukey-Kramer approximation | List of matrices (class TukeyHSD) | Base graphics plot() | No (requires multcompView) |
| multcomp::glht | lm, glm, lmer, coxph | Simultaneous multivariate t/normal inference | glht object with summary/confint | Base graphics plot(confint()) | Yes (via cld()) |
| emmeans::emmeans | lm, glm, lmer, nlme, brms | Estimated marginal means (least-squares) | emmGrid and emm_list objects | ggplot2 via plot() and emmip() | Yes (via multcomp::cld) |
| rstatix::tukey_hsd | data.frame with formula, or aov | Tukey-Kramer approximation | Tidy tibble (tidyverse compatible) | ggpubr / ggplot2 integration | No |
| agricolae::HSD.test | aov, lm | Harmonic mean sample size correction | agricolae list object with groups table | Base graphics barplot/plot() | Yes (native group table) |
| DescTools::PostHocTest | aov | Tukey-Kramer method | PostHocTest table object | Base graphics plot() | No |
Sources
- R stats package TukeyHSD documentation (accessed 2026-11-15)
- CRAN multcomp package reference manual (accessed 2026-11-15)
- CRAN emmeans package comparisons vignette (accessed 2026-11-15)
- CRAN agricolae package documentation (accessed 2026-11-15)
- CRAN rstatix package reference (accessed 2026-11-15)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







