Written and maintained by CASRAI Editorial Board
Last updated
Tukey’s HSD (honestly significant difference) test is the standard post hoc procedure for testing every possible pair of group means against each other after a significant one-way ANOVA — and it is built specifically for that all-pairwise case, not for a smaller set of planned comparisons or for groups with unequal variances. Reach for a different post hoc test the moment either of those two conditions doesn’t hold: Bonferroni (or Scheffé) when you’re testing a small, pre-specified subset of comparisons rather than all pairs, and Games-Howell when Levene’s test flags unequal variances across groups. This guide covers what Tukey HSD actually controls, the assumptions that make it valid, how it compares to the alternatives researchers reach for instead, a worked post-hoc table, and the exact line to write in a Results section.
What Tukey’s HSD test actually does
A significant omnibus one-way ANOVA tells you that at least one group mean differs from the others somewhere among the groups — it does not tell you which specific pairs differ. Tukey’s HSD, developed by John Tukey in the 1950s, answers that follow-up question by testing every pairwise combination of group means at once, using the studentized range distribution (denoted q) rather than the ordinary t-distribution a single pairwise test would use.
That choice of distribution is the whole point of the test: the studentized range distribution accounts for the fact that, with k groups, you are implicitly evaluating k(k−1)/2 pairwise differences drawn from the same set of sample means, and the largest of those differences is expected to look bigger than any single planned comparison purely by chance. Tukey HSD sets its critical value so that the family-wise error rate — the probability of at least one false-positive pairwise conclusion across the entire set of comparisons — stays at the nominal alpha (typically .05), rather than inflating with every additional pair tested.
For exactly this job — testing all pairwise comparisons among group means, with no comparisons pre-specified in advance — Tukey HSD is the most statistically powerful correction available: it holds the family-wise error rate at alpha while sacrificing less power than a general-purpose correction like Bonferroni has to, because Bonferroni’s correction isn’t built around the specific structure of an all-pairs comparison set.
The assumptions Tukey HSD depends on
Tukey HSD inherits the assumptions of the ANOVA it follows, plus one that specifically determines whether Tukey is even the right post hoc test to reach for:
- Homogeneity of variance across groups. This is the assumption that decides whether Tukey HSD is appropriate at all. Tukey’s original test assumes the groups being compared have equal (or approximately equal) population variances, commonly checked with Levene’s test before running the post hoc comparisons. When Levene’s test is significant, the standard Tukey HSD test is the wrong tool — see the Games-Howell section below.
- Independence of observations. Each observation is independent of every other observation, both within and across groups — the same requirement the underlying ANOVA depends on.
- Approximately normal residuals within each group, particularly important at smaller sample sizes.
- Equal (or reasonably close) group sizes. The original Tukey HSD formula was derived for equal n per group. With unequal group sizes, most statistical software (SPSS, R’s
TukeyHSD(), SAS) automatically substitutes the Tukey-Kramer adjustment, which replaces the shared per-group n in the standard error formula with a harmonic mean of the two group sizes being compared for each pairwise test. This is a genuine extension of the same method, not a different test — when a source or a software manual says “Tukey-Kramer,” it is describing Tukey HSD run on unequal group sizes, and it is the version almost every modern package actually runs by default whenever n is unequal.
Tukey HSD is also only valid as a follow-up to a factor with three or more levels tested in a genuine one-way (or the relevant single factor within a factorial) ANOVA — running it after only two groups is unnecessary, since a two-group comparison is already a single pairwise test with no multiplicity problem to correct for.
Tukey HSD vs. the alternatives
The post hoc test that belongs in a given analysis is determined by two design questions: how many of the possible pairwise comparisons are actually being tested, and whether the equal-variance assumption holds. Tukey HSD is the right default only when the answer to the first question is “all of them” and the second is “yes.”
| Test | Use it when | Variance assumption | Comparison set |
|---|---|---|---|
| Tukey HSD (Tukey-Kramer) | Comparing every possible pair of group means; the most powerful choice for exactly this job | Assumes equal variances (check with Levene’s test first) | All pairwise comparisons |
| Games-Howell | Levene’s test is significant — variances differ meaningfully across groups | Does not assume equal variances; uses a Welch-type adjusted degrees of freedom per pair | All pairwise comparisons |
| Bonferroni | A small number of comparisons decided before seeing the data, not necessarily every pair | No variance-homogeneity assumption built into the correction itself | Any pre-specified subset — becomes very conservative as the number of comparisons grows |
| Scheffé’s method | Testing complex contrasts beyond simple pairs (e.g., the average of two groups vs. a third) | Assumes equal variances | Any linear contrast, including but not limited to pairwise — the most conservative of this group for pairwise-only comparisons |
| Dunnett’s test | Comparing several treatment groups back to a single shared control, not to each other | Assumes equal variances | Each treatment vs. control only, not all pairs |
The practical decision rule: run Levene’s test first. If it’s not significant, and every pairwise comparison genuinely matters (not just comparisons back to a control group), Tukey HSD is the standard choice. If Levene’s test is significant, move to Games-Howell — it is built for exactly this situation and remains an all-pairwise test, so it doesn’t force you to also narrow which comparisons you’re allowed to make. Reach for Bonferroni instead when the actual research question only calls for a handful of specific, pre-specified comparisons — applying Bonferroni’s correction across all pairs when you only need three planned comparisons pays an unnecessary power cost for protection you don’t need. Reserve Dunnett’s test specifically for the many-treatments-vs-one-control design, where it is both more powerful and more directly aligned to the actual research question than treating the control as just another group in an all-pairs Tukey comparison.
A worked example
The following is a hypothetical illustrative worked example, not data from a real study, included to show what a Tukey HSD post hoc table looks like once it’s populated. Suppose a one-way ANOVA on four training methods (A, B, C, D) applied to a continuous performance score returns a significant omnibus result, F(3, 76) = 5.91, p = .001, with Levene’s test non-significant (p = .41), so the equal-variance assumption holds and Tukey HSD is the appropriate post hoc test.
| Comparison | Mean difference | Std. error | Adjusted p (Tukey) | 95% CI |
|---|---|---|---|---|
| A vs. B | 4.20 | 1.55 | .041 | [0.15, 8.25] |
| A vs. C | 6.80 | 1.55 | <.001 | [2.75, 10.85] |
| A vs. D | 1.10 | 1.55 | .892 | [−2.95, 5.15] |
| B vs. C | 2.60 | 1.55 | .343 | [−1.45, 6.65] |
| B vs. D | −3.10 | 1.55 | .199 | [−7.15, 0.95] |
| C vs. D | −5.70 | 1.55 | .003 | [−9.75, −1.65] |
Reading this table the way a Results section would: the omnibus test says at least one pair differs; the post hoc table shows exactly which ones — A significantly outperforms both B and C, C significantly underperforms D, and the remaining three pairs (A vs. D, B vs. C, B vs. D) show no significant difference once the Tukey adjustment is applied. Notice that the adjusted p-values are the ones that matter here, not an unadjusted pairwise t-test p-value — software that reports both will label the Tukey-adjusted column distinctly (SPSS’s Multiple Comparisons table, R’s TukeyHSD() output, or SAS’s ADJUST=TUKEY option all report it this way).
How to report Tukey HSD results
A complete Results section reports the omnibus ANOVA first, then the post hoc comparisons — and for each significant pairwise difference, the mean difference, the adjusted p-value, and (increasingly expected, and required by APA 7th edition style) a confidence interval for the difference, not the adjusted p-value alone. Using the worked example above, the standard APA-style line reads:
“A one-way ANOVA revealed a significant effect of training method on performance score, F(3, 76) = 5.91, p = .001. Tukey’s HSD post hoc comparisons indicated that Group A scored significantly higher than Group C (Mdiff = 6.80, 95% CI [2.75, 10.85], p < .001) and Group B (Mdiff = 4.20, 95% CI [0.15, 8.25], p = .041), and that Group D scored significantly higher than Group C (Mdiff = 5.70, 95% CI [1.65, 9.75], p = .003). No other pairwise comparisons reached significance (all p > .19).”
Three details that reviewers most often flag when this is done incorrectly:
- Report which correction was used, by name. “Post hoc tests showed Group A differed from Group C” without naming Tukey HSD (or whichever correction was actually applied) leaves the reader unable to judge whether family-wise error was controlled appropriately for the comparison set actually being made.
- Report the adjusted p-value, not the raw pairwise p-value. Some software prints both in the same output table; submitting the unadjusted figure defeats the purpose of running a correction at all.
- State the non-significant comparisons too, at least in summary (“all other comparisons p > .19”), rather than reporting only the significant pairs and leaving the rest for the reader to infer from a table that may not even be included in the manuscript.
Frequently asked questions
Do I need a significant ANOVA before running Tukey HSD?
Yes. Tukey HSD is a follow-up procedure, not a standalone test — it identifies which pairs of means differ once the omnibus ANOVA has already established that some difference exists among the groups. Running post hoc pairwise tests after a non-significant omnibus ANOVA is a common source of spurious “significant” pairs and is generally discouraged, though some methodologists note the two procedures don’t test mathematically identical hypotheses and a small minority of designs report post hoc results regardless — the safer default for most manuscripts is to run post hoc comparisons only when the omnibus test is significant.
What’s the difference between Tukey HSD and Tukey-Kramer?
They’re the same underlying test. “Tukey-Kramer” specifically refers to Tukey’s method applied with unequal group sizes, using a harmonic-mean adjustment to the standard error for each pairwise comparison. Most statistical software applies this adjustment automatically whenever group sizes differ, so in practice “Tukey HSD” output from SPSS, R, or SAS on unbalanced data is already the Tukey-Kramer version.
Can Tukey HSD be used with unequal variances if I just want to be conservative?
No — this is a common misconception. Using Tukey HSD when variances are meaningfully unequal doesn’t make the result “conservative,” it makes the family-wise error rate calculation invalid, because the test’s critical value assumes a single shared variance across groups. Run Levene’s test first; if it’s significant, switch to Games-Howell rather than keeping Tukey HSD and treating it as an extra-safe choice.
Is Tukey HSD more or less conservative than Bonferroni?
For the specific case of testing all pairwise comparisons, Tukey HSD is less conservative (more powerful) than Bonferroni, because it’s built around the exact structure of an all-pairs comparison set rather than applying a general-purpose correction that doesn’t know how many comparisons are actually related to each other. Bonferroni becomes the better choice only when the comparison set is smaller than “every pair” and decided in advance of seeing the data.
Related CASRAI resources
See CASRAI’s guide to running a one-way ANOVA in SPSS for the full procedure including Tukey and Games-Howell post hoc output, running ANOVA in R and ANOVA in Stata for those platforms, and two-way ANOVA in SPSS and MANOVA in SPSS for factorial and multivariate designs where post hoc comparisons follow a different structure. For the underlying concepts, see CASRAI’s entries on p-values, confidence intervals, effect size, and the Bonferroni correction and multiple comparisons more broadly.
