Written and maintained by CASRAI Editorial Board
Last updated
The Scheffé test is the only common post hoc procedure built to test any linear contrast among group means after ANOVA — not just the pairwise differences Tukey’s HSD handles, but complex, non-pairwise comparisons like “is the average of Group A and Group B different from the average of Group C and Group D?” That generality is also its cost: because Scheffé’s critical value has to protect against every contrast you could possibly form from your data, not just the specific one you’re testing, it is the most conservative of the standard post hoc corrections whenever what you actually want is a simple pairwise comparison. Reach for it specifically when your comparison of interest isn’t a simple pair — for straightforward all-pairs testing, Tukey’s HSD is more powerful, and for a small, pre-planned set of comparisons, Bonferroni often is too.
What the Scheffé test actually tests
A significant omnibus ANOVA tells you that group means differ somewhere among your k groups, but not which comparison drives that difference. Most researchers reach for a post hoc test that checks every pair — but a pairwise comparison is only one kind of contrast. A contrast is any linear combination of group means whose coefficients sum to zero: comparing one group to another is a contrast with coefficients like (1, −1, 0, 0), but so is comparing the average of two groups to the average of two others, with coefficients like (0.5, 0.5, −0.5, −0.5), or comparing one control group to the average of three treatment arms. Tukey’s HSD and the Newman-Keuls procedure are built specifically around the pairwise case and don’t have a defined critical value for anything else. Scheffé’s method, developed by the statistician Henry Scheffé, is built around the general case: it supplies a single critical value valid for testing every contrast you could construct from the group means, including ones you didn’t decide to test until after looking at the data.
The mechanics: one critical value that covers every contrast
For a one-way ANOVA with k groups and N total observations, Scheffé’s critical value is:
S = √[(k − 1) × Fα; k−1, N−k]
where Fα; k−1, N−k is the same critical F-value the omnibus ANOVA itself uses at significance level α. For any contrast L = Σci·x̄i (coefficients ci summing to zero), you compute its standard error SE = √[MSwithin × Σ(ci²/ni)], then declare the contrast significant if |L| / SE exceeds S. Because S is built from the omnibus F-distribution rather than a distribution calibrated to a specific comparison structure, it’s valid no matter which contrast — pairwise, complex, planned in advance, or dreamed up after eyeballing the means — you plug in. That’s also exactly why it’s conservative for the pairwise case specifically: Tukey’s HSD uses the studentized range distribution, calibrated tightly to the pairwise-comparison structure, so it can afford a smaller (less demanding) critical value for that one job. Scheffé’s S has to be large enough to hold the family-wise error rate at α across the entire, effectively infinite space of possible contrasts, which pushes it above what a test built for pairs alone needs.
A worked example: a complex contrast vs. a simple one
The numbers below come from a seeded, reproducible simulation (not a real study) — four groups of n=9, representing a hypothetical comparison of four teaching-method conditions on a 0–100 assessment score, generated from a fixed random seed so the calculation can be checked independently.
| Group | n | Mean |
|---|---|---|
| A — lecture-only | 9 | 77.40 |
| B — flipped classroom | 9 | 79.77 |
| C — peer instruction | 9 | 84.27 |
| D — self-paced module | 9 | 74.61 |
The omnibus one-way ANOVA on this data is significant: F(3, 32) = 3.90, MSwithin = 38.55, against a critical F.05; 3,32 = 2.90 — so at least one real difference exists somewhere among the four groups. Scheffé’s critical value for any contrast at α = .05 is S = √(3 × 2.90) = 2.95.
Complex contrast — average of the two active-learning conditions (B, C) vs. average of the two passive conditions (A, D), coefficients (−0.5, 0.5, 0.5, −0.5). This is exactly the kind of comparison Tukey’s HSD has no critical value for, because it isn’t one of the k(k−1)/2 pairwise differences — Scheffé handles it directly. L = 6.01, SE = 2.07, ratio = |L|/SE = 2.90. That falls just short of S = 2.95, so this particular contrast is not significant by Scheffé’s stricter standard even though the omnibus test is.
Simple pairwise contrast, for comparison — peer instruction (C) vs. lecture-only (A), coefficients (−1, 0, 1, 0). L = 6.87, SE = 2.93, ratio = 2.35. An unadjusted two-sample comparison at this same df (32) would call that significant — the unadjusted two-tailed critical t-value here is 2.04, and 2.35 clears it. But Scheffé’s ratio needs to clear 2.95, and 2.35 doesn’t — so the same comparison that would look significant if you tested it in isolation, with no correction, does not survive Scheffé’s family-wise protection. That gap between 2.04 and 2.95 is the conservatism in concrete terms: it’s the price of a single critical value that stays valid across the entire space of contrasts you could have formed from this data, not just the one pair you happened to pick.
Why Scheffé is conservative — and when it isn’t
The trade-off is structural, not a quirk of this example. Because S is calibrated to protect against every possible contrast, including ones no one actually tests, it spends statistical power on protection you don’t need whenever you’re only testing a handful of comparisons — particularly pairwise ones, where Tukey’s HSD is calibrated exactly to that narrower job and is correspondingly more powerful for it. The flip side: when you genuinely are testing many contrasts — more than the number of pairwise comparisons a Bonferroni correction would have to split alpha across — Scheffé’s single omnibus-based critical value can end up less conservative than Bonferroni, because Bonferroni’s penalty grows directly with the number of tests while Scheffé’s does not. The practical rule: the more comparisons you’re testing, and the less those comparisons are limited to simple pairs, the more Scheffé’s flexibility earns back the power it costs elsewhere.
Scheffé vs. Tukey HSD vs. Bonferroni
| Test | Handles | Best for | Relative power for pairwise-only |
|---|---|---|---|
| Tukey HSD | All pairwise comparisons only | Testing every pair, nothing else | Highest |
| Bonferroni | A small, pre-specified set of comparisons (pairwise or not) | A few planned comparisons decided before seeing the data | Moderate; degrades as the number of planned comparisons grows |
| Scheffé | Any linear contrast, planned or post hoc, pairwise or complex | Complex/non-pairwise contrasts, or exploring many contrasts after the fact | Lowest for pairwise-only use |
If every comparison you care about is a simple pair and you’re testing all of them, use Tukey. If you have a short, specific list of comparisons decided in advance, Bonferroni is usually simpler and adequately powered. Reach for Scheffé specifically when at least one comparison you need is a genuine complex contrast — an average-vs-average, a control-vs-combined-treatments comparison, or any comparison you only thought to test after looking at the pattern of means — since those aren’t comparisons Tukey or a standard Bonferroni list is built to cover at all.
How to report a Scheffé result
Report the contrast definition, its estimate, and the comparison against Scheffé’s critical value (or, if your software reports it directly, the Scheffé-adjusted p-value) alongside the omnibus ANOVA. For the complex contrast above, a Results-section line would read: “A Scheffé-adjusted contrast comparing the two active-learning conditions to the two passive conditions was not significant, L = 6.01, SE = 2.07, ratio = 2.90 (critical value S = 2.95), despite a significant omnibus effect, F(3, 32) = 3.90, p < .05." Most statistical software (R's DescTools::ScheffeTest(), SPSS’s post hoc options, SAS’s PROC GLM with LSMEANS/ADJUST=SCHEFFE) reports this as an adjusted p-value directly rather than requiring you to compare ratios by hand — report whichever your software gives you, but state the contrast’s exact definition either way, since “Scheffé test” alone doesn’t tell a reader which comparison was tested.
Frequently asked questions
Is the Scheffé test only for post hoc comparisons?
No — it’s valid for planned contrasts too, though if your contrasts are genuinely planned in advance and few in number, Bonferroni is usually more powerful for that specific case. Scheffé’s advantage is that it stays valid even for contrasts you didn’t plan, which planned-comparison methods aren’t built to allow.
Can Scheffé test more than two groups at once in a single contrast?
Yes — that’s exactly the case it’s built for. A contrast can involve any number of the k groups, with any coefficients that sum to zero, such as comparing one group to the average of the rest.
Why did my omnibus ANOVA come back significant but no Scheffé contrast is?
This can genuinely happen, and it isn’t a contradiction. The omnibus F-test and Scheffé’s contrast tests use related but not identical logic: the omnibus test detects that some linear combination of the means differs from zero, while each specific contrast you test is only one candidate for that combination. If the real pattern in the data doesn’t line up neatly with the specific contrasts you happened to test, the omnibus test can flag real variation that no individual tested contrast clears the (conservative) Scheffé bar to confirm.
Does Scheffé assume equal variances across groups, like Tukey does?
Yes — the standard Scheffé method inherits the same homogeneity-of-variance and independence assumptions as the ANOVA it follows, since it’s built directly from the ANOVA’s pooled MSwithin and residual degrees of freedom. If Levene’s test flags unequal variances, a variance-robust alternative is generally more appropriate than either Scheffé or Tukey.
