ANOVA (Analysis of Variance) tests whether the means of a continuous outcome differ across three or more groups in a single test. It is one of the most commonly reported inferential statistics in quantitative research, and one of the most commonly misreported — a significant result is routinely misread as saying more than it actually says. This guide covers the logic behind the test, how to build and read an ANOVA table with a fully worked illustrative example, the major variants (one-way, two-way, repeated-measures, factorial, MANOVA, ANCOVA), what to do when its assumptions fail, the Kruskal-Wallis nonparametric alternative, post-hoc testing, effect size, and how to report results.
Why ANOVA Exists: The Problem With Running Multiple t-Tests
A t-test compares the means of exactly two groups. It is tempting, when comparing three or more groups, to simply run a t-test on every possible pair. This is the single most common ANOVA-adjacent error, and it is worth understanding exactly why it is wrong before anything else in this guide.
Each individual t-test carries its own chance of a false positive (a Type I error) — conventionally set at α = .05, or 5%. That 5% risk applies per test. Compare three groups (A vs. B, A vs. C, B vs. C) and you have run three tests; compare five groups and you have run ten pairwise tests. Assuming the tests were independent, the probability that at least one of them produces a false positive by chance climbs well above 5% as the number of comparisons grows — with three tests it approaches roughly 14%, and with ten tests it approaches roughly 40%. This inflation is called the familywise error rate, and it is the entire reason ANOVA exists.
ANOVA solves this by testing all the groups at once in a single omnibus test, holding the overall Type I error rate at the stated alpha level regardless of how many groups are being compared. It answers one question first: is there evidence that the group means are not all equal? Only after that omnibus test is significant do you move to post-hoc tests (covered below) to work out which specific groups differ — and those post-hoc procedures carry their own correction for multiple comparisons, which uncontrolled pairwise t-tests never had in the first place.
The Core Idea: Partitioning Variance and the F-Ratio
ANOVA’s name describes its actual mechanism, which is easy to lose sight of: it is a test of variances that answers a question about means. That is the single point that confuses nearly everyone encountering it for the first time.
The logic works like this. Take every observation in the dataset and its distance from the overall (grand) mean. That total variability, the total sum of squares (SST), can be split cleanly into two components:
- Between-group variance (SSB) — how far each group’s own mean sits from the grand mean, weighted by group size. This is the variability you would expect if the groups genuinely differ.
- Within-group variance (SSW) — how much individual observations vary around their own group’s mean. This is background noise: the variability you would see even if the groups were identical, just from ordinary sampling variation.
ANOVA compares these two quantities as a ratio, the F-statistic: F = (between-group variance) / (within-group variance), each divided by its own degrees of freedom to produce a mean square. If the groups do not really differ, both the between-group and within-group variances are estimating the same thing (random noise), so F should come out close to 1. If the group means genuinely differ, the between-group variance is inflated by that real difference on top of the noise, pushing F above 1. A large F-ratio means the differences between group means are larger than what within-group noise alone would produce by chance — which is exactly the question a mean-comparison test needs answered, even though the calculation itself is entirely about variances.
Worked Example: Building an ANOVA Table
Illustrative example, not a real study or dataset. Suppose fifteen participants are split into three equal groups (n = 5 each) and scored on some continuous outcome:
| Group A | Group B | Group C |
|---|---|---|
| 4, 6, 8, 5, 7 | 6, 8, 10, 9, 7 | 8, 10, 12, 11, 9 |
| Mean = 6 | Mean = 8 | Mean = 10 |
Grand mean across all 15 scores = 120 / 15 = 8.
Between-groups sum of squares (SSB) = Σ ni(meani − grand mean)² = 5(6−8)² + 5(8−8)² + 5(10−8)² = 20 + 0 + 20 = 40.
Within-groups sum of squares (SSW) = sum of each observation’s squared deviation from its own group mean, summed within each group and then across groups. Group A: (−2)²+0²+2²+(−1)²+1² = 10. Group B: 10. Group C: 10. SSW = 10 + 10 + 10 = 30.
SST = SSB + SSW = 40 + 30 = 70 (confirmed directly by summing every observation’s squared deviation from the grand mean of 8, which also totals 70).
| Source | SS | df | MS | F |
|---|---|---|---|---|
| Between groups | 40 | k−1 = 2 | 40/2 = 20 | 20 / 2.5 = 8.00 |
| Within groups | 30 | N−k = 12 | 30/12 = 2.5 | — |
| Total | 70 | N−1 = 14 | — | — |
With F(2, 12) = 8.00, the associated p-value is approximately .006 — well below the conventional .05 threshold, so this illustrative result would be reported as statistically significant. (dfbetween = number of groups minus one; dfwithin = total N minus number of groups.)
Types of ANOVA
One-way ANOVA tests one categorical factor with three or more levels (e.g., three teaching methods) against one continuous outcome. It is the simplest case and the one worked above.
Two-way (factorial) ANOVA tests two categorical factors at once — for example, treatment condition and sex — and estimates three things: the main effect of each factor on its own, and the interaction effect between them. An interaction means the effect of one factor depends on the level of the other: for example, a treatment might work well for one sex but not the other. A significant interaction changes how the main effects should be interpreted, since a “main effect” averaged across levels of the other factor can be misleading when the two factors genuinely interact. Designs with more than two factors are called three-way, four-way, and so on, following the same logic.
Repeated-measures ANOVA is used when the same participants are measured under multiple conditions or at multiple time points, rather than different participants being assigned to different groups. Because repeated observations on the same person are correlated, this design carries an additional assumption: sphericity, meaning the variances of the differences between every pair of related conditions should be roughly equal. Mauchly’s test checks this assumption; when it is violated, the standard fix is a Greenhouse-Geisser correction (or the less conservative Huynh-Feldt correction), which adjusts the degrees of freedom downward to keep the test’s Type I error rate accurate rather than inflated.
MANOVA (multivariate analysis of variance) extends ANOVA to two or more continuous outcome variables analyzed together rather than one at a time, using statistics such as Wilks’ Lambda. It is appropriate when the outcomes are conceptually related and testing them separately would itself inflate the familywise error rate — the same logic that motivates ANOVA over separate t-tests in the first place.
ANCOVA (analysis of covariance) extends ANOVA by statistically controlling for one or more continuous covariates — a pre-test score or age, for example — before testing the group effect, which can increase statistical power by removing variance attributable to the covariate. See regression analysis for the broader modeling framework ANCOVA draws on.
Assumptions, and What to Do When They Fail
Standard (fixed-effects) ANOVA rests on three assumptions:
- Independence of observations — each data point is independent of every other; this is a design property, not something a statistical test can check after the fact.
- Approximate normality of the residuals within each group. See normality of distribution for how this is assessed. ANOVA is reasonably robust to moderate normality violations, especially with larger, roughly equal-sized groups.
- Homogeneity of variance (homoscedasticity) — the groups should have approximately equal variances. This is commonly checked with Levene’s test, which tests the null hypothesis that group variances are equal.
When variances are clearly unequal (Levene’s test is significant, or one group’s spread is visibly much larger than another’s), the standard fix is Welch’s ANOVA, a variant that does not assume equal variances and adjusts the degrees of freedom accordingly. It is increasingly recommended as a more robust default even when variances look roughly similar, since it loses little power when the assumption does hold.
When normality is badly violated, sample sizes are small, or the outcome is only ordinal (ranked) rather than truly continuous, the standard alternative is not a correction to ANOVA at all but a different, rank-based test: the Kruskal-Wallis test, covered next.
The Kruskal-Wallis Test: ANOVA’s Nonparametric Alternative
The Kruskal-Wallis test (sometimes called the Kruskal-Wallis H test, or Kruskal-Wallis one-way analysis of variance by ranks) is the nonparametric counterpart to one-way ANOVA. It answers a related but not identical question to ANOVA’s: rather than comparing group means directly, it tests whether the distributions of the groups differ, formally by comparing the sums of ranks across groups after the entire dataset (all groups combined) is ranked from lowest to highest.
This ranking step is the key mechanical difference from ANOVA. Kruskal-Wallis discards the raw magnitude of each observation and works only with its rank position, which is what makes it robust to outliers and to non-normal, skewed, or ordinal data where the mean is not even a meaningful summary. Its test statistic, H, is compared against a chi-square distribution with k−1 degrees of freedom (k = number of groups) to obtain a p-value.
Use Kruskal-Wallis instead of ANOVA when: the outcome is ordinal rather than interval/ratio; the sample is small and clearly non-normal; there are influential outliers that would distort group means; or a Levene’s test / visual inspection shows badly unequal variances that Welch’s ANOVA does not adequately address. The trade-off is a loss of statistical power relative to ANOVA when ANOVA’s assumptions genuinely do hold — Kruskal-Wallis should not be used as a default in place of ANOVA when the data are well-behaved.
A significant Kruskal-Wallis result, like a significant ANOVA F-test, only establishes that some group differs from the others in the combined ranking — it does not identify which pairs. The standard post-hoc procedure is Dunn’s test, a rank-based pairwise comparison with its own multiple-comparison correction (commonly Bonferroni-adjusted), analogous to running Tukey’s HSD or Bonferroni-corrected pairwise tests after a significant ANOVA.
Post-Hoc Tests: A Significant F Says Some Groups Differ, Not Which Ones
This is the step most often skipped or done incorrectly. A significant omnibus F-test (or Kruskal-Wallis H-test) tells you only that not all group means (or distributions) are equal — it does not tell you which specific groups differ from which. Identifying that requires a dedicated post-hoc test, chosen for the comparison structure at hand:
- Tukey’s HSD (honestly significant difference) — the standard choice for comparing every possible pair of groups when group sizes are equal or close to equal; controls the familywise error rate across all pairwise comparisons.
- Bonferroni correction — divides alpha by the number of comparisons being made; simple and broadly applicable, but conservative (more likely to miss real differences) as the number of comparisons grows.
- Scheffé’s method — the most conservative common option, but flexible: it can test complex contrasts beyond simple pairwise comparisons (e.g., the average of two groups against a third), not just pairs.
- Dunnett’s test — used specifically when every group is being compared back to a single reference or control group, rather than to each other; more powerful than Tukey or Bonferroni for that specific comparison structure because it runs fewer tests.
The choice matters: reporting only the omnibus F-test and then informally describing which groups “looked different,” without naming the post-hoc procedure and how multiplicity was controlled, reintroduces exactly the familywise error problem ANOVA was designed to solve in the first place.
Effect Size: Why F and p Alone Are Not Enough
A significant F-test and its p-value say whether an effect is statistically detectable, not how large it is. Reporting F and p without an accompanying effect size is incomplete by current reporting standards, since statistical significance is heavily influenced by sample size and says nothing on its own about practical importance — see statistical significance vs. clinical significance. The standard ANOVA effect sizes are:
- Eta-squared (η²) = SSB / SST — the proportion of total variance attributable to the group factor. In the worked example above, η² = 40 / 70 ≈ .57, a very large effect (by convention, roughly .01 = small, .06 = medium, .14 = large for eta-squared). It tends to run somewhat inflated, especially in small samples or designs with multiple factors.
- Partial eta-squared (η²p) = SSeffect / (SSeffect + SSerror) — the standard measure in factorial and repeated-measures designs, since it isolates one factor’s share of variance after removing the variance attributable to other factors, rather than dividing by the full SST as plain eta-squared does. In a one-way design with only one factor, partial eta-squared and eta-squared are numerically identical.
- Omega-squared (ω²) = (SSB − dfbetween × MSW) / (SST + MSW) — a less biased estimate of the population effect size than eta-squared, since it corrects for the tendency of eta-squared to overestimate the true effect, particularly in smaller samples. In the worked example, ω² = (40 − 2 × 2.5) / (70 + 2.5) = 35 / 72.5 ≈ .48.
How to Report ANOVA Results
The conventional (APA-style) reporting format is:
F(dfbetween, dfwithin) = [F-value], p = [p-value], [effect size]
Applied to the worked example above: “A one-way ANOVA revealed a significant effect of group on the outcome, F(2, 12) = 8.00, p = .006, η² = .57.” A complete report also states, or references, the post-hoc test used to locate which specific groups differed and how it corrected for multiple comparisons; for Welch’s ANOVA, report the Welch-adjusted (typically non-integer) degrees of freedom, since they differ from the standard formula; for Kruskal-Wallis, report H, its degrees of freedom, and p, e.g. H(2) = [H-value], p = [p-value].
Common Errors
- Running a series of t-tests instead of ANOVA when comparing three or more groups — inflates the familywise error rate rather than controlling it.
- Reading a significant F as “all groups differ from each other.” It only establishes that at least one difference exists somewhere among the groups; only a post-hoc test identifies which pairs.
- Skipping post-hoc tests entirely, or describing group differences informally without naming which correction was used.
- Ignoring assumption violations — running standard ANOVA on badly unequal variances or clearly non-normal, small-sample data instead of switching to Welch’s ANOVA or Kruskal-Wallis.
- Confusing statistical significance with practical or clinical significance — a very large sample can make a trivially small group difference statistically significant; effect size, not the p-value alone, indicates whether the difference is meaningful.
Frequently Asked Questions
What is the difference between ANOVA and a t-test?
A t-test compares the means of exactly two groups. ANOVA compares the means of three or more groups in a single omnibus test, avoiding the inflated false-positive rate that would result from running multiple t-tests on the same data. Run on exactly two groups, ANOVA and a t-test produce mathematically equivalent results (F equals t-squared).
What does a significant ANOVA result actually mean?
It means the evidence is inconsistent with all group means being equal — that is, at least one group differs from at least one other somewhere in the comparison. It does not, by itself, identify which specific groups differ; that requires a post-hoc test.
When should I use Kruskal-Wallis instead of ANOVA?
When the outcome is ordinal rather than interval/ratio, the sample is small and clearly non-normal, there are influential outliers, or variance is badly unequal across groups in a way Welch’s ANOVA does not adequately address. When ANOVA’s assumptions are reasonably well met, ANOVA has more statistical power and is the better default.
What is a “good” F-value in ANOVA?
There is no fixed threshold in isolation — whether an F-value is “large” depends on its degrees of freedom, which set the comparison distribution. What matters is the resulting p-value against your chosen alpha (conventionally .05), together with the effect size, not the raw F-value on its own.
Can ANOVA be used to analyze data from just two groups?
Mechanically yes, but it is not standard practice: with exactly two groups, a t-test is the conventional choice and reports the identical underlying comparison more directly. ANOVA is intended for three or more groups.
What is the difference between eta-squared and omega-squared?
Both estimate the proportion of variance in the outcome attributable to the group factor. Eta-squared is simpler to compute but tends to overestimate the true population effect, especially in small samples; omega-squared applies a correction for that bias and is generally considered the more accurate estimate.
