ANOVA (Analysis of Variance)

When ANOVA is the right test

ANOVA tests whether the means of a continuous outcome differ across three or more groups, or across combinations of two or more categorical factors. It answers a narrower question than it is sometimes given credit for: it tells you whether at least one group mean differs from the others somewhere among the groups being compared — it does not, by itself, tell you which specific pairs differ. That is what post-hoc tests are for (see below).

ANOVA is the wrong tool in two common situations authors run into when drafting a Results section:

• Only two groups. Comparing exactly one pair of means calls for an independent-samples or paired t-test, not ANOVA. Running ANOVA on two groups produces a result mathematically equivalent to a t-test’s, but reporting it as ANOVA is not standard practice when there is a single pairwise comparison.
• A categorical outcome. If the outcome itself is categorical — remission yes/no, treatment success/failure — the comparison across groups is a comparison of proportions, which calls for a chi-square test of independence, not ANOVA. ANOVA assumes a continuous (or at least ordinal/interval) dependent variable.

The reason ANOVA exists at all, rather than authors simply running a series of pairwise t-tests across every group combination, is family-wise error rate: each individual t-test carries its own chance of a false-positive (Type I error), and running many of them on the same dataset compounds that risk well above the nominal alpha level (typically .05). A single omnibus ANOVA tests all the groups at once, holding the overall Type I error rate at the stated alpha, before any pairwise follow-up is done.

One-way, two-way, and repeated-measures ANOVA

The distinctions authors most often need to get right when describing their design:

• One-way ANOVA tests the effect of a single categorical factor (e.g., treatment arm: placebo, low-dose, high-dose) with three or more levels on one continuous outcome.
• Two-way (factorial) ANOVA tests two categorical factors simultaneously (e.g., treatment arm and sex), estimating each factor’s main effect on its own and the interaction effect between them — whether the effect of one factor depends on the level of the other. Designs with more than two factors follow the same logic and are described as three-way, four-way, and so on.
• Repeated-measures ANOVA is used when the same participants are measured under multiple conditions or at multiple time points (a within-subjects design), rather than different participants being assigned to different groups (a between-subjects design). It accounts for the correlation between repeated observations on the same participant, which a standard between-subjects ANOVA does not.
• ANCOVA (analysis of covariance) extends ANOVA by statistically controlling for one or more continuous covariates — baseline scores or age, for example — before testing the group effect.

Assumptions

Standard (fixed-effects, between-subjects) ANOVA rests on three assumptions that a Methods or supplementary section is generally expected to address, at least briefly:

• Independence of observations — each participant’s data point is independent of every other participant’s.
• Normality of the residuals (the outcome should be approximately normally distributed within each group, particularly important in smaller samples).
• Homogeneity of variances (homoscedasticity) — the groups being compared should have approximately equal variances, commonly checked with Levene’s test.

When these assumptions are clearly violated and the sample is too small for the Central Limit Theorem to compensate, the standard alternative is the Kruskal-Wallis test, a non-parametric analogue of one-way ANOVA that compares group rank distributions rather than means.

What to report in a manuscript

A complete ANOVA result in a manuscript’s Results section reports the test statistic, both degrees of freedom, the p-value, and an effect size — not the F-statistic and p-value alone. The conventional (APA-style) format is:

F(df_between, df_within) = [F-value], p = [p-value], [effect size]

For example: “A one-way ANOVA revealed a significant effect of treatment condition on pain reduction, F(2, 87) = 6.42, p = .003, η² = .13.” Here, 2 is the between-groups degrees of freedom (number of groups minus one) and 87 is the within-groups (error) degrees of freedom. The effect size — commonly eta-squared (η²) or partial eta-squared (η&sub2;p) for ANOVA — is what distinguishes a statistically detectable difference from a practically meaningful one; see Statistical Significance vs. Clinical Significance for why a manuscript needs both a p-value and an effect size, not just the former.

Post-hoc tests and multiple comparisons

A significant omnibus ANOVA result establishes that some group difference exists but not which one. Identifying which specific pairs of groups differ requires a post-hoc test run after the ANOVA, with its own correction for multiple comparisons — commonly Tukey’s HSD (honestly significant difference), Bonferroni correction, or Scheffé’s method. Reporting only the omnibus F-test and then describing which groups “differed” without naming which post-hoc procedure was used and how multiplicity was controlled is a frequent source of reviewer comments.

Related concepts

ANOVA is one of several inferential tests available once a manuscript has established what its data can and cannot claim descriptively; see Descriptive Statistics vs. Inferential Statistics for the broader distinction between summarizing a sample and generalizing beyond it.