Direct comparison
Planned Comparisons vs Post-Hoc Tests
Planned comparisons are set before seeing the data; post-hoc tests explore after. What must be pre-specified, and the correction each one requires.
Written and maintained by CASRAI Editorial Board
Last updated
Ask CASRAI · free to try
Ask about Planned Comparisons vs Post-Hoc Tests
Ask your first 2 questions free below. Subscribers get 150 a day for $29 a month.
An AI assistant specialized in research administration. It cites the sources behind every answer, labels web answers and says when it can't answer.
Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.
Works on this site and inside Claude, Cursor and the AI tools you already use.
Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays free to read, with no account and no card.
How do Planned Comparisons, Post-Hoc Tests compare side by side?
The table below compares Planned Comparisons, Post-Hoc Tests across 9 procurement-relevant dimensions, from what it is through where it belongs in the paper.
Side-by-side comparison
| Dimension | Planned Comparisons | Post-Hoc Tests |
|---|---|---|
| What it is | A specific contrast among group means, specified before the data are collected, following from a real hypothesis. | An exploratory comparison chosen after seeing the results, typically to follow up a significant omnibus test. |
| When it is decided | Before data collection -- ideally documented in a statistical analysis plan or pre-registration. | After the data have been examined, often after a significant omnibus ANOVA F-test. |
| Typical number of comparisons | Small and bounded -- up to k-1 orthogonal contrasts for k groups. | Can be the entire pairwise set -- k(k-1)/2 comparisons for k groups. |
| Requires a significant omnibus F first? | No -- a genuine a priori contrast is licensed by the hypothesis, not by the omnibus result (this is what separates it from Fisher’s protected LSD). | Usually yes -- most post-hoc procedures assume a significant omnibus test already flagged that some difference exists. |
| Typical correction method | Bonferroni or Šidák across the small pre-specified set; sometimes none, for very few truly orthogonal contrasts. | Tukey HSD (all pairwise), Scheffé (any contrast, including data-suggested ones), or Dunnett (versus a single control). |
| Statistical power | Higher -- fewer, targeted comparisons cost less of the family-wise error budget. | Lower for a given comparison -- correcting across the full comparison space is more conservative. |
| Misuse risk | Labeling a comparison "planned" after already seeing the data -- a form of HARKing that understates the real multiplicity. | Reporting a cherry-picked pairwise result without correcting for the full comparison space actually tested. |
| Representative test | Planned orthogonal contrasts (e.g. linear/quadratic trend contrasts), tested via a t- or F-test on the contrast. | Tukey HSD for all-pairwise comparisons; Scheffé for the most general, most conservative case. |
| Where it belongs in the paper | Stated as a hypothesis in Methods / the analysis plan, before results are reported. | Reported in Results, explicitly labeled exploratory or post-hoc. |
Common questions
Common questions about Planned Comparisons vs Post-Hoc Tests
Does a planned comparison still need a significant ANOVA first?
+
No -- that is actually the test of whether a comparison is genuinely planned. A true a priori contrast is licensed by the hypothesis, not by the omnibus F reaching significance; requiring significance first is Fisher’s protected LSD logic, a separate and more conservative rule.
Can I call a comparison "planned" if I only decided to test it after seeing a trend in the data?
+
No. If the decision to test it happened after looking at the results, it is post-hoc by definition, regardless of how plausible it seems in hindsight. Presenting it as planned without a dated prior record is a form of HARKing.
What has to be written down beforehand for a contrast to count as planned?
+
The specific groups being compared and the predicted direction of the effect, recorded in a statistical analysis plan or pre-registration dated before the data were analyzed -- not merely before the study was designed.
Which post-hoc correction should I use?
+
It depends on the comparison set: Tukey HSD for every pairwise difference among group means from one ANOVA; Dunnett when every group is compared only to a single control; Scheffé for testing any contrast, including data-suggested ones, at the cost of substantially lower power.
