Direct comparison
Planned Comparisons vs Post-Hoc Tests
Planned comparisons are set before seeing the data; post-hoc tests explore after. What must be pre-specified, and the correction each one requires.
Written and maintained by CASRAI Editorial Board
Last updated
Ask CASRAI · included with Regulatory Radar
Ask about Planned Comparisons vs Post-Hoc Tests
Ask CASRAI answers research-administration questions and cites the passages behind every claim — and says so when the corpus does not cover something, instead of guessing. It comes with a Regulatory Radar subscription at $29 a month, alongside the daily digest of regulatory changes and the dashboard of what changed.
150 questions a day, on this site, over the API, or inside your own tools through the CASRAI MCP server.
Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays free to read, with no account and no card.
How do Planned Comparisons, Post-Hoc Tests compare side by side?
The table below compares Planned Comparisons, Post-Hoc Tests across 9 procurement-relevant dimensions, from what it is through where it belongs in the paper.
Side-by-side comparison
| Dimension | Planned Comparisons | Post-Hoc Tests |
|---|---|---|
| What it is | A specific contrast among group means, specified before the data are collected, following from a real hypothesis. | An exploratory comparison chosen after seeing the results, typically to follow up a significant omnibus test. |
| When it is decided | Before data collection -- ideally documented in a statistical analysis plan or pre-registration. | After the data have been examined, often after a significant omnibus ANOVA F-test. |
| Typical number of comparisons | Small and bounded -- up to k-1 orthogonal contrasts for k groups. | Can be the entire pairwise set -- k(k-1)/2 comparisons for k groups. |
| Requires a significant omnibus F first? | No -- a genuine a priori contrast is licensed by the hypothesis, not by the omnibus result (this is what separates it from Fisher’s protected LSD). | Usually yes -- most post-hoc procedures assume a significant omnibus test already flagged that some difference exists. |
| Typical correction method | Bonferroni or Šidák across the small pre-specified set; sometimes none, for very few truly orthogonal contrasts. | Tukey HSD (all pairwise), Scheffé (any contrast, including data-suggested ones), or Dunnett (versus a single control). |
| Statistical power | Higher -- fewer, targeted comparisons cost less of the family-wise error budget. | Lower for a given comparison -- correcting across the full comparison space is more conservative. |
| Misuse risk | Labeling a comparison "planned" after already seeing the data -- a form of HARKing that understates the real multiplicity. | Reporting a cherry-picked pairwise result without correcting for the full comparison space actually tested. |
| Representative test | Planned orthogonal contrasts (e.g. linear/quadratic trend contrasts), tested via a t- or F-test on the contrast. | Tukey HSD for all-pairwise comparisons; Scheffé for the most general, most conservative case. |
| Where it belongs in the paper | Stated as a hypothesis in Methods / the analysis plan, before results are reported. | Reported in Results, explicitly labeled exploratory or post-hoc. |
Common questions
Common questions about Planned Comparisons vs Post-Hoc Tests
Does a planned comparison still need a significant ANOVA first?
+
No -- that is actually the test of whether a comparison is genuinely planned. A true a priori contrast is licensed by the hypothesis, not by the omnibus F reaching significance; requiring significance first is Fisher’s protected LSD logic, a separate and more conservative rule.
Can I call a comparison "planned" if I only decided to test it after seeing a trend in the data?
+
No. If the decision to test it happened after looking at the results, it is post-hoc by definition, regardless of how plausible it seems in hindsight. Presenting it as planned without a dated prior record is a form of HARKing.
What has to be written down beforehand for a contrast to count as planned?
+
The specific groups being compared and the predicted direction of the effect, recorded in a statistical analysis plan or pre-registration dated before the data were analyzed -- not merely before the study was designed.
Which post-hoc correction should I use?
+
It depends on the comparison set: Tukey HSD for every pairwise difference among group means from one ANOVA; Dunnett when every group is compared only to a single control; Scheffé for testing any contrast, including data-suggested ones, at the cost of substantially lower power.
