The Friedman Test and Its Post-Hoc Procedures: Nemenyi vs. Wilcoxon-Bonferroni

A significant Friedman test tells you that at least one of your repeated conditions differs from the others — it does not tell you which ones. That follow-up question is where most write-ups of this test stop short: they explain the test statistic and then leave the reader to guess at a post-hoc procedure, or worse, run pairwise comparisons with no correction at all. This guide covers both halves — when the Friedman test is the right tool, and the two post-hoc procedures (the Nemenyi test and pairwise Wilcoxon signed-rank tests with a Bonferroni correction) that a significant result actually requires.

What the Friedman test actually does

The Friedman test (Friedman, 1937) is the nonparametric alternative to a one-way repeated-measures ANOVA. It applies to a within-subject design: the same set of subjects (or matched blocks) is measured under three or more related conditions, and you want to know whether the conditions differ. Instead of working with the raw scores, the test ranks each subject’s scores across the conditions — separately for every subject — then checks whether the resulting rank sums for each condition differ more than chance alone would produce.

That within-subject ranking is the whole mechanism. If every condition were equivalent, each subject’s ranks would be randomly assigned among 1 through k, and every condition’s rank sum, averaged across subjects, would converge on the same value. The Friedman statistic measures how far the observed rank sums deviate from that null expectation.

When to reach for it instead of repeated-measures ANOVA

Use the Friedman test when your design is within-subject (each unit contributes a score to every condition) and at least one of the following holds:

• The dependent variable is ordinal rather than interval/ratio (e.g., a rating scale), so a mean is not a meaningful summary.
• The differences between conditions are not normally distributed, and your sample is too small for the central limit theorem to rescue a repeated-measures ANOVA.
• Mauchly’s test flags a sphericity violation severe enough that even a Greenhouse-Geisser or Huynh-Feldt correction feels like a patch on a model that was never appropriate for the data in the first place.
• The data contain outliers that would distort condition means but not condition ranks.

Two boundary cases matter. With exactly two related conditions, don’t use Friedman — the Wilcoxon signed-rank test is the correct two-condition tool and Friedman reduces to a less informative version of it. And if your design compares independent groups rather than repeated measures on the same subjects, you want the Kruskal-Wallis test, Friedman’s between-subjects counterpart, not this one.

The test statistic, step by step

With N subjects and k related conditions, rank each subject’s k scores from 1 (lowest) to k (highest), tied scores sharing the average rank. Sum the ranks for each condition across all subjects to get R_j for condition j. The Friedman statistic is:

χ²_F = [12 / (Nk(k+1))] × ΣR_j² − 3N(k+1)

Under the null hypothesis of no difference between conditions, χ²_F approximately follows a chi-square distribution with k − 1 degrees of freedom. That chi-square approximation is known to be conservative for small N or small k, which is why Iman and Davenport (1980) proposed an F-distributed alternative that most modern statistical software computes alongside it:

F = [(N − 1) × χ²_F] / [N(k − 1) − χ²_F], with df₁ = k − 1, df₂ = (k − 1)(N − 1)

The F-approximation rejects the null more readily than the chi-square version at the same nominal alpha, particularly when N is small relative to k — use whichever your software reports, but know the two can disagree at borderline p-values.

Worked example

Eight lab technicians each ran the same assay under three protocol versions (A, B, C), and completion time in minutes was recorded for each technician under each protocol — a within-subject design with N = 8 and k = 3. This is an illustrative dataset, not a reported study.

Technician	Protocol A (min)	Protocol B (min)	Protocol C (min)	Ranks (A, B, C)
1	42	38	45	2, 1, 3
2	37	34	39	2, 1, 3
3	50	48	53	2, 1, 3
4	29	31	33	1, 2, 3
5	44	40	46	2, 1, 3
6	35	33	38	2, 1, 3
7	41	44	47	1, 2, 3
8	33	30	36	2, 1, 3
Rank sum Rj	14	10	24	—

With N = 8, k = 3, and rank sums 14, 10, and 24: ΣR_j² = 14² + 10² + 24² = 872. Substituting into the formula above:

χ²_F = [12 / (8 × 3 × 4)] × 872 − (3 × 8 × 4) = 13.0, df = 2

Because a chi-square distribution with 2 degrees of freedom has the closed-form CDF 1 − e^−x/2, the exact p-value here is e^−13/2 ≈ 0.0015 — comfortably below any conventional alpha. The Iman-Davenport F-approximation agrees: F(2, 14) = 30.3, also far past its critical value. The Friedman test rejects the null of no difference among the three protocols. That is where most treatments of this test stop — but rejecting the omnibus null only says some protocol differs from some other protocol, not which pair.

A significant result is the start, not the finish

Exactly as with a significant one-way ANOVA or a significant Kruskal-Wallis test, a significant Friedman χ²_F is an omnibus result: it tells you the k conditions are not all equal somewhere, and nothing about where. Reporting a significant Friedman test alone, with no follow-up comparison, is incomplete — and running uncorrected pairwise tests instead inflates the family-wise Type I error rate: with k = 3 conditions there are k(k−1)/2 = 3 pairwise comparisons, and with k = 6 there are 15, each carrying its own chance of a false positive if left uncorrected. Two procedures handle this correctly.

Post-hoc option 1: the Nemenyi test

The Nemenyi test (Nemenyi, 1963) is the standard all-pairs post-hoc for a significant Friedman result. It compares every pair of conditions’ average ranks against a single critical difference (CD):

CD = q_α × √[k(k+1) / (6N)]

where q_α is a critical value from the studentized range distribution, tabulated by the number of conditions being compared. Two conditions are declared significantly different if their average-rank difference exceeds the CD. This is a single threshold applied to every pair at once, which keeps the family-wise error rate controlled without a separate correction step — but it comes at a real cost: the Nemenyi test is conservative, especially with more than four or five conditions, and can fail to detect pairwise differences that a targeted comparison would catch.

Continuing the worked example: average ranks were A = 1.75, B = 1.25, C = 3.00 (each rank sum divided by N = 8). With k = 3 conditions, q_0.05 ≈ 2.343, giving CD = 2.343 × √(12/48) = 2.343 × 0.5 ≈ 1.17. Comparing each pair’s average-rank difference to that threshold: A vs. B differs by 0.50 (below CD — not significant), A vs. C differs by 1.25 (above CD — significant), and B vs. C differs by 1.75 (above CD — significant). Protocol C is significantly slower than both A and B; A and B are not distinguishable from each other at this sample size.

Post-hoc option 2: pairwise Wilcoxon signed-rank tests with a Bonferroni correction

The alternative is to run a Wilcoxon signed-rank test on each pair of conditions separately, then apply a Bonferroni correction to control the family-wise error rate across however many pairs you ran. With k conditions there are k(k−1)/2 comparisons, so the per-comparison significance threshold becomes alpha / [k(k−1)/2] — for the three-protocol example above, that’s 0.05 / 3 ≈ 0.0167, not the usual 0.05.

This route is generally the more powerful of the two when you’re only interested in a specific subset of comparisons rather than every possible pair — the Bonferroni denominator only has to account for the comparisons you actually ran, whereas the Nemenyi CD is fixed by the full k. It is also the more common choice in applied practice, because most statistical software (R’s pairwise.wilcox.test(), SPSS’s post-hoc pairwise comparisons after a nonparametric related-samples test) implements it directly. Some packages instead offer Conover’s test as a third option — a rank-based procedure similar in spirit to Nemenyi’s but generally less conservative; if your software offers it, it is a reasonable substitute for either of the two covered here, though it is used less often in the published literature than Nemenyi or Wilcoxon-Bonferroni.

Which one to use

Neither procedure is universally correct; the choice depends on what you’re asking:

• Use Nemenyi when you want every possible pairwise comparison and a single, simple threshold to apply to all of them — common when reporting a full comparison matrix, e.g. comparing several algorithms or conditions against each other exhaustively.
• Use Wilcoxon with Bonferroni when you have specific, planned comparisons in mind (e.g., each treatment vs. a control condition, not every treatment vs. every other treatment) — restricting the correction to only the comparisons you actually need preserves more statistical power than Nemenyi’s blanket threshold.
• If you ran every possible pair anyway, expect the two methods to agree in most cases but not always at borderline p-values — they use different distributional assumptions (studentized range vs. Wilcoxon’s rank-sum) and Nemenyi’s CD is generally the more conservative of the two as k grows.

Effect size: report Kendall’s W alongside the test statistic

A significant Friedman result does not indicate how large the effect is, only that it’s unlikely to be chance. Kendall’s coefficient of concordance (Kendall’s W) is the standard companion effect size, computed directly from the same statistic:

W = χ²_F / [N(k − 1)]

W ranges from 0 (no agreement in ranking across subjects) to 1 (every subject ranked the conditions identically). Conventional benchmarks treat W < 0.3 as weak agreement, 0.3–0.5 as moderate, and above 0.5 as strong — treat these as rough orientation, not a substitute for reporting the actual value alongside your effect size. For the worked example, W = 13 / (8 × 2) = 0.81, indicating strong, consistent agreement among technicians about the relative ordering of the three protocols.

How to report a Friedman test with post-hoc results

A complete report states the omnibus result, the post-hoc method and correction, and which specific pairs differed — not just “p < .05”:

A Friedman test indicated a significant difference in completion time across the three assay protocols, χ²_F(2) = 13.00, p = .002, N = 8, Kendall’s W = .81. Post-hoc pairwise comparisons using the Nemenyi test (critical difference = 1.17 at α = .05) indicated that Protocol C required significantly more time than both Protocol A (average rank difference = 1.25) and Protocol B (average rank difference = 1.75); Protocols A and B did not differ significantly from each other (average rank difference = 0.50).

See how to report p-values for the general formatting conventions this follows, and always name the specific post-hoc procedure and correction used — “post-hoc tests were conducted” without naming which one is not reproducible.

Related checks before you commit to Friedman

• If your design has only two related conditions, use the Wilcoxon signed-rank test directly — Friedman with k = 2 is a less informative version of it.
• If your groups are independent rather than repeated measures on the same subjects, use the Kruskal-Wallis test instead.
• If your dependent variable is binary rather than ranked/continuous and you have exactly two related conditions, McNemar’s test is the correct tool, not Friedman.
• Before defaulting to the nonparametric route, confirm the parametric alternative actually fails: check normality and run Mauchly’s test for sphericity on the repeated-measures ANOVA first. Friedman trades some statistical power for fewer assumptions — it’s the right trade only when those assumptions genuinely don’t hold.
• If your repeated measures are collected on a Likert scale, Friedman (not repeated-measures ANOVA) is usually the appropriate omnibus test, for the same ordinal-data reason covered above.

Frequently asked questions

Is the Friedman test the same as a repeated-measures ANOVA?

No. Repeated-measures ANOVA analyzes the raw scores and assumes sphericity and (for small samples) normally distributed differences; the Friedman test analyzes only the within-subject ranks and makes no distributional assumption about the raw scores. Friedman is the nonparametric alternative used when repeated-measures ANOVA’s assumptions don’t hold.

What’s the difference between the Friedman test and the Kruskal-Wallis test?

Both are rank-based omnibus tests for three or more conditions, but Friedman is for related samples (the same subjects measured under every condition) and Kruskal-Wallis is for independent samples (different subjects in each condition). Using the wrong one ignores the correlation structure in your data.

Do I need a post-hoc test if the Friedman test isn’t significant?

No. If the omnibus Friedman test does not reach significance, there is no statistical basis for a post-hoc comparison — running one anyway (and reporting whichever pair happens to look different) is a form of Type I error inflation sometimes called “fishing,” and it defeats the purpose of running the omnibus test first.

Should I use the Nemenyi test or Wilcoxon signed-rank tests with a Bonferroni correction?

Use Nemenyi for an exhaustive all-pairs comparison with one simple threshold; use Wilcoxon-Bonferroni when you have a smaller, planned set of comparisons, since restricting the Bonferroni correction to only those comparisons preserves more power than Nemenyi’s blanket threshold across every possible pair.

What effect size should I report with a Friedman test?

Kendall’s coefficient of concordance (Kendall’s W), computed as χ²_F divided by N(k − 1). It ranges from 0 to 1 and describes how consistently subjects ranked the conditions.

How many subjects do I need for a Friedman test?

There’s no universal minimum, but power is driven by both N (subjects) and k (conditions) together, and the chi-square approximation to the test statistic is known to be conservative when both are small — run a power analysis for your specific N and k rather than relying on a rule of thumb, and prefer the Iman-Davenport F-approximation over the plain chi-square version when your sample is on the small side.