How to Choose a Statistical Test: A Decision Tree

Every statistical-test selection question collapses into four questions, asked in order. Get these four right and the test picks itself — there is no fifth judgment call needed for the overwhelming majority of research designs. This page is a navigable decision tree that walks through them and routes to a full CASRAI guide for whichever test you land on, so it works as a single starting point for every test on the site rather than a list you have to already know how to search.

If you already know your outcome type and design, skip straight to the decision tree or the quick-reference table. If you are not sure how to characterize your data yet, start with the four questions below.

The four questions that determine your test

1. What type of outcome variable do you have?

This is the question that matters most and the one researchers most often skip. A continuous (interval or ratio) variable is one where the distance between values is meaningful and arithmetic on it makes sense — age, reaction time, blood pressure, a validated scale total treated as continuous. A categorical (nominal) variable sorts cases into unordered groups — treatment vs. control, disease present vs. absent, department affiliation. An ordinal variable has a meaningful order but not meaningful distances between levels — a five-point Likert item, a pain scale, a stage classification. See CASRAI’s guide to levels of measurement for the full distinction, including the specific mistake of treating a Likert item as interval data and running a t-test on it when an ordinal test was called for.

2. How many groups or conditions are you comparing?

One sample against a fixed benchmark, two groups or conditions, or three or more. This is usually unambiguous, but watch for a common miscount: a single group measured at three time points is a three-condition repeated-measures design, not “one group,” even though only one sample was recruited.

3. Are the groups independent or paired/related?

Independent groups contain different, unrelated cases — a treatment arm and a separate control arm, drawn from different people. Paired or related data links observations across conditions — the same subject measured before and after an intervention, matched case-control pairs, or repeated measures on the same unit. Using an independent-groups test on paired data throws away the correlation between the paired observations and typically understates statistical power; using a paired test on genuinely independent data is a smaller error but still the wrong model.

4. Does your data meet the parametric assumptions?

Parametric tests (t-tests, ANOVA, Pearson correlation) assume the outcome is approximately normally distributed within each group (or that the sampling distribution of the mean is, which large samples largely guarantee via the central limit theorem) and, for multi-group comparisons, that variances are roughly equal across groups. Check normality with a Shapiro-Wilk test, a Q-Q plot, or simply a histogram for obvious skew or outliers — see CASRAI’s guide to normality of distribution and testing normality in SPSS. Check equal variances with Levene’s test. If either assumption is clearly violated and your sample is small, the nonparametric branch of the tree below is the safer default; with a genuinely large sample, mild non-normality matters much less because of the central limit theorem, and Welch’s version of the t-test (which does not assume equal variances) is the standard default regardless.

The decision tree

Click each branch to expand it. Every leaf links to a full CASRAI guide for that test, including assumptions, how to run it in SPSS/R/Stata where a guide exists, and how to report the result.

My outcome variable is continuous (interval or ratio)

Comparing means across groups or conditions

One group, compared against a known/fixed value

Data roughly normal → One-sample t-test. Data clearly non-normal and the sample is small → one-sample Wilcoxon signed-rank test (covered alongside the paired case in CASRAI’s Wilcoxon signed-rank guide).

Two groups, independent

Parametric → Independent-samples t-test (Welch’s version by default — see running it in R, SPSS, or Stata). Non-normal or small-sample → Mann-Whitney U test (also documented for Stata’s ranksum command).

Two conditions, paired or related (same subjects, before/after, or matched pairs)

Parametric → Paired-samples t-test. Non-normal differences or small sample → Wilcoxon signed-rank test.

Three or more groups, independent

Parametric → One-way ANOVA (walkthroughs for SPSS, R, and Stata), followed by a post hoc test such as Tukey HSD (equal variances) or Games-Howell (unequal variances) to locate which pairs differ. Non-normal or small-sample → Kruskal-Wallis test.

Three or more conditions, repeated measures on the same subjects

Parametric → Repeated-measures ANOVA (check sphericity; correct with Greenhouse-Geisser or Huynh-Feldt if violated). Non-normal or small-sample → Friedman test, the nonparametric repeated-measures analogue — not yet a dedicated CASRAI guide, but the logic mirrors Kruskal-Wallis applied within-subjects.

Two independent grouping factors at once (factorial design)

→ Two-way ANOVA, which tests both main effects and their interaction in one model.

Several correlated continuous outcomes measured on the same groups

→ MANOVA, which tests group differences across the outcomes jointly rather than running a separate ANOVA per outcome and inflating the false-positive rate.

Testing the association between two continuous variables

Roughly linear relationship, both variables roughly normal → Pearson correlation (SPSS walkthrough). Non-linear-but-monotonic relationship, ordinal data, or outliers → Spearman rank correlation (covered in the same correlation coefficient guide).

Predicting or explaining a continuous outcome from one or more predictors

→ Linear regression (multiple regression in SPSS, Stata). Check multicollinearity/VIF and read the residual plot before trusting the coefficients.

My outcome variable is categorical (nominal, including binary)

Comparing proportions/frequencies across independent groups

→ Chi-square test of independence. If any expected cell count falls below 5 (common with small samples or a sparse table), use Fisher’s exact test instead — both are covered together in CASRAI’s chi-square-in-Stata guide, which documents Stata’s built-in Fisher’s exact option.

Testing whether one categorical variable matches an expected distribution

→ Chi-square goodness-of-fit test.

Paired binary outcome (the same subjects, measured before and after, or two raters on the same cases)

→ McNemar’s test, which tests for a shift in the discordant pairs rather than treating the two measurements as independent samples. Not yet a dedicated CASRAI guide; treat it as the paired-data counterpart to the chi-square test above.

Predicting a binary or categorical outcome from one or more predictors

Binary outcome → Binary logistic regression (SPSS, R, Stata). Outcome with three or more unordered categories → Multinomial logistic regression.

My outcome variable is ordinal

Ordinal outcomes with two or three-plus groups usually route to the same rank-based tests as non-normal continuous data, because both rely on ranks rather than means:

Two independent groups

→ Mann-Whitney U test.

Two paired/related conditions

→ Wilcoxon signed-rank test.

Three or more independent groups

→ Kruskal-Wallis test.

Association between two ordinal variables

→ Spearman rank correlation (see the correlation coefficient guide).

Predicting an ordinal outcome from predictors

→ Ordinal (proportional-odds) logistic regression, which extends the logic of binary logistic regression to an ordered outcome; check the proportional-odds assumption (a Brant test in R, or the SPSS PLUM procedure’s built-in Test of Parallel Lines) before trusting a single set of coefficients across all thresholds.

Quick-reference table

The same routing, flattened for skimming or for matching against a design you already know precisely.

Worked example: walking the tree end to end

To show what “running the tree” actually looks like, and to be honest about how often the parametric and non-parametric answers agree in practice rather than always producing a dramatic reversal, here is a full pass through with real (simulated) numbers, not just the routing logic.

Illustrative simulated data, not a real study. Two workflows (A: current, B: proposed) for resolving a support ticket, n = 11 tickets timed under each. To make the assumption-checking step meaningful, group A was generated from a roughly normal distribution and group B was generated from a right-skewed (lognormal) distribution — a realistic shape for time-to-complete data, where a few tickets take much longer than the rest. The dataset was produced with a seeded pseudorandom generator (mulberry32, seed 20260826) plus a Box-Muller normal transform, in a small Node.js script run for this page; every figure below is computed directly from that script’s output, not estimated or adjusted afterward.

• Group A (minutes): 25.8, 25.5, 24.9, 23.7, 21.8, 23.4, 20.2, 23.4, 21.7, 18.9, 27.6 — mean 23.36, SD 2.57, sample skewness −0.17 (essentially symmetric)
• Group B (minutes): 17.6, 27.0, 41.6, 17.5, 29.0, 25.3, 13.2, 21.2, 30.1, 13.2, 15.7 — mean 22.86, SD 8.73, sample skewness 0.89 (visibly right-skewed, exactly as generated)

Walking the tree: the outcome (minutes to resolve) is continuous → two groups → independent (different tickets in each workflow) → check the parametric assumption. Group B’s skewness of 0.89 and its markedly larger spread (SD 8.73 vs. 2.57, driven by the single 41.6-minute ticket) are exactly the pattern that should push you toward the non-parametric branch, so the tree recommends the Mann-Whitney U test here rather than the independent-samples t-test.

For illustration, both tests were actually run on this dataset:

• Welch’s independent-samples t-test (the branch the tree recommends against, run anyway for comparison): t = 0.182, df = 11.72 (Welch-Satterthwaite), two-tailed p = 0.859
• Mann-Whitney U test (the branch the tree recommends): U = 53, z = 0.493 (normal approximation, tie-corrected), two-tailed p = 0.622

Honest result, not a tidy one: both tests reach the same substantive conclusion here — no statistically significant difference between the two workflows at this sample size — despite the p-values themselves differing by a fair margin (0.86 vs. 0.62). That is a realistic outcome, not a failure of the example: most of the time, a defensible parametric and non-parametric test on the same moderate-sized dataset will agree on significance even though the exact p-value moves, because both are approximating the same underlying question. The value of picking the right branch is that the Mann-Whitney result does not depend on the mean and SD being meaningful summaries of a skewed distribution the way the t-test’s does — with a smaller sample, a single influential outlier like the 41.6-minute ticket, or a less symmetric skew, the two tests can and do disagree on statistical significance, not just on the precise p-value. Reporting whichever test the skew and sample size actually justify, rather than whichever gives the smaller p-value, is the point of using the tree at all.

Common trip points

• Treating an ordinal scale as continuous. Running a t-test or Pearson correlation on a 5-point Likert item assumes the distance between “agree” and “strongly agree” equals the distance between “neutral” and “agree” — rarely defensible. Use the ordinal branch above, or a validated multi-item scale total (which behaves more like continuous data) instead of a single item.
• Running repeated pairwise t-tests instead of ANOVA plus a post hoc test. Comparing three groups with three separate t-tests inflates the family-wise false-positive rate beyond the nominal alpha; ANOVA followed by a correction such as Tukey HSD controls it properly.
• Ignoring the paired structure of the data. Before/after measurements on the same subjects analyzed with an independent-samples test discard the correlation between the two measurements and typically understate power — use the paired branch.
• Chi-square with sparse cells. The chi-square test’s p-value is unreliable when expected cell counts fall below about 5; switch to Fisher’s exact test rather than reporting a chi-square result you cannot trust.
• Pre-testing normality to decide, then reporting only the “winning” test. Deciding between the parametric and non-parametric branch based on a significance test on the same data you are about to analyze is a known source of distorted error rates in simulation studies — decide on assumption-checking grounds (sample size, visible skew, known measurement properties) documented before you see the result, not by picking whichever test gives the smaller p-value afterward.

Frequently asked questions

What’s the real difference between a parametric and a non-parametric test?

A parametric test assumes the data follows a specific distribution (almost always the normal distribution) and estimates parameters of that distribution, such as the mean. A non-parametric test makes no distributional assumption and typically works on the ranks of the data rather than the raw values, which is what makes it robust to skew and outliers at some cost in statistical power when the parametric assumptions genuinely hold.

Can I just always use the non-parametric test to be safe?

Not without a cost. When the parametric assumptions are actually met, the parametric test has more statistical power — it is more likely to detect a real effect at the same sample size — because it uses the actual values rather than just their ranks. The decision tree exists specifically so you check the assumption rather than defaulting to the conservative option every time.

What if my outcome is a count (e.g., number of events)?

Count data is a distinct case the continuous/categorical/ordinal split above does not fully capture. For low counts or over-dispersed data, Poisson or negative-binomial regression is the standard approach rather than treating the count as continuous and running ordinary linear regression on it.

What if I have more than one outcome variable?

If the outcomes are correlated and conceptually related, MANOVA (for group comparisons) tests them jointly rather than running a separate test per outcome, which both controls the false-positive rate and can detect multivariate effects a set of univariate tests would miss.

Does sample size change which branch I should use?

Yes, in two ways. First, normality tests like Shapiro-Wilk have very little power to detect non-normality in small samples and very high power to flag trivial deviations in large ones, so a formal normality test result should never be the sole basis for the decision — look at the shape of the data too. Second, with a genuinely large sample, the central limit theorem means the sampling distribution of the mean is close to normal even if the raw data is not, which is part of why the parametric branch is the practical default once samples get large, regardless of mild skew in the raw values.

Related CASRAI guides

This page is the index for CASRAI’s quantitative-analysis content; see the Research Methods & Statistics pillar for the full cluster, including qualitative methods, sampling and power, and measurement/validity. Related building blocks: descriptive statistics, what a p-value actually means, statistical significance, effect size, and power analysis and sample size.