Statistical Test

Operational definition

A statistical test (also called a hypothesis test or significance test) is a procedure for making a decision about a population based on evidence from a sample. It answers a specific, narrow question: is the pattern seen in this sample data larger, or more extreme, than what would be expected under a stated null hypothesis of ‘no effect’ or ‘no difference’ — or is it small enough to be plausibly explained by ordinary sampling variation? Every named statistical test (t-test, ANOVA, chi-square, and dozens of others) is a specific mathematical implementation of that same general logic, built for a particular combination of data type and study design.

The general procedure has four steps regardless of which specific test is used: (1) state a null hypothesis and an alternative hypothesis; (2) choose a test appropriate to the data’s measurement type, number of groups, and design; (3) compute a test statistic from the sample and derive a p-value; (4) compare the p-value to a pre-specified significance threshold (conventionally α = 0.05) to decide whether to reject the null hypothesis. A statistical test only supports a valid inferential claim if its underlying assumptions genuinely hold for the data it is applied to — using the wrong test for the data type or design produces a p-value that does not mean what it appears to mean.

Major categories of statistical test

Parametric vs. non-parametric

Parametric tests assume the sampled data follows a known distribution — almost always the normal distribution — and make inferences using population parameters such as the mean and standard deviation. Because they use more information about the distribution, parametric tests are more statistically powerful when their assumptions hold. The t-test, ANOVA, and Pearson correlation are all parametric.

Non-parametric tests make no assumption about the shape of the underlying distribution; many work by ranking the data rather than using raw values directly. They are the appropriate choice when a continuous outcome is heavily skewed, contains outliers, is ordinal rather than truly continuous, or when the sample is too small to assess normality reliably. The trade-off is lower statistical power when a parametric test’s assumptions would have held. The Mann-Whitney U test, Wilcoxon signed-rank test, Kruskal-Wallis test, and Spearman correlation are the non-parametric counterparts to, respectively, the independent-samples t-test, the paired t-test, one-way ANOVA, and Pearson correlation.

A separate family of tests — the chi-square test and related procedures — is used specifically for categorical (count) data and does not fit neatly into the parametric/non-parametric split, since there is no continuous distribution being assumed in the first place.

One-sample, two-sample, and multi-group tests

Tests are also categorized by how many groups or samples are being compared. A one-sample test compares a single sample against a known or hypothesized population value. A two-sample test compares two groups, and can be further split into independent (unrelated participants or units in each group) or paired/repeated-measures (the same units measured twice, e.g. before and after an intervention) designs, which require different test formulas. Tests comparing three or more groups — ANOVA being the standard parametric example — require dedicated multi-group procedures rather than running repeated two-group tests, which would inflate the false-positive rate across multiple comparisons.

How to select a statistical test

Test selection is driven by three questions asked in sequence, not by familiarity or convention:

• What is the outcome’s measurement type? Continuous/interval data (e.g. reaction time, blood pressure, a scale score) opens the door to parametric tests. Categorical/nominal data (e.g. yes/no, treatment arm, disease subtype) routes to chi-square-family tests. Ordinal data (e.g. a 5-point Likert scale) is usually best handled with non-parametric tests.
• How many groups or variables, and how are they related? One sample vs. two vs. three-or-more; independent groups vs. paired/repeated observations on the same units; and whether the goal is comparing group means, testing an association between two variables, or testing agreement with an expected distribution.
• Do the data meet the assumptions of a parametric test? Approximate normality (often assessed visually or with a formal normality test), homogeneity of variance between groups, and independence of observations. When these do not hold — commonly with small samples, skewed distributions, or ordinal data — the non-parametric counterpart is the more defensible choice, even though it typically has less statistical power.

Because the correct test depends entirely on these upstream design decisions, test selection should be planned alongside the study design itself — ideally specified in advance, as part of a pre-registration or analysis plan — rather than chosen after the data have been collected and inspected, which risks selecting whichever test produces a significant result (a form of p-hacking).

Statistical tests published on CASRAI

CASRAI’s dictionary covers specific statistical tests in operational depth, each with worked examples and the assumptions that make it the right (or wrong) choice for a given dataset:

• ANOVA (Analysis of Variance) — the standard parametric test for comparing means across three or more groups or factor combinations.
• Chi-Square Test (χ²) — the standard test for categorical data, covering both the test of independence and the goodness-of-fit test.

Related inferential concepts that feed directly into interpreting any statistical test’s result: effect size (how large the observed difference or association actually is, distinct from whether it is statistically significant) and confidence interval (the range of plausible population values implied by the sample, an alternative or complement to a simple significant/not-significant verdict).

For the broader research-writing context in which test selection and reporting decisions get made and reviewed, see the scholarly writing pillar.

Frequently asked questions

What is the difference between a statistical test and a p-value?

A statistical test is the whole procedure; the p-value is one output of that procedure — specifically, the probability of observing a result at least as extreme as the one obtained, if the null hypothesis were actually true. The p-value alone does not indicate the size or practical importance of an effect, which is why it is typically reported alongside an effect size or confidence interval.

How do I know if my data should use a parametric or non-parametric test?

Check whether the outcome is continuous and approximately normally distributed (or can reasonably be assumed so, e.g. via the central limit theorem for larger samples), and whether variance is roughly similar across the groups being compared. If either assumption is doubtful — commonly with small samples, skewed data, ordinal scales, or clear outliers — the non-parametric counterpart to the intended parametric test is the safer choice.

Can I run a t-test to compare more than two groups?

Not without a correction. Running repeated pairwise t-tests across three or more groups inflates the overall false-positive rate beyond the nominal significance threshold, because each additional comparison adds its own chance of a false positive. The standard solution is a single ANOVA (or its non-parametric counterpart, the Kruskal-Wallis test), followed by a multiple-comparison-corrected post-hoc test only if the overall test is significant.