Written and maintained by CASRAI Editorial Board
Last updated
A t-test’s output is never just a p-value. Every real t-test result — a software printout, a summary table, a sentence in a results section — carries at least five numbers that only mean something read together: the t-statistic, the degrees of freedom (df), the p-value, the confidence interval around the mean difference, and an effect size. Reading only the p-value and skipping the rest produces two specific, avoidable errors: treating a small p-value as if it measures how large the effect is, and reporting a “significant” result without noticing the confidence interval is too wide to say much of anything. This guide walks through what each number in the block actually tells you, in the order worth reading them, and closes with a worked example turning the output into an honest reporting sentence.
For what a t-test itself does and which of the three designs — one-sample, independent-samples, paired — applies to your data, see CASRAI’s guide to the t-test. This guide assumes the output is already in front of you, from SPSS, R, Stata, Excel, or a hand calculation, and covers reading it, not producing it — for the software-specific mechanics, see running a t-test in SPSS, in R, or in Stata.
The full output block, piece by piece
Software labels these differently, but a t-test’s output always contains the same underlying quantities. A typical independent-samples result looks something like t(58) = 2.14, p = .037, 95% CI [0.08, 2.61], d = 0.56, alongside a row deciding whether equal variances were assumed. Read as a table, each piece answers a different question:
| Element | Example | What it actually tells you |
|---|---|---|
| t-statistic | t = 2.14 | The observed difference expressed relative to its standard error — a signal-to-noise ratio, not a difference in the outcome’s original units. |
| Degrees of freedom (df) | df = 58 | Which t-distribution the p-value is drawn from; tracks sample size, adjusted for the design. See CASRAI’s guide to degrees of freedom. |
| p-value | p = .037 | The probability of a t this extreme, or more extreme, if the null hypothesis were exactly true. Not the probability the null is true, and not a measure of how large the effect is. |
| 95% CI of the mean difference | [0.08, 2.61] | A range of plausible values for the true difference given this data — usually the single most informative number in the block, and the one most reports skip. |
| Effect size (Cohen’s d) | d = 0.56 | The size of the difference in standardized units, largely independent of sample size. |
| Equal-variances row | Levene’s p = .41 (not significant) | Decides which of the printed t/df/p rows — equal variances assumed, or not assumed (Welch’s) — is the one to actually cite. |
t and df: a signal-to-noise ratio, not a difference score
The t-statistic is the estimated difference divided by its standard error. A larger absolute value of t means the difference is large relative to the noise in the estimate — it does not mean the difference itself is large in real terms. A t of 4.5 on a trivially small difference (because the standard error happened to be even smaller) is entirely possible, and common in large samples. Degrees of freedom track sample size, adjusted for the design: for a standard independent-samples t-test they run roughly n₁ + n₂ − 2, but under Welch’s correction for unequal variances they’re computed via the Welch–Satterthwaite approximation and can come out as a non-integer — seeing df = 54.3 on your printout is normal, not an error. Neither number, on its own, says anything about whether the difference matters practically; both exist to produce the p-value and the confidence interval that follow.
The p-value: what .037 actually claims
A p-value of .037 states that, if there were truly no difference in the population, a difference this large or larger would occur about 3.7% of the time from sampling variation alone. It does not state the probability that the null hypothesis is true, and — the trap most specific to t-test output — it does not measure the size of the effect. The American Statistical Association’s 2016 consensus statement on p-values is explicit on this point: p-values do not measure the size of an effect or the importance of a result (Wasserstein & Lazar, The American Statistician, 70(2), 2016). A p-value’s magnitude is driven jointly by the effect size, the sample size, and the variance in the data — the same p = .001 can come from a large, meaningful difference in a small sample or a trivial, practically meaningless difference in a very large one. Reverse-engineering “how big” from “how small the p-value is” is not something the number supports, no matter how tempting the shorthand. For the full mechanics of what a p-value is and isn’t, see CASRAI’s guides on what a p-value is and statistical significance.
The confidence interval: the number most reports skip
The 95% confidence interval on the mean difference — [0.08, 2.61] in the running example — gives a range of plausible values for the true difference, consistent with the data. It carries two traps of its own:
- A significant result can still have a wide, uninformative interval. [0.08, 2.61] technically excludes zero, so p is under .05 — but the range of plausible true effects spans from negligible to substantial. The test “passed,” and the study still hasn’t pinned down how big the effect actually is. A narrow interval close to the point estimate is a precise result; a wide one that happens to clear zero is a lucky significant result, not a precise one.
- A non-significant interval that includes zero is not proof of no effect. A 95% CI of [−0.4, 1.9] includes zero, so the test is not significant — but it also includes 1.9, which could be a practically important effect. The honest reading is “this data can’t rule out zero, and can’t rule out a meaningful effect either,” not “there is no effect.” The width of the interval, not just whether it crosses zero, is the information worth reporting.
The APA Publication Manual (7th edition, Section 6.5) states that, for each primary outcome, an effect size and confidence interval should be reported alongside the significance test — the reporting convention exists precisely because the p-value alone underdetermines both of these questions. See CASRAI’s guides on confidence intervals and reporting confidence intervals in a manuscript for the full mechanics.
Effect size: a separate axis from significance
Cohen’s d standardizes the mean difference into units of pooled standard deviation, largely independent of sample size — which is exactly what makes it answer a different question than the p-value does. A tiny p-value can pair with a trivial d in a large enough sample; a substantial d can pair with a non-significant p in a small, underpowered one. “Is this difference distinguishable from noise” (the p-value’s job) and “how large is the difference” (the effect size’s job) are genuinely separate questions, and a t-test’s output answers both only if you read both numbers. See CASRAI’s guides on calculating Cohen’s d and interpreting Cohen’s d for field-calibrated benchmarks rather than the often-misapplied 0.2/0.5/0.8 thresholds.
Which line do you report? The equal-variances row
Independent-samples t-test output from SPSS, R, and most other packages prints two candidate rows — one assuming the two groups have equal population variances (Student’s t), one that doesn’t (Welch’s t) — each with its own t, df, and p. A variance-equality test, typically Levene’s test, sits above or beside these rows and decides which one is the result to actually cite: if Levene’s test is significant (commonly p < .05, meaning the variances differ meaningfully), report the “equal variances not assumed” (Welch’s) row; otherwise, report the “equal variances assumed” row. Citing the wrong row — or citing both without saying which one is the real result — is a common, easily avoided error once you know which line the Levene’s test result is actually gating. See CASRAI’s guide to Levene’s test in SPSS for where this row appears and what to do when it’s significant.
Putting it together: from output block to reporting sentence
Take the running example: an independent-samples t-test comparing two groups, Levene’s test not significant (equal variances assumed), t(58) = 2.14, p = .037, 95% CI [0.08, 2.61], d = 0.56. Read in sequence, this says: the difference is large enough relative to sampling noise to clear the conventional significance threshold (t and p); the plausible range for the true difference is wide but excludes zero (the CI); and the standardized size of the difference is moderate (d). Turned into a results-section sentence, following the APA reporting convention of pairing the significance test with both a confidence interval and an effect size:
“An independent-samples t-test showed that Group A scored significantly higher than Group B, t(58) = 2.14, p = .037, 95% CI [0.08, 2.61], d = 0.56.”
Every number in that sentence is doing distinct work — none of them, alone, would let a reader reconstruct what the other four say. See CASRAI’s guides on reporting p-values and reporting confidence intervals for the exact APA formatting conventions for each piece.
Four over-interpretation traps, in one place
- Treating p as a stand-in for effect size. A smaller p-value does not mean a bigger effect; it means more evidence against the null, which can come from a larger sample just as easily as a larger difference.
- Reading a wide-but-significant confidence interval as precise. Clearing zero and being precise are not the same claim — check the width, not just the sign.
- Reading a confidence interval that includes zero as proof of no effect. It usually means the data lacks the precision to rule zero out, not that the true effect is zero.
- Reporting the equal-variances row when Levene’s test says not to, or vice versa. The variance-equality test exists specifically to select which t/df/p triple is the real result.
Frequently asked questions
Does a smaller p-value mean a bigger effect?
No. A p-value’s size reflects the combination of effect size, sample size, and variability — not the effect size alone. The same small p-value can come from a large effect in a small sample or a trivial effect in a very large one. The ASA’s 2016 statement on p-values states this directly: p-values do not measure the size of an effect. Check the effect size and confidence interval separately.
What does it mean if the 95% confidence interval includes zero?
It means the data cannot rule out a true difference of zero at that confidence level — not that the true difference is zero. A wide interval that includes both zero and a practically meaningful value reflects an underpowered or noisy estimate more often than it reflects a genuinely null effect.
Which t-test line should I report — equal variances assumed or not assumed?
Check the variance-equality test (Levene’s test in most software) printed alongside the t-test. If it’s significant, the variances differ meaningfully, so report the “equal variances not assumed” (Welch’s) row. If it’s not significant, report the “equal variances assumed” row.
Is a higher t-value always more important than a lower one?
Not on its own. A larger |t| means the observed difference is larger relative to its standard error, which can happen because the difference is genuinely large or because the sample is large enough to shrink the standard error. Pair it with the effect size to know which.
My t-test is significant, but Cohen’s d is small — what should I write?
Report both numbers honestly rather than leading with only the significant one. A statistically detectable difference and a practically meaningful one are different claims; a small d alongside a significant p is common in large samples and should be described as such, not framed as a strong effect.
