Kolmogorov-Smirnov Test: One- vs Two-Sample

Q: What is the difference between the one-sample and two-sample Kolmogorov-Smirnov test?

The one-sample test compares one sample against a fully specified theoretical distribution — named, with all parameters fixed in advance. The two-sample test compares two samples against each other with no theoretical distribution involved. The practical difference is that the one-sample test can be invalidated by estimating parameters from the data, and the two-sample test cannot, because it has no parameters to estimate.

Q: Can I use the KS test to check normality?

Only with the Lilliefors correction, and even then Shapiro-Wilk and Anderson-Darling are more powerful. The uncorrected version — the one you get from ks.test(x, "pnorm", mean(x), sd(x)) — rejected genuinely normal data 0.01% of the time in the 10,000-replicate simulation above, against a nominal 5%, and detected an obviously skewed distribution only 15.7% of the time. It is not a normality test in any useful sense.

Q: What does “Lilliefors Significance Correction” mean in SPSS output?

It means SPSS estimated the mean and SD from your data and then compared the resulting D statistic against the Lilliefors reference distribution rather than the Kolmogorov one. It is a note that the correct adjustment has already been applied, not a warning about your data. The SPSS Explore column is the corrected test; the equivalent in R is nortest::lillie.test(x) , not ks.test .

Q: Why does the KS test give a huge p-value when my data are clearly not normal?

Almost always because the parameters were estimated from the same sample. Fitting the reference distribution to the data shrinks D by construction, but the software compares that shrunken D against a reference distribution built for an un-fitted statistic, so the p-value comes out far too large. Rerun with Lilliefors or Anderson-Darling. If the parameters really were pre-specified, then a large p-value is a genuine result — though for a small sample it mostly reflects low power.

Q: Is the Kolmogorov-Smirnov test sensitive to the tails?

No, and this is its main structural weakness. The statistic is the maximum unweighted difference between two CDFs, and the variance of that difference is largest at the median and approaches zero in the tails, so the maximum is usually found near the centre. Anderson-Darling weights the deviation by the reciprocal of that variance and is the standard choice when tail behaviour is what matters. Against a heavy-tailed t(5) alternative at n = 100, the KS-family test detected the departure 33% of the time against Anderson-Darling’s 48%.

Q: What do I do about ties in a KS test?

Recognise that the classical assumptions no longer hold. Ties cannot occur under a continuous distribution, so their presence means the reference distribution for D is not the one being used. R warns on ties in the one-sample case and cannot compute an exact p-value at all when they are present; it can still compute an exact two-sample p-value with ties. If the data are genuinely discrete, use a chi-square goodness-of-fit test or a permutation test instead. If they are continuous data that were rounded, R’s documentation notes the test may be approximately valid but that “even modest amounts of rounding can have a significant effect on the calculated statistic” — report the ties either way.

Q: Does a significant two-sample KS test mean the medians differ?

No. It means the distributions differ somewhere — in location, spread, shape, or any combination. Two distributions with identical medians and different variances can produce a highly significant D. If your hypothesis concerns central tendency specifically, a rank-sum test answers that question directly and with more power against that particular alternative.

Q: How large a sample does the KS test need?

There is no clean threshold, and the more useful framing is what the test can do at a given n rather than whether it is “allowed”. Below roughly n = 20 the one-sample test has very little power against anything but gross departures, so a non-significant result carries almost no information. Above a few hundred it will detect departures too small to affect any downstream procedure. R switches from exact to asymptotic p-values at n = 100 for the one-sample test and at n 1 n 2 = 10,000 for the two-sample test, so those are the points at which the reported p-value changes in character even if the statistic does not.

Q: Is Anderson-Darling always better than Kolmogorov-Smirnov?

For testing normality with estimated parameters, essentially yes — it was more powerful than Lilliefors against both alternatives simulated above, and the nortest documentation states the Lilliefors test “is known to perform worse” than Anderson-Darling and Cramer-von Mises. But the two-sample KS test has no Anderson-Darling equivalent in base R, and for a fully pre-specified reference distribution the one-sample KS test is exactly valid and perfectly usable. “Better” applies to the composite normality case, not to the test in general.

Two different tests share the name “Kolmogorov-Smirnov.” The one-sample test (Kolmogorov) asks whether one sample came from one fully specified distribution. The two-sample test (Smirnov) asks whether two samples came from the same, unspecified, distribution. They use a similar-looking statistic, they are called by the same function in most software, and they have different validity conditions. Confusing them is the single most common way the test gets misused.

The condition that gets broken most often is on the one-sample side: the reference distribution’s parameters must be specified in advance, not estimated from the same data. Estimating the mean and standard deviation from your sample and then running a KS test against that normal distribution does not produce a slightly wrong p-value. In the simulation below it produces a test that rejects truly-normal data 0.01% of the time instead of 5%, and detects an obviously skewed distribution only 15.7% of the time where the corrected version manages 83%. The Lilliefors correction exists precisely to fix this, and SPSS’s “Lilliefors Significance Correction” footnote is the software telling you it has already applied it.

The two tests, side by side

	One-sample KS (Kolmogorov)	Two-sample KS (Smirnov)
Question	Did this sample come from this specific distribution?	Did these two samples come from the same distribution?
Null hypothesis	F = F₀, where F₀ is named and its parameters are fixed in advance (a simple null)	F₁ = F₂, with neither distribution named (a composite null, but no parameters are estimated)
Statistic	D = sup_x \|F_n(x) − F₀(x)\|	D = sup_x \|F_n1(x) − F_n2(x)\|
Parameters estimated from the data?	Not allowed. Doing it invalidates the reference distribution of D	None to estimate. This is why the two-sample test is the more robust of the pair
Common misuse	Testing normality with `mean(x)` and `sd(x)` plugged in	Reading a significant result as “the medians differ”
R call	`ks.test(x, "pnorm", 100, 15)`	`ks.test(x, y)`

Both tests compare cumulative distribution functions. The empirical CDF, F_n(x), is simply the proportion of observations at or below x — a step function that rises by 1/n at each data point. The KS statistic D is the largest vertical gap between two such curves (or between one curve and a theoretical CDF). Everything else about the test is a question of what the sampling distribution of that gap looks like under the null — and that is exactly what estimating parameters destroys.

The one-sample test: what “fully specified” really means

R’s own documentation for ks.test states the requirement flatly:

“If a one-sample test is used, the parameters specified in ... must be pre-specified and not estimated from the data. There is some more refined distribution theory for the KS test with estimated parameters (see Durbin 1973), but that is not implemented in ks.test.”

— R 4.6.1, ?ks.test, Details

The reason is that D’s null distribution — the Kolmogorov distribution, which is what turns D into a p-value — is distribution-free only when F₀ is fixed before you look at the data. Once you fit the mean and SD to the sample, the fitted curve is pulled toward the data by construction. D shrinks. But the software still compares that shrunken D against the reference distribution for an un-fitted D, which was built on the assumption that no such shrinkage happened.

There is a second, quieter consequence of “fully specified” that trips people up in the opposite direction. Because F₀ is fixed, the one-sample test is testing location and scale and shape simultaneously. A sample that is beautifully normal but centred somewhere other than the hypothesised mean will fail the test — correctly, because that is the null it was given. “Is this normal?” and “is this N(100, 15)?” are different questions, and the plain one-sample KS test only answers the second.

The same fifty numbers, two answers

The following was run in R 4.6.1 with set.seed(4102). Fifty observations were drawn from a genuine N(100, 15); the sample happened to land at a mean of 95.71 with an SD of 14.40. Three tests were then run on that one vector.

x <- rnorm(50, mean = 100, sd = 15)

ks.test(x, "pnorm", 100, 15)            # parameters specified in advance
ks.test(x, "pnorm", mean(x), sd(x))     # parameters estimated from x
nortest::lillie.test(x)                 # Lilliefors correction

PART 1  n=50 draw from N(100, 15);  mean(x)=95.7053 sd(x)=14.4017
  specified  N(100,15):   D = 0.17978   p = 0.0693
  estimated  N(xbar,s):   D = 0.08176   p = 0.8648
  Lilliefors (same D):    D = 0.08176   p = 0.5525

Three things in that output are worth reading carefully.

D collapsed from 0.180 to 0.082 when the parameters were fitted. That is the shrinkage described above, on real numbers. The fitted normal is closer to the data than the true generating normal, because it was chosen to be.
The specified test returned p = 0.069 — a near-miss, and an honest one: this sample really did drift from N(100, 15). The test is not broken; it is answering the question it was asked.
Lilliefors returns p = 0.5525 from the identical statistic that the uncorrected call reported as p = 0.8648. The D values match to five decimal places. Only the reference distribution differs — and that alone moves the p-value by more than 0.3.

The nortest package documents this equivalence explicitly, and it is the clearest one-sentence statement of the problem in any software manual:

“Although the test statistic obtained from lillie.test(x) is the same as that obtained from ks.test(x, "pnorm", mean(x), sd(x)), it is not correct to use the p-value from the latter for the composite hypothesis of normality (mean and variance unknown), since the distribution of the test statistic is different when the parameters are estimated.”

— nortest 1.0.4, ?lillie.test, Note

How badly wrong: 10,000 replicates

A single dataset shows the mechanism but not the magnitude. The following simulation draws 10,000 fresh samples of n = 50 from a true N(100, 15) — so every rejection is a false positive by construction — and records how often each test rejects at α = 0.05. A correctly calibrated test should sit at 0.05.

PART 2  rejection rate at alpha=0.05, 10,000 replicates of TRULY normal data, n=50
  KS, parameters specified :  0.0517
  KS, parameters estimated :  0.0001
  Lilliefors               :  0.0507
  Shapiro-Wilk             :  0.0522
  Anderson-Darling         :  0.0491

One in ten thousand. The uncorrected estimated-parameter version rejected once in 10,000 samples where a calibrated test rejects about 500 times. It is not marginally conservative; it is roughly 500-fold conservative at this sample size. Every other test in the table lands within simulation noise of 0.05, including the properly specified KS test — confirming that the KS statistic itself is fine and the fault lies entirely in pairing an estimated-parameter D with an unestimated-parameter reference distribution.

Extreme conservatism is easy to shrug off — a test that never rejects sounds safe. It is not, because a test that cannot make a Type I error also cannot make a discovery. The same 10,000-replicate machinery, pointed at an unmistakably non-normal distribution (chi-square with 3 degrees of freedom, strongly right-skewed), shows what that conservatism costs:

PART 3  power at alpha=0.05, 10,000 replicates from chi-square(3) (skewed), n=50
  KS, parameters estimated :  0.1565
  Lilliefors               :  0.8299
  Shapiro-Wilk             :  0.9887
  Anderson-Darling         :  0.9626

The uncorrected version misses a blatantly skewed distribution five times out of six. Lilliefors, computing the same statistic and differing only in the reference distribution it compares it to, catches it five times out of six. That gap — 15.7% versus 83.0% — is the entire practical content of the Lilliefors correction, and it is why running the uncorrected version and reporting “p > 0.05, normality assumption satisfied” is close to reporting nothing at all.

What SPSS’s “Lilliefors Significance Correction” note actually means

In SPSS, Analyze → Descriptive Statistics → Explore with “Normality plots with tests” requested produces a “Tests of Normality” table carrying a Kolmogorov-Smirnov column, a Shapiro-Wilk column, and a footnote reading “Lilliefors Significance Correction”. That footnote is not a caveat about the output — it is SPSS telling you it has already done the right thing: it estimated the mean and SD from your data (because you never gave it any), computed D, and then looked that D up in the Lilliefors reference distribution rather than the Kolmogorov one. The SPSS column is the corrected test.

The trap is on the other side. R’s ks.test(x, "pnorm", mean(x), sd(x)) is a natural-looking call that many people write while trying to reproduce the SPSS output, and it silently produces the uncorrected version — the 0.0001 row. Python’s scipy.stats.kstest behaves the same way: its documentation notes that “both tests are valid only for continuous distributions” but does not flag estimated parameters at all, so nothing in the call warns you. If you want the SPSS number in R, the call is nortest::lillie.test(x); in Python, see the software section below.

Where the KS test is genuinely weak: the tails

The KS statistic is a supremum of the unweighted vertical distance between two CDFs. That choice has a consequence that is rarely stated plainly: the variance of F_n(x) − F(x) is F(x)[1 − F(x)]/n, which is largest at F(x) = 0.5 and shrinks to zero as F(x) approaches 0 or 1. Absolute deviations are therefore biggest near the median and structurally tiny in the tails, no matter how badly the tails actually disagree. The supremum is usually attained somewhere near the centre, so that is where the test does its looking.

The Anderson-Darling statistic addresses this directly by weighting the squared deviation by 1/{F(x)[1 − F(x)]} — the exact reciprocal of that variance — which restores the tails to comparable footing. That is not a stylistic preference; it is measurable. The simulation below draws from a t distribution with 5 degrees of freedom, which is symmetric and centrally very close to a normal but has distinctly heavier tails. It is, deliberately, an alternative that lives almost entirely in the region where KS is weakest:

PART 4  power at alpha=0.05, 10,000 replicates from t(5) (heavy tails only), n=100
  Lilliefors (KS family)   :  0.3320
  Anderson-Darling         :  0.4828
  Shapiro-Wilk             :  0.5609

Even at n = 100, and even with the Lilliefors correction applied so the comparison is fair, the KS-family test detects the heavy tails about a third of the time against Anderson-Darling’s roughly one-half. Compare that against the skewed alternative in Part 3, where Lilliefors reached 83% — the same test family, similar sample size, very different performance depending on where the departure sits. The nortest manual states the ranking without qualification: the Lilliefors test “is known to perform worse” than Anderson-Darling and Cramer-von Mises.

The practical rule: if what you care about is tail behaviour — extreme values, risk modelling, whether a fitted distribution will behave sensibly at the 99th percentile — the KS test is the wrong instrument and Anderson-Darling is the standard replacement. If what you care about is a gross shift or a differently-shaped middle, KS is adequate.

Ties and discrete data break the classical assumptions

Both KS tests are built on continuous distributions, in which the probability of any two observations being exactly equal is zero. Real data are rounded, recorded to two decimal places, collected on Likert scales, or counted. Every one of those produces ties, and ties are not a cosmetic problem — they change the null distribution of D. R’s documentation again:

“The presence of ties always generates a warning in the one-sample case, as continuous distributions do not generate them. If the ties arose from rounding the tests may be approximately valid, but even modest amounts of rounding can have a significant effect on the calculated statistic.”

— R 4.6.1, ?ks.test, Details

The observable effect on a real comparison: two samples of 60 from normal distributions separated by 0.6 SD, tested as drawn and then tested again after rounding both to integers.

PART 5  two-sample KS, continuous data, n1=n2=60, shift of 0.6 SD
  D = 0.36667   p = 0.00056
  after rounding both samples to integers (ties present):
  D = 0.35000   p = 0.00021   [R emits: ties should not be present]
  distinct values after rounding: 7 of 120 observations

Rounding compressed 120 observations onto 7 distinct values — and the statistic still moved, in this case producing a smaller p-value than the untied data. The direction is not the point; the point is that D is now being read against a reference distribution built for a situation that no longer holds. For genuinely discrete data, use a test designed for it: the chi-square goodness-of-fit test for counts against expected proportions, or an exact/permutation approach. R will compute an exact two-sample p-value with ties present (via the Schroer and Trenkler algorithm), but exact one-sample p-values are unavailable whenever ties exist.

The two-sample test: the more defensible of the pair

The two-sample test escapes the estimated-parameter problem entirely, because it never names a distribution and therefore never estimates one. It compares two empirical CDFs against each other, and under the null that both samples come from the same continuous distribution, the null distribution of D depends only on n₁ and n₂. This is the version of the test with the fewest ways to go wrong.

Its real hazards are interpretive rather than technical.

It is an omnibus test, not a location test

A significant two-sample KS result says the two distributions differ somewhere. It does not say the medians differ, the means differ, or in which direction anything moved. Two samples with identical medians but very different spreads can produce a highly significant D. If your research question is specifically about a shift in central tendency, the Wilcoxon signed-rank test (paired) or its rank-sum counterpart, or the Kruskal-Wallis test for three or more groups, tests that question directly and with more power against that specific alternative. Use KS when “do these differ in any way at all?” is genuinely the question — comparing a simulated distribution against an observed one, checking whether two measurement batches are exchangeable, testing whether a covariate distribution matches between arms.

The one-sided alternatives point the opposite way to `t.test`

This is a documented trap and it catches experienced users. From R’s documentation:

“Thus in the two-sample case alternative = "greater" includes distributions for which x is stochastically smaller than y (the CDF of x lies above and hence to the left of that for y), in contrast to t.test or wilcox.test.”

— R 4.6.1, ?ks.test, Details

The alternative names refer to the CDF, not to the values. A CDF that sits higher corresponds to values that are smaller. If you write a one-sided KS test by analogy with a one-sided t-test, you will get the direction backwards. Two-sided is the safe default, and is what almost every published KS test should be.

Exact versus asymptotic p-values

R computes an exact p-value automatically when the sample size is under 100 in the one-sample case and there are no ties, and when the product of the two sample sizes is under 10,000 in the two-sample case (with or without ties). Otherwise it falls back to the asymptotic Kolmogorov distribution, whose approximation, in R’s words, “may be inaccurate in small samples.” On the Part 5 data the two agree closely:

PART 6  exact vs asymptotic two-sample p-value, same data as PART 5
  exact       p = 0.00056
  asymptotic  p = 0.00063

The difference is negligible here because n₁n₂ = 3,600 is comfortable. It is not negligible at n = 8 versus n = 10, where the exact distribution of D is visibly lumpy and the asymptotic approximation is poor. Report which one you used, and if the software chose for you, say what it chose.

The sample-size problem applies here too

Everything above concerns whether the KS test computes a valid p-value. A separate question is whether a normality test should be gating your analysis at all — and the answer, on the evidence, is usually no. Every null-hypothesis test of exact normality, KS and Lilliefors included, becomes near-certain to reject at large n (where trivial departures are still detectable and the Central Limit Theorem has already made the downstream test robust) and near-powerless at small n (where real departures matter most and go undetected). That argument, with a full simulation of Shapiro-Wilk rejection rates climbing from 13.6% to 100% while the t-test’s Type I error stays flat at 5%, is set out in detail in how to check the normal distribution assumption. It is not repeated here, but it applies to the one-sample KS test unchanged — and it means that fixing the Lilliefors problem gives you a correct answer to a question that was probably the wrong one.

Where the KS test earns its place is the situation it was actually designed for: a genuinely pre-specified reference distribution. Checking that a random number generator produces uniform output. Verifying that simulated data match the theoretical distribution they were meant to come from. Testing p-values from a null simulation against Uniform(0,1). In each of those, F₀ is fixed by theory before the data exist, nothing is estimated, and the one-sample KS test is exactly correct.

Running it: R, Python, SPSS

R

# Two-sample: the safe default
ks.test(group_a, group_b)

# One-sample, parameters genuinely pre-specified (e.g. testing a simulation)
ks.test(x, "punif", 0, 1)
ks.test(x, "pnorm", 100, 15)

# Composite normality (mean and SD unknown) -- do NOT use ks.test here
nortest::lillie.test(x)     # Lilliefors: the KS statistic, correct reference distribution
nortest::ad.test(x)         # Anderson-Darling: better against tail departures
shapiro.test(x)             # Shapiro-Wilk: generally the most powerful of the three

Note that ks.test also accepts a formula interface, ks.test(value ~ group, data = df), where the right-hand side is a two-level factor.

Python (SciPy)

from scipy import stats

stats.ks_2samp(a, b)                          # two-sample
stats.kstest(x, "uniform")                    # one-sample, pre-specified
stats.kstest(x, "norm", args=(100, 15))       # one-sample, pre-specified

# Composite hypothesis with parameters estimated from the data:
stats.goodness_of_fit(stats.norm, x, statistic="ad")   # or "ks", "cvm", "filliben"

scipy.stats.goodness_of_fit is the correct Python answer to the estimated-parameter problem and is much less widely known than it should be. SciPy’s documentation describes it as “a Monte Carlo test in which parameters that specify the distribution from which samples are drawn have been estimated from the data” — it fits the remaining parameters, then builds the null distribution of the chosen statistic by simulation rather than assuming one. Because the null distribution is generated for your specific situation, it sidesteps the whole problem instead of correcting for it, and it accepts the Anderson-Darling, Kolmogorov-Smirnov, Cramer-von Mises and Filliben statistics. It is slower than a table lookup, which is the only reason not to use it.

Plain scipy.stats.kstest takes method="auto" by default and will choose among "exact", "approx" and "asymp"; set it explicitly if the choice matters to your reporting.

SPSS

Two different procedures produce KS output and they are not interchangeable. Analyze → Descriptive Statistics → Explore gives the Lilliefors-corrected one-sample test for normality (the footnoted column described above). Analyze → Nonparametric Tests offers both a one-sample KS against a named distribution and a two-independent-samples KS; the one-sample version there lets you supply distribution parameters or use sample estimates, and choosing sample estimates without the correction reintroduces exactly the conservatism measured in Part 2. If your goal is a normality check, use Explore.

Reporting a KS test

A reportable KS result names which KS test was run and what the reference was. The minimum set:

Which test — one-sample or two-sample. “A Kolmogorov-Smirnov test” alone is ambiguous.
The reference distribution and where its parameters came from, for one-sample tests. “Against N(0,1)” and “against a normal with mean and SD estimated from the sample, with the Lilliefors correction” are different tests and must be distinguished.
Whether a correction was applied. If SPSS produced it, say Lilliefors-corrected. If R produced it via ks.test with fitted parameters, that is uncorrected — and should be rerun.
D, both sample sizes, and the p-value, with the p-value given to a sensible precision rather than as “p < 0.05”.
Exact or asymptotic, where the software offered a choice or made one for you.
Ties, if the data are rounded or discrete. Report that they were present and that the p-value is approximate.

A worked sentence, using Part 5: “A two-sided two-sample Kolmogorov-Smirnov test found the distributions differed (D = 0.367, n₁ = n₂ = 60, exact p = .0006). As an omnibus test this indicates a difference in distribution rather than specifically in central tendency.”

And for a normality check, if you run one at all: “Normality was assessed with the Lilliefors-corrected Kolmogorov-Smirnov test (D = 0.082, n = 50, p = .55) alongside a Q-Q plot.” Reporting a non-significant result as “the data were normal” overstates it in every case — failing to reject a null hypothesis is not evidence for it.

Which test to actually use

Your question	Use	Why not KS
Does this sample come from a distribution fixed by theory, with no fitted parameters?	One-sample KS	— this is its correct use
Do these two samples come from the same distribution, in any respect?	Two-sample KS	— this is its correct use
Are these data normal, with mean and SD unknown?	Shapiro-Wilk, or Anderson-Darling; Lilliefors if you want the KS statistic	Plain KS with fitted parameters is invalid; see Parts 2 and 3
Do the tails depart from the assumed distribution?	Anderson-Darling	KS weights the centre and is structurally blind to the tails
Do two groups differ in central tendency?	Mann-Whitney/Wilcoxon rank-sum, or a t-test	KS is omnibus and less powerful against a pure location shift
Do observed counts match expected proportions?	Chi-square goodness-of-fit	KS assumes continuity; ties invalidate the reference distribution
Should I gate a t-test or ANOVA on a normality test at all?	Usually no — inspect a Q-Q plot, consider the sample size	See the normality assumption guide

Frequently asked questions

What is the difference between the one-sample and two-sample Kolmogorov-Smirnov test?

The one-sample test compares one sample against a fully specified theoretical distribution — named, with all parameters fixed in advance. The two-sample test compares two samples against each other with no theoretical distribution involved. The practical difference is that the one-sample test can be invalidated by estimating parameters from the data, and the two-sample test cannot, because it has no parameters to estimate.

Can I use the KS test to check normality?

Only with the Lilliefors correction, and even then Shapiro-Wilk and Anderson-Darling are more powerful. The uncorrected version — the one you get from ks.test(x, "pnorm", mean(x), sd(x)) — rejected genuinely normal data 0.01% of the time in the 10,000-replicate simulation above, against a nominal 5%, and detected an obviously skewed distribution only 15.7% of the time. It is not a normality test in any useful sense.

What does “Lilliefors Significance Correction” mean in SPSS output?

It means SPSS estimated the mean and SD from your data and then compared the resulting D statistic against the Lilliefors reference distribution rather than the Kolmogorov one. It is a note that the correct adjustment has already been applied, not a warning about your data. The SPSS Explore column is the corrected test; the equivalent in R is nortest::lillie.test(x), not ks.test.

Why does the KS test give a huge p-value when my data are clearly not normal?

Almost always because the parameters were estimated from the same sample. Fitting the reference distribution to the data shrinks D by construction, but the software compares that shrunken D against a reference distribution built for an un-fitted statistic, so the p-value comes out far too large. Rerun with Lilliefors or Anderson-Darling. If the parameters really were pre-specified, then a large p-value is a genuine result — though for a small sample it mostly reflects low power.

Is the Kolmogorov-Smirnov test sensitive to the tails?

No, and this is its main structural weakness. The statistic is the maximum unweighted difference between two CDFs, and the variance of that difference is largest at the median and approaches zero in the tails, so the maximum is usually found near the centre. Anderson-Darling weights the deviation by the reciprocal of that variance and is the standard choice when tail behaviour is what matters. Against a heavy-tailed t(5) alternative at n = 100, the KS-family test detected the departure 33% of the time against Anderson-Darling’s 48%.

What do I do about ties in a KS test?

Recognise that the classical assumptions no longer hold. Ties cannot occur under a continuous distribution, so their presence means the reference distribution for D is not the one being used. R warns on ties in the one-sample case and cannot compute an exact p-value at all when they are present; it can still compute an exact two-sample p-value with ties. If the data are genuinely discrete, use a chi-square goodness-of-fit test or a permutation test instead. If they are continuous data that were rounded, R’s documentation notes the test may be approximately valid but that “even modest amounts of rounding can have a significant effect on the calculated statistic” — report the ties either way.

Does a significant two-sample KS test mean the medians differ?

No. It means the distributions differ somewhere — in location, spread, shape, or any combination. Two distributions with identical medians and different variances can produce a highly significant D. If your hypothesis concerns central tendency specifically, a rank-sum test answers that question directly and with more power against that particular alternative.

How large a sample does the KS test need?

There is no clean threshold, and the more useful framing is what the test can do at a given n rather than whether it is “allowed”. Below roughly n = 20 the one-sample test has very little power against anything but gross departures, so a non-significant result carries almost no information. Above a few hundred it will detect departures too small to affect any downstream procedure. R switches from exact to asymptotic p-values at n = 100 for the one-sample test and at n₁n₂ = 10,000 for the two-sample test, so those are the points at which the reported p-value changes in character even if the statistic does not.

Is Anderson-Darling always better than Kolmogorov-Smirnov?

For testing normality with estimated parameters, essentially yes — it was more powerful than Lilliefors against both alternatives simulated above, and the nortest documentation states the Lilliefors test “is known to perform worse” than Anderson-Darling and Cramer-von Mises. But the two-sample KS test has no Anderson-Darling equivalent in base R, and for a fully pre-specified reference distribution the one-sample KS test is exactly valid and perfectly usable. “Better” applies to the composite normality case, not to the test in general.

Reproducing the simulations

All figures on this page come from a single R script run under R 4.6.1 with nortest 1.0.4 and set.seed(4102). Each rejection-rate figure is 10,000 independent replicates at α = 0.05. The structure, for anyone wanting to rerun or extend it:

library(nortest)
set.seed(4102)

R <- 10000; n <- 50; alpha <- 0.05
p_est <- p_lil <- numeric(R)
for (i in seq_len(R)) {
  z <- rnorm(n, 100, 15)                                  # a TRUE null
  p_est[i] <- ks.test(z, "pnorm", mean(z), sd(z))$p.value  # uncorrected
  p_lil[i] <- lillie.test(z)$p.value                       # corrected
}
mean(p_est < alpha)   # 0.0001
mean(p_lil < alpha)   # 0.0507

Swap rnorm(n, 100, 15) for rchisq(n, df = 3) to reproduce the power comparison in Part 3, or for rt(n, df = 5) at n = 100 to reproduce the tail comparison in Part 4. Monte Carlo error on a 10,000-replicate proportion near 0.05 is roughly ±0.004, so the differences reported above are far larger than simulation noise; the 0.0001 figure is a single rejection and should be read as “essentially never” rather than as a precise rate.

Normality of distribution: how to check the normal distribution assumption — why normality pre-testing is a questionable gate at all, with the sample-size simulation
Cumulative distribution function (CDF) — the object both KS tests compare
Chi-square test: independence, goodness-of-fit, assumptions — the goodness-of-fit test for discrete data
The Wilcoxon signed-rank test — when the question is location, not distribution
The Kruskal-Wallis test — the rank-based alternative for three or more groups
How to read a residual plot — the assumption check that usually matters more than normality
Bootstrapping in statistics — resampling when no reference distribution fits
Statistical test — the controlled-vocabulary definition
Research methods — the full cluster