Negative Binomial Distribution: Formula, Worked Example and Interpretation

The negative binomial distribution gives the probability of seeing exactly k failures before the r-th success in a series of independent yes/no trials that all share the same success probability. This guide gives the formula, one worked example with every step of the arithmetic, a convention table (the main source of confusion), and checked R and Python calls.

Negative binomial distribution: definition in one paragraph

Run independent Bernoulli trials, each succeeding with probability p, and stop at a pre-set number of successes, r. The number of failures you see along the way follows a negative binomial distribution. SciPy’s documentation describes this as a sequence of i.i.d. Bernoulli trials repeated “until a predefined, non-random number of successes occurs”, and R’s manual likewise counts “the number of failures which occur in a sequence of Bernoulli trials” before the target number of successes. With r = 1 it reduces to the geometric waiting-time case (one success, count the failures). It is also widely used for overdispersed counts, where the variance exceeds the mean, which is the situation the Poisson distribution cannot handle.

Negative binomial distribution formula

Using the failures-counted convention that R and SciPy implement, the probability mass function is:

P(X = k) = C(k + r – 1, k) · p^r · (1 – p)^k, for k = 0, 1, 2, …

• k: the number of failures observed before the stopping point.
• r: the target number of successes (called n in SciPy and size in R).
• p: the probability of success on each trial, with 0 < p ≤ 1.
• C(k + r – 1, k): the number of ways to arrange the k failures among the first k + r – 1 trials, because the last trial must be the r-th success.

SciPy writes the same function as C(k + n – 1, n – 1) pⁿ (1 – p)^k; the two binomial coefficients are equal. R’s manual gives it in gamma-function form, Γ(x + n) / (Γ(n) x!) pⁿ (1 – p)^x, which also allows non-integer n.

Formula for negative binomial distribution: mean and variance

R’s manual gives the mean as n(1 – p)/p and the variance as n(1 – p)/p². In this guide’s symbols: mean = r(1 – p)/p and variance = r(1 – p)/p². Dividing, variance / mean = 1/p. Because 1/p is at least 1, the variance is never smaller than the mean, and it is strictly larger whenever p < 1.

Negative binomial distribution examples: a worked calculation

The scenario below is a hypothetical illustration, not data from a study. A lab repeats a cell-culture attempt, and each independent attempt succeeds with probability p = 0.4. The team stops after r = 3 successful cultures. What is the probability of exactly k = 5 failed attempts before the third success?

1. Count arrangements: C(k + r – 1, k) = C(7, 5) = 7! / (5! · 2!) = 21.
2. Success factor: p^r = 0.4³ = 0.064.
3. Failure factor: (1 – p)^k = 0.6⁵ = 0.07776.
4. Multiply: 21 × 0.064 × 0.07776 = 0.10451 (about 0.1045, or 10.5%).

The mean number of failures is 3 × 0.6 / 0.4 = 4.5, and the variance is 3 × 0.6 / 0.4² = 1.8 / 0.16 = 11.25. The variance-to-mean ratio is 11.25 / 4.5 = 2.5, which equals 1/p, as the formula predicts.

The first nine probabilities from the same calculation (rounded to four places) show the right-skewed shape:

Failures k	0	1	2	3	4	5	6	7	8
P(X = k)	0.0640	0.1152	0.1382	0.1382	0.1244	0.1045	0.0836	0.0645	0.0484

The cumulative probability of at most 5 failures is 0.6846, so the chance of needing 8 or more failures is 1 – P(X ≤ 7) = 0.1673. Setting r = 1 gives the geometric case: P(X = 2) = 0.4 × 0.6² = 0.144.

Two conventions: failures counted versus trials counted

Textbooks disagree on what the random variable counts. Check this before comparing any formula, table or software output.

Convention	What X counts	Smallest value	Mean	Used by
Failures counted	Failures before the r-th success	0	r(1 – p)/p	R dnbinom, SciPy nbinom
Trials counted	Total trials up to and including the r-th success	r	r/p	Alternative definition in some texts; check your textbook

The two are linked by trials = failures + r. In the worked example, 5 failures before the third success is the same event as 8 total trials, with the same probability, 0.1045.

Mean-based (size and mu) parameterization

R documents an alternative in terms of the mean mu: prob = size / (size + mu) and variance = mu + mu²/size. SciPy gives the matching conversion p = n / (n + mu). With size = 3 and mu = 4.5 this returns prob = 3 / 7.5 = 0.4 and variance = 4.5 + 20.25 / 3 = 11.25, the same distribution as the example. R’s manual also notes this is a gamma mixture of Poisson distributions. This is the form used in regression, covered in negative binomial regression for overdispersion.

Interpretation: reading the result

Quantity	How to read it
P(X = k)	Probability of exactly k failures before the r-th success (0.1045 in the example).
P(X ≤ k)	Cumulative probability, from pnbinom or nbinom.cdf (0.6846 for k = 5).
Mean r(1 – p)/p	Expected number of failures before the target is reached (4.5).
Variance / mean = 1/p	Always at least 1; the lower the success probability, the more spread relative to the mean (2.5 here).
r = 1	Geometric case: failures before the first success.

Assumptions and common mistakes

• Independent trials with a constant p. The formula assumes every trial has the same success probability and does not depend on earlier results.
• A fixed, non-random target r. The stopping rule is “stop at the r-th success”, not “stop after a fixed number of trials” (that is the ordinary binomial distribution).
• Mixing conventions. Plugging a trials-counted value into a failures-counted function shifts the answer by r. Subtract r first.
• Passing mu where prob is expected. In R, supply either prob or mu, not the mean in the prob slot; in SciPy convert with p = n / (n + mu).
• Using Poisson when the variance exceeds the mean. The Poisson variance equals its mean; if the sample variance is clearly larger, the negative binomial is the usual next candidate, and a quasi-Poisson versus negative binomial comparison helps choose between fixes.

How to report it

Report the model’s form and the fitted values so a reader can reproduce the probabilities. A template, using the hypothetical example (adapt the formatting to your journal or style guide):

The number of failed culture attempts before the third success was modeled with a negative binomial distribution (failures-counted parameterization) with r = 3 and p = .40. The expected number of failures was 4.5 (SD = 3.35), and the probability of exactly five failures was .10.

State the convention, the parameter values, whether they were fixed by design or estimated from data, and the software and function used. See also the guides on standard deviation and the incidence rate ratio for related reporting of count data.

How to calculate it in R or Python

R

dnbinom(5, size = 3, prob = 0.4)    # P(X = 5)  = 0.1045
pnbinom(5, size = 3, prob = 0.4)    # P(X <= 5) = 0.6846
dnbinom(5, size = 3, mu = 4.5)      # same distribution, mean form

Python (SciPy)

from scipy.stats import nbinom
n, p = 3, 0.4
nbinom.pmf(5, n, p)                 # 0.1045
nbinom.cdf(5, n, p)                 # 0.6846
nbinom.mean(n, p), nbinom.var(n, p) # 4.5, 11.25

Both functions use the failures-counted convention, so k is the number of failures. For fitting negative binomial regression in software, see Poisson and negative binomial regression in Stata.

Frequently asked questions

What is the formula for the negative binomial distribution?

P(X = k) = C(k + r – 1, k) p^r (1 – p)^k, where k is the number of failures before the r-th success and p is the success probability. The mean is r(1 – p)/p and the variance is r(1 – p)/p².

How is it different from the binomial distribution?

The binomial fixes the number of trials and counts successes. The negative binomial fixes the number of successes and counts the failures (or trials) needed to reach it.

How is it related to the Poisson distribution?

Both model counts, but the Poisson has variance equal to its mean, while the negative binomial has variance mu + mu²/size, which is larger. R’s manual describes it as a gamma mixture of Poisson distributions.

Is the geometric distribution a special case?

Yes. With r = 1 the probability mass function becomes p(1 – p)^k, the geometric distribution of failures before the first success.

Sources

• SciPy documentation, scipy.stats.nbinom. Read 2026-10-29.
• R Documentation, The Negative Binomial Distribution (stats::NegBinomial). Read 2026-10-29.