Skip to main content
v2026.11,986 entries · CC-BY 4.0

Kullback-Leibler Divergence: Formula & Example

Master Kullback-Leibler divergence: clear formulas, worked arithmetic example, interpretation table, and calculation code in Python, R, and Excel.

Written and maintained by CASRAI Editorial Board

Published

On this page

Definition in One Paragraph

Kullback-Leibler (KL) divergence, also designated relative entropy or directed divergence, is a non-negative statistical measure of the difference between an approximating probability distribution Q and a reference distribution P. Originating in information theory and mathematical statistics, it quantifies the expected excess surprise or information lost per observation when encoding outcomes from P using a code optimized for Q. Across discrete settings such as categorical distributions, KL divergence equals zero if and only if both distributions are identical and grows larger as their shapes diverge, serving as an asymmetric statistical discrepancy measure rather than a metric distance.

Ask CASRAI · free to tryQuestion about this guide? Ask CASRAI answers with linked sources. 2 free questions a day, no account.

Ask your first question freeSubscribe

Formula With Every Symbol Explained

Kullback-Leibler divergence is formulated for both discrete and continuous random variables, expressing the expected logarithmic difference between true and approximating probabilities.

Discrete Distributions

For discrete probability distributions P and Q on sample space 𝒳:

DKL(P ∥ Q) = ∑x ∈ 𝒳 P(x) log(P(x) / Q(x))

Equivalently: DKL(P ∥ Q) = ∑x ∈ 𝒳 P(x) [log P(x) − log Q(x)]

Continuous Distributions

For continuous distributions with density functions p(x) and q(x):

DKL(P ∥ Q) = ∫𝒳 p(x) log(p(x) / q(x)) dx

Explanation of Symbols

  • P: The reference or true baseline probability distribution.
  • Q: The candidate, model, or approximating probability distribution.
  • x: An observed category, discrete event, or continuous coordinate in support domain 𝒳.
  • P(x), p(x): The probability mass or density under the reference distribution.
  • Q(x), q(x): The probability mass or density under the approximating distribution.
  • log: The logarithmic function determining the unit of information:
    • Natural log (ln, base e): Yields nats (natural units), standard in statistics.
    • Binary log (log2, base 2): Yields bits (or shannons). One nat equals ≈ 1.442695 bits.
  • Relation to Entropy: With Shannon entropy H(P) = −∑ P(x) log P(x) and cross-entropy H(P, Q) = −∑ P(x) log Q(x), divergence satisfies: DKL(P ∥ Q) = H(P, Q) − H(P).

Worked Example With Real Numbers and Arithmetic Shown

Consider an empirical study evaluating patient clinical response across three discrete categories: Remission (C1), Partial Response (C2), and Non-Response (C3).

The observed historical distribution P and candidate model distribution Q are:

  • Reference (P): P(C1) = 0.50, P(C2) = 0.30, P(C3) = 0.20 (sum = 1.00)
  • Approximation (Q): Q(C1) = 0.40, Q(C2) = 0.40, Q(C3) = 0.20 (sum = 1.00)

We calculate DKL(P ∥ Q) in natural logarithm units (nats):

Category-by-Category Calculation

  • Category 1: 0.50 × ln(0.50 / 0.40) = 0.50 × ln(1.25) = 0.50 × 0.22314355 = 0.11157178 nats
  • Category 2: 0.30 × ln(0.30 / 0.40) = 0.30 × ln(0.75) = 0.30 × (−0.28768207) = −0.08630462 nats
  • Category 3: 0.20 × ln(0.20 / 0.20) = 0.20 × ln(1.00) = 0.20 × 0.00000000 = 0.00000000 nats

Total Divergence

Summing the individual contributions:

DKL(P ∥ Q) = 0.11157178 − 0.08630462 + 0.00000000 = 0.02526716 nats

Rounded to four decimal places, DKL(P ∥ Q) = 0.0253 nats (or 0.02526716 × 1.442695 = 0.0365 bits).

Demonstration of Asymmetry

Calculating the reverse divergence DKL(Q ∥ P):

  • Category 1: 0.40 × ln(0.40 / 0.50) = 0.40 × (−0.22314355) = −0.08925742 nats
  • Category 2: 0.40 × ln(0.40 / 0.30) = 0.40 × 0.28768207 = 0.11507283 nats
  • Category 3: 0.20 × ln(0.20 / 0.20) = 0.00000000 nats

Sum: DKL(Q ∥ P) = 0.02581541 nats (0.0258 nats). Because 0.0253 nats ≠ 0.0258 nats, directionality is non-interchangeable.

Interpretation

By Gibbs’ inequality, DKL(P ∥ Q) ≥ 0, with equality holding if and only if P = Q. Divergence measures information loss: lower values indicate higher fidelity to the true data distribution.

In applied statistics, KL divergence underpins empirical estimation via the likelihood function. Maximizing log-likelihood is mathematically equivalent to minimizing sample KL divergence. In model selection frameworks such as the Akaike Information Criterion (AIC), the index acts as an asymptotically unbiased estimator of relative expected KL divergence between true reality and approximating models.

Divergence Range (nats) Distribution Similarity Information Loss Practical Research Meaning
DKL = 0.0000 Identical Zero loss Model Q perfectly matches reference distribution P across all states.
0.0000 < DKL ≤ 0.0500 High fidelity Minimal loss Excellent approximation; differences are negligible for inference.
0.0500 < DKL ≤ 0.2000 Moderate fidelity Noticeable divergence Adequate approximation; category probability deviations are noticeable.
0.2000 < DKL ≤ 0.5000 Low fidelity Substantial loss Substantial discrepancy; estimates or forecasts carry material error.
DKL > 0.5000 Poor fidelity Severe mismatch Model fails to capture reference structure; redesign model specification.
DKL = ∞ Discontinuous Infinite penalty Model Q assigns zero probability to an event that occurs in P.

Assumptions and Common Mistakes

  • Absolute Continuity: P must be absolutely continuous with respect to Q. Wherever P(x) > 0, Q(x) must be strictly positive. If Q(x) = 0 for any P(x) > 0, the ratio diverges to infinity (DKL = ∞). When empirical counts contain zeroes, apply Laplace smoothing. Conversely, if P(x) = 0, the term evaluates to 0.
  • Treating Divergence as a Metric: KL divergence violates symmetry (DKL(P ∥ Q) ≠ DKL(Q ∥ P)) and the triangle inequality. It cannot serve as a Euclidean distance metric.
  • Directional Sensitivity: Minimizing forward divergence DKL(P ∥ Q) produces mean-seeking behavior covering all support of P. Minimizing reverse divergence DKL(Q ∥ P) produces mode-seeking behavior focusing on major modes, which is central to variational inference and t-SNE dimensionality reduction.
  • Mixing Log Bases: Comparing values across studies without converting between nats (natural log) and bits (base 2) invalidates conclusions.
  • Unnormalized Vectors: Input arrays must sum to 1.0; unnormalized raw frequencies yield invalid results.

How to Report It (APA Example)

When reporting Kullback-Leibler divergence conforming to APA 7th edition standards, specify the reference distribution P, candidate model Q, logarithmic base, resulting numerical divergence, and practical inference. For nested models, pair divergence with a likelihood ratio test.

“To evaluate the distributional fidelity of the parametric clinical outcome model against observed cohort data, we calculated Kullback-Leibler divergence across all three response categories. Using natural logarithms, the directed divergence from empirical reference distribution P to candidate model Q was DKL(P ∥ Q) = 0.0253 nats (0.0365 bits), with reverse divergence DKL(Q ∥ P) = 0.0258 nats. Because the information discrepancy remained below the pre-established threshold of 0.0500 nats, candidate model Q was accepted for subsequent trial simulation.”

How to Calculate It in Excel, R or Python

Below are verified implementations across common analytical software environments.

Python (SciPy)

import numpy as np
from scipy.stats import entropy

# Define probability distributions (must sum to 1.0)
p = np.array([0.50, 0.30, 0.20])
q = np.array([0.40, 0.40, 0.20])

# Compute KL divergence in nats (default base=None uses ln)
kl_nats = entropy(p, q)
print(f"KL Divergence (nats): {kl_nats:.4f}")  # 0.0253

# Compute KL divergence in bits (base=2)
kl_bits = entropy(p, q, base=2)
print(f"KL Divergence (bits): {kl_bits:.4f}")  # 0.0365

R

p <- c(0.50, 0.30, 0.20)
q <- c(0.40, 0.40, 0.20)

# Base R calculation with zero protection
calc_kl <- function(p, q) {
  terms <- ifelse(p == 0, 0, p * log(p / q))
  sum(terms)
}
kl_nats <- calc_kl(p, q)
cat("KL Divergence (nats):", round(kl_nats, 4), "n")  # 0.0253

# Alternative using philentropy package
library(philentropy)
kl_pkg <- KL(rbind(p, q), unit = "log")
cat("philentropy KL:", round(kl_pkg, 4), "n")          # 0.0253

Microsoft Excel

  1. Place reference probabilities P in cells B2:B4 (0.50, 0.30, 0.20) and model probabilities Q in cells C2:C4 (0.40, 0.40, 0.20).
  2. In cell D2, enter: =IF(B2=0, 0, B2 * LN(B2 / C2)).
  3. Copy D2 down through D3:D4.
  4. In cell D5, sum the column: =SUM(D2:D4), yielding 0.0253 nats. Divide by LN(2) for bits (0.0365 bits).

Frequently Asked Questions

Is Kullback-Leibler divergence a true distance metric?

No. A metric requires symmetry (d(x, y) = d(y, x)) and satisfaction of the triangle inequality. KL divergence violates both conditions: DKL(P ∥ Q) ≠ DKL(Q ∥ P), classifying it as a statistical divergence or premetric.

Can Kullback-Leibler divergence be negative?

No. By Gibbs’ inequality, DKL(P ∥ Q) ≥ 0 for all valid probability distributions. It equals zero if and only if P = Q almost everywhere. Negative results indicate arithmetic error or unnormalized distributions.

What is the difference between KL divergence and cross-entropy?

Cross-entropy H(P, Q) = −∑ P(x) log Q(x) measures total expected code length using approximation Q. KL divergence DKL(P ∥ Q) = H(P, Q) − H(P) measures only the excess code length relative to the intrinsic entropy of P.

What units are used for Kullback-Leibler divergence?

Units depend on the logarithm base: natural log (base e) gives nats, base-2 log gives bits, and base-10 log gives hartleys. Multiply nats by log2(e) ≈ 1.442695 to convert to bits.

What happens if an approximating probability is zero?

If P(x) > 0 and Q(x) = 0, division by zero produces an infinite divergence (DKL = ∞). In statistical practice, this acts as an infinite penalty for assigning zero probability to an observed event. Practical applications use smoothing constants to ensure non-zero support.

Sources

  • Burnham, K. P., & Anderson, D. R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach (2nd ed.). Springer-Verlag. ISBN: 978-0-387-95364-9. Checked 2026-11-07.
  • Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). John Wiley & Sons, Inc. ISBN: 978-0-471-24195-9. Checked 2026-11-07.
  • Kullback, S., & Leibler, R. A. (1951). On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1), 79–86. https://doi.org/10.1214/aoms/1177729694. Checked 2026-11-07.
  • SciPy Community. (2024). scipy.stats.entropy Reference Manual. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html. Checked 2026-11-07.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about Kullback-Leibler Divergence: Formula & Example

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →