Written and maintained by CASRAI Editorial Board
Published
On this page
Definition in One Paragraph
Kullback-Leibler (KL) divergence, also designated relative entropy or directed divergence, is a non-negative statistical measure of the difference between an approximating probability distribution Q and a reference distribution P. Originating in information theory and mathematical statistics, it quantifies the expected excess surprise or information lost per observation when encoding outcomes from P using a code optimized for Q. Across discrete settings such as categorical distributions, KL divergence equals zero if and only if both distributions are identical and grows larger as their shapes diverge, serving as an asymmetric statistical discrepancy measure rather than a metric distance.
Ask CASRAI · free to tryQuestion about this guide? Ask CASRAI answers with linked sources. 2 free questions a day, no account.
Formula With Every Symbol Explained
Kullback-Leibler divergence is formulated for both discrete and continuous random variables, expressing the expected logarithmic difference between true and approximating probabilities.
Discrete Distributions
For discrete probability distributions P and Q on sample space 𝒳:
DKL(P ∥ Q) = ∑x ∈ 𝒳 P(x) log(P(x) / Q(x))
Equivalently: DKL(P ∥ Q) = ∑x ∈ 𝒳 P(x) [log P(x) − log Q(x)]
Continuous Distributions
For continuous distributions with density functions p(x) and q(x):
DKL(P ∥ Q) = ∫𝒳 p(x) log(p(x) / q(x)) dx
Explanation of Symbols
- P: The reference or true baseline probability distribution.
- Q: The candidate, model, or approximating probability distribution.
- x: An observed category, discrete event, or continuous coordinate in support domain 𝒳.
- P(x), p(x): The probability mass or density under the reference distribution.
- Q(x), q(x): The probability mass or density under the approximating distribution.
- log: The logarithmic function determining the unit of information:
- Natural log (ln, base e): Yields nats (natural units), standard in statistics.
- Binary log (log2, base 2): Yields bits (or shannons). One nat equals ≈ 1.442695 bits.
- Relation to Entropy: With Shannon entropy H(P) = −∑ P(x) log P(x) and cross-entropy H(P, Q) = −∑ P(x) log Q(x), divergence satisfies: DKL(P ∥ Q) = H(P, Q) − H(P).
Worked Example With Real Numbers and Arithmetic Shown
Consider an empirical study evaluating patient clinical response across three discrete categories: Remission (C1), Partial Response (C2), and Non-Response (C3).
The observed historical distribution P and candidate model distribution Q are:
- Reference (P): P(C1) = 0.50, P(C2) = 0.30, P(C3) = 0.20 (sum = 1.00)
- Approximation (Q): Q(C1) = 0.40, Q(C2) = 0.40, Q(C3) = 0.20 (sum = 1.00)
We calculate DKL(P ∥ Q) in natural logarithm units (nats):
Category-by-Category Calculation
- Category 1: 0.50 × ln(0.50 / 0.40) = 0.50 × ln(1.25) = 0.50 × 0.22314355 = 0.11157178 nats
- Category 2: 0.30 × ln(0.30 / 0.40) = 0.30 × ln(0.75) = 0.30 × (−0.28768207) = −0.08630462 nats
- Category 3: 0.20 × ln(0.20 / 0.20) = 0.20 × ln(1.00) = 0.20 × 0.00000000 = 0.00000000 nats
Total Divergence
Summing the individual contributions:
DKL(P ∥ Q) = 0.11157178 − 0.08630462 + 0.00000000 = 0.02526716 nats
Rounded to four decimal places, DKL(P ∥ Q) = 0.0253 nats (or 0.02526716 × 1.442695 = 0.0365 bits).
Demonstration of Asymmetry
Calculating the reverse divergence DKL(Q ∥ P):
- Category 1: 0.40 × ln(0.40 / 0.50) = 0.40 × (−0.22314355) = −0.08925742 nats
- Category 2: 0.40 × ln(0.40 / 0.30) = 0.40 × 0.28768207 = 0.11507283 nats
- Category 3: 0.20 × ln(0.20 / 0.20) = 0.00000000 nats
Sum: DKL(Q ∥ P) = 0.02581541 nats (0.0258 nats). Because 0.0253 nats ≠ 0.0258 nats, directionality is non-interchangeable.
Interpretation
By Gibbs’ inequality, DKL(P ∥ Q) ≥ 0, with equality holding if and only if P = Q. Divergence measures information loss: lower values indicate higher fidelity to the true data distribution.
In applied statistics, KL divergence underpins empirical estimation via the likelihood function. Maximizing log-likelihood is mathematically equivalent to minimizing sample KL divergence. In model selection frameworks such as the Akaike Information Criterion (AIC), the index acts as an asymptotically unbiased estimator of relative expected KL divergence between true reality and approximating models.
| Divergence Range (nats) | Distribution Similarity | Information Loss | Practical Research Meaning |
|---|---|---|---|
| DKL = 0.0000 | Identical | Zero loss | Model Q perfectly matches reference distribution P across all states. |
| 0.0000 < DKL ≤ 0.0500 | High fidelity | Minimal loss | Excellent approximation; differences are negligible for inference. |
| 0.0500 < DKL ≤ 0.2000 | Moderate fidelity | Noticeable divergence | Adequate approximation; category probability deviations are noticeable. |
| 0.2000 < DKL ≤ 0.5000 | Low fidelity | Substantial loss | Substantial discrepancy; estimates or forecasts carry material error. |
| DKL > 0.5000 | Poor fidelity | Severe mismatch | Model fails to capture reference structure; redesign model specification. |
| DKL = ∞ | Discontinuous | Infinite penalty | Model Q assigns zero probability to an event that occurs in P. |
Assumptions and Common Mistakes
- Absolute Continuity: P must be absolutely continuous with respect to Q. Wherever P(x) > 0, Q(x) must be strictly positive. If Q(x) = 0 for any P(x) > 0, the ratio diverges to infinity (DKL = ∞). When empirical counts contain zeroes, apply Laplace smoothing. Conversely, if P(x) = 0, the term evaluates to 0.
- Treating Divergence as a Metric: KL divergence violates symmetry (DKL(P ∥ Q) ≠ DKL(Q ∥ P)) and the triangle inequality. It cannot serve as a Euclidean distance metric.
- Directional Sensitivity: Minimizing forward divergence DKL(P ∥ Q) produces mean-seeking behavior covering all support of P. Minimizing reverse divergence DKL(Q ∥ P) produces mode-seeking behavior focusing on major modes, which is central to variational inference and t-SNE dimensionality reduction.
- Mixing Log Bases: Comparing values across studies without converting between nats (natural log) and bits (base 2) invalidates conclusions.
- Unnormalized Vectors: Input arrays must sum to 1.0; unnormalized raw frequencies yield invalid results.
How to Report It (APA Example)
When reporting Kullback-Leibler divergence conforming to APA 7th edition standards, specify the reference distribution P, candidate model Q, logarithmic base, resulting numerical divergence, and practical inference. For nested models, pair divergence with a likelihood ratio test.
“To evaluate the distributional fidelity of the parametric clinical outcome model against observed cohort data, we calculated Kullback-Leibler divergence across all three response categories. Using natural logarithms, the directed divergence from empirical reference distribution P to candidate model Q was DKL(P ∥ Q) = 0.0253 nats (0.0365 bits), with reverse divergence DKL(Q ∥ P) = 0.0258 nats. Because the information discrepancy remained below the pre-established threshold of 0.0500 nats, candidate model Q was accepted for subsequent trial simulation.”
How to Calculate It in Excel, R or Python
Below are verified implementations across common analytical software environments.
Python (SciPy)
import numpy as np
from scipy.stats import entropy
# Define probability distributions (must sum to 1.0)
p = np.array([0.50, 0.30, 0.20])
q = np.array([0.40, 0.40, 0.20])
# Compute KL divergence in nats (default base=None uses ln)
kl_nats = entropy(p, q)
print(f"KL Divergence (nats): {kl_nats:.4f}") # 0.0253
# Compute KL divergence in bits (base=2)
kl_bits = entropy(p, q, base=2)
print(f"KL Divergence (bits): {kl_bits:.4f}") # 0.0365
R
p <- c(0.50, 0.30, 0.20)
q <- c(0.40, 0.40, 0.20)
# Base R calculation with zero protection
calc_kl <- function(p, q) {
terms <- ifelse(p == 0, 0, p * log(p / q))
sum(terms)
}
kl_nats <- calc_kl(p, q)
cat("KL Divergence (nats):", round(kl_nats, 4), "n") # 0.0253
# Alternative using philentropy package
library(philentropy)
kl_pkg <- KL(rbind(p, q), unit = "log")
cat("philentropy KL:", round(kl_pkg, 4), "n") # 0.0253
Microsoft Excel
- Place reference probabilities P in cells
B2:B4(0.50,0.30,0.20) and model probabilities Q in cellsC2:C4(0.40,0.40,0.20). - In cell
D2, enter:=IF(B2=0, 0, B2 * LN(B2 / C2)). - Copy
D2down throughD3:D4. - In cell
D5, sum the column:=SUM(D2:D4), yielding 0.0253 nats. Divide byLN(2)for bits (0.0365 bits).
Frequently Asked Questions
Is Kullback-Leibler divergence a true distance metric?
No. A metric requires symmetry (d(x, y) = d(y, x)) and satisfaction of the triangle inequality. KL divergence violates both conditions: DKL(P ∥ Q) ≠ DKL(Q ∥ P), classifying it as a statistical divergence or premetric.
Can Kullback-Leibler divergence be negative?
No. By Gibbs’ inequality, DKL(P ∥ Q) ≥ 0 for all valid probability distributions. It equals zero if and only if P = Q almost everywhere. Negative results indicate arithmetic error or unnormalized distributions.
What is the difference between KL divergence and cross-entropy?
Cross-entropy H(P, Q) = −∑ P(x) log Q(x) measures total expected code length using approximation Q. KL divergence DKL(P ∥ Q) = H(P, Q) − H(P) measures only the excess code length relative to the intrinsic entropy of P.
What units are used for Kullback-Leibler divergence?
Units depend on the logarithm base: natural log (base e) gives nats, base-2 log gives bits, and base-10 log gives hartleys. Multiply nats by log2(e) ≈ 1.442695 to convert to bits.
What happens if an approximating probability is zero?
If P(x) > 0 and Q(x) = 0, division by zero produces an infinite divergence (DKL = ∞). In statistical practice, this acts as an infinite penalty for assigning zero probability to an observed event. Practical applications use smoothing constants to ensure non-zero support.
Sources
- Burnham, K. P., & Anderson, D. R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach (2nd ed.). Springer-Verlag. ISBN: 978-0-387-95364-9. Checked 2026-11-07.
- Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). John Wiley & Sons, Inc. ISBN: 978-0-471-24195-9. Checked 2026-11-07.
- Kullback, S., & Leibler, R. A. (1951). On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1), 79–86. https://doi.org/10.1214/aoms/1177729694. Checked 2026-11-07.
- SciPy Community. (2024).
scipy.stats.entropyReference Manual. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html. Checked 2026-11-07.







