KL Divergence

mediumPROGenerative ModelsMachine Learning 2

Compute the Kullback-Leibler divergence between two discrete probability distributions.

$D_{KL}(P \| Q) = \sum_{i} P(i) \ln\frac{P(i)}{Q(i)}$

where $P$ and $Q$ are probability distributions over the same set of events.

By convention, if $P(i) = 0$ , the term contributes 0 to the sum (since $0 \ln 0 = 0$ ). If $P(i) > 0$ but $Q(i) = 0$ , the divergence is infinity.

Return the KL divergence rounded to 4 decimal places. Return float('inf') if undefined.

Input

p = [0.4, 0.6]
q = [0.5, 0.5]

Output

0.0201

Reasoning

p and q are two probability distributions over the same two events: $P = (0.4, 0.6)$ is the "true" distribution and $Q = (0.5, 0.5)$ is the uniform approximation being compared against it.
Each event contributes $P(i)\ln\frac{P(i)}{Q(i)}$ , weighted by how likely it is under $P$ .
Event 1: the ratio is $\frac{0.4}{0.5} = 0.8$ , and $0.4 \ln 0.8 = 0.4 \times (-0.22314) \approx -0.08926$ . It is negative because $Q$ puts more mass here than $P$ does.
Event 2: the ratio is $\frac{0.6}{0.5} = 1.2$ , and $0.6 \ln 1.2 = 0.6 \times 0.18232 \approx 0.10939$ .
Summing: $-0.08926 + 0.10939 = 0.02013$ , which rounded to 4 decimal places is 0.0201.
Notice the total is small but positive: individual terms can be negative, yet the sum is never below 0, and it is exactly 0 only when $P = Q$ . It is also not symmetric: swapping the roles of $P$ and $Q$ gives $0.0204$ , not $0.0201$ .

Background Knowledge