KL Divergence
Compute the Kullback-Leibler divergence between two discrete probability distributions.
DKL(P∥Q)=∑iP(i)lnQ(i)P(i)
where P and Q are probability distributions over the same set of events.
By convention, if P(i)=0, the term contributes 0 to the sum (since 0ln0=0). If P(i)>0 but Q(i)=0, the divergence is infinity.
Return the KL divergence rounded to 4 decimal places. Return float('inf') if undefined.
Example:
p = [0.4, 0.6] q = [0.5, 0.5]
0.0201
- The probability distributions are given as P=[0.4,0.6] and Q=[0.5,0.5].
- We calculate the KL divergence using the formula: DKL(P∥Q)=0.4ln0.50.4+0.6ln0.50.6.
- Evaluating the expression: DKL(P∥Q)=0.4ln(0.8)+0.6ln(1.2), which is approximately 0.4⋅(−0.223)+0.6⋅0.182.
- The final result is approximately −0.089+0.109=0.020, rounded to 4 decimal places: 0.0201.
Constraints:
- p, q: lists of probabilities (same length, sum to 1)
- Use natural log
- Return float rounded to 4 decimal places
- Return float('inf') if P(i) > 0 and Q(i) = 0
Background Knowledge
The Kullback-Leibler divergence is a fundamental concept in information theory and machine learning, particularly in generative models. It measures the difference between two probability distributions, P and Q, over the same set of events. The KL divergence is often used to quantify the similarity between two distributions, with a value of 0 indicating that the distributions are identical. The KL divergence is not symmetric, meaning that DKL(P∥Q)=DKL(Q∥P) in general.
The KL divergence has several important properties. Firstly, it is always non-negative, meaning that DKL(P∥Q)≥0. Secondly, it is zero if and only if P=Q. The KL divergence is also related to the entropy of a distribution, which measures the amount of uncertainty or randomness in the distribution. The KL divergence can be thought of as a measure of the additional uncertainty or information required to describe P using Q.
In the context of generative models, the KL divergence is often used to regularize the model and prevent it from producing unrealistic or unlikely samples. By minimizing the KL divergence between the model's distribution and the true data distribution, the model can learn to generate more realistic samples. The KL divergence is also used in variational inference, which is a technique for approximating complex distributions using simpler distributions.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.