📘

Information Gain

Compute the information gain from splitting a dataset.

Given a parent set of labels and the labels in two child subsets after a split, compute:

$IG = H(parent) - \frac{|left|}{|parent|} H(left) - \frac{|right|}{|parent|} H(right)$

where $H$ is the entropy: $H = -\sum_{k=1}^{K} p_k \log_2(p_k)$

Use $0 \log_2(0) = 0$ by convention. Return the information gain rounded to 4 decimal places.

Input:

parent = [1, 1, 0, 0]
left = [1, 1]
right = [0, 0]

Output:

1.0

Reasoning:

First, we calculate the entropy of the parent set: $H(parent) = -\left(\frac{2}{4} \log_2\left(\frac{2}{4}\right) + \frac{2}{4} \log_2\left(\frac{2}{4}\right)\right) = -\left(\frac{1}{2} \log_2\left(\frac{1}{2}\right) + \frac{1}{2} \log_2\left(\frac{1}{2}\right)\right) = 1$
Then, we calculate the entropy of the left and right child sets: $H(left) = -\left(\frac{2}{2} \log_2\left(\frac{2}{2}\right)\right) = 0$ and $H(right) = -\left(\frac{2}{2} \log_2\left(\frac{2}{2}\right)\right) = 0$
Next, we apply the information gain formula: $IG = H(parent) - \frac{|left|}{|parent|} H(left) - \frac{|right|}{|parent|} H(right) = 1 - \frac{2}{4} \cdot 0 - \frac{2}{4} \cdot 0 = 1$
The final output is $1.0$ after rounding to 4 decimal places

Editor

Python 3.13.1

0/0

Run code to see test results.

📘