📘

Entropy of a Policy

EasyReinforcement Learning 📚 RL 1: Foundations

Problem Statement

Policy entropy encourages exploration and is often added as a bonus. For a policy distribution pi:

$H(\pi) = -\sum_a \pi(a)\, \log \pi(a)$

Use natural log. Treat pi(a) = 0 as contributing 0 (since 0 log 0 = 0). Implement policy_entropy(pi) returning a float.

Example:

Input:

policy_entropy([0.5, 0.5])

Output:

0.6931

Reasoning:

Initialize the entropy accumulator to $0$ to sum the contributions of each action in the policy distribution.
Process the first probability $p = 0.5$ : since it is non-zero, calculate its contribution as $-0.5 \times \ln(0.5) \approx 0.3466$ .
Process the second probability $p = 0.5$ : similarly, calculate its contribution as $-0.5 \times \ln(0.5) \approx 0.3466$ .
Sum these individual contributions to find the total entropy: $0.3466 + 0.3466 = 0.6931$ .
The final output is 0.6931

Constraints:

pi is a valid distribution (sums to 1); entries >= 0.
Skip zero-probability actions.
Return a float.

More from RL 1: Foundations

GAE Over a Full TrajectoryHard PPO Clipped Objective (Single Sample)Medium TD Residuals for GAEMedium Return Standardization BaselineMedium REINFORCE Loss TermMedium

1. Background Knowledge

Entropy measures the uncertainty or "surprise" of a probability distribution. For a discrete policy $\pi$ over a set of actions $\mathcal{A}$ , the entropy is defined as:

H(\pi) = -\sum_{a \in \mathcal{A}} \pi(a) \log \pi(a)

A deterministic policy (one action has probability 1, others 0) has zero entropy, meaning no exploration is needed. A uniform policy over $n$ actions has maximum entropy $\log n$ , representing maximal uncertainty. In policy gradient methods, adding an entropy bonus to the objective function encourages the agent to keep exploring rather than collapsing to a greedy policy too early.

The term $\pi(a) \log \pi(a)$ is well-behaved in the limit: as $\pi(a) \to 0^+$ , the product $\pi(a) \log \pi(a) \to 0$ . This is a standard result from calculus (L'Hôpital's rule or the fact that $x \log x \to 0$ as $x \to 0^+$ ). Therefore, actions with zero probability contribute nothing to the sum, and we should explicitly handle them to avoid computing $\log(0) = -\infty$ .

In practice, policies are often represented as probability vectors (e.g., from a softmax output). The entropy is computed over this vector using the natural logarithm ( $\ln$ ), not log base 2 or base 10.

2. Algorithm Approach

This is a straightforward element-wise computation over a probability vector:

Iterate over each action's probability $\pi(a)$ .
If $\pi(a) > 0$ , compute $\pi(a) \cdot \ln(\pi(a))$ and accumulate.
If $\pi(a) = 0$ , skip (contribution is 0).
Return the negative of the accumulated sum.

The key insight is that the formula involves a sum of products, so you need a single pass over the vector with a running total. No sorting, no recursion, no data structures beyond a scalar accumulator.

3. Step-by-Step Strategy

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.