Binary Cross-Entropy Loss

1. Background Knowledge

Binary Cross-Entropy (BCE) loss quantifies the difference between predicted probabilities $\hat{y}$ and true binary labels $y \in \{0, 1\}$ for binary classification tasks. The formula is:

$L = -(y \cdot \log($\hat{y}$) + (1 - y) \cdot \log(1 -$\hat{y}$))$

This derives from information theory as the Kullback-Leibler divergence between true and predicted distributions, measuring prediction uncertainty. BCE penalizes confident wrong predictions heavily (e.g., $\hat{y} \approx 0$ when $y=1$) and approaches 0 for perfect predictions. It's the standard loss for logistic regression and binary classification in neural networks.

Key prerequisites:

• Natural logarithm: $\log$ is base-$e$; Python's math.log computes this.
• Numerical stability: $\log(0)$ is undefined ($-\infty$), so clip $\hat{y}$ with $\epsilon = 10^{-15}$: $\hat{y} = \max(\min($\hat{y}$, 1-$\epsilon$),$\epsilon$)$.
• Single sample: No averaging over batches needed here.

2. Algorithm Approach

Direct computation using the closed-form formula—no optimization or iteration required. Common techniques:

• Clipping for stability: Standard practice to prevent NaN/inf.
• Vectorized implementation (for batches, though not needed here): Use NumPy for efficiency.
• Alternatives in literature: Weighted BCE for imbalance, but vanilla BCE suffices for this balanced single-sample case.

The approach is O(1) arithmetic: log, multiply, add.