Sigmoid Gradient

1. Background Knowledge

The sigmoid function is a fundamental activation function in neural networks, defined as: $\sigma(z) = \frac{1}{1 + e^{-z}}$

It maps any real input $z \in \mathbb{R}$ to the range $(0, 1)$, making it ideal for binary classification and probability outputs. Key properties include:

• Smooth and differentiable everywhere
• S-shaped (S-curve) with inflection point at $z = 0$, where $\sigma(0) = 0.5$
• Asymptotic behavior: $\sigma(z) \to 1$ as $z \to \infty$, $\sigma(z) \to 0$ as $z \to -\infty$

Critical property for backpropagation: The derivative can be computed directly from the output: $\sigma'(z) = \sigma(z) \cdot (1 - \sigma(z))$

Proof sketch: Let g = \sigma(z). Then: $\sigma'(z) = \frac{d}{dz} \left( \frac{1}{1 + e^{-z}} \right) = \frac{e^{-z}}{(1 + e^{-z})^2} = \frac{1}{1 + e^{-z}} \cdot \frac{e^{-z}}{1 + e^{-z}} = g \cdot (1 - g)$

This eliminates the need to store $z$ values during the forward pass, saving memory in deep networks.

Prerequisites:

• Basic calculus (chain rule, quotient rule)
• Understanding of neural network backpropagation
• Python list operations and rounding

2. Algorithm Approach

Direct computation using the closed-form derivative formula—no numerical differentiation needed:

1. For each element $g_i$ in the input list, compute $g_i \times (1 - g_i)$
2. Round result to 4 decimal places
3. Return as list

Alternative approaches (not needed here but educational):

• Numerical approximation: \sigma'(z) \approx \frac{\sigma$(z + h) -$\sigma(z - h)}{2h} (less efficient, error-prone)
• Vectorized operations (NumPy preferred for larger arrays)

Time complexity: $O(n)$ where $n$ is list length (1-100 elements).

3. Step-by-Step Strategy

1. Input validation (optional but good practice): Ensure $0 \leq g_i \leq 1$
2. Element-wise computation: For each $g$ in sigmoid_output, calculate $g \times (1 - g)$
3. Precision handling: Use round(value, 4) for each result
4. List construction: Collect results in new list
5. Return: Output list matching input length

Sample implementation:

def sigmoid_derivative(sigmoid_output):
 return [round(g * (1 - g), 4) for g in sigmoid_output]

Verification:

• Input [0.5, 0.8] → [0.5×0.5=0.25, 0.8×0.2=0.16] ✓
• Edge cases: [0.0] → [0.0], [1.0] → [0.0], maximum at [0.5] → [0.25]

4. Common Pitfalls

• Not rounding to 4 decimals: 0.159999999 instead of 0.1600

# Wrong
[g * (1 - g) for g in sigmoid_output] # [0.16] may print as [0.159999999]

# Correct
[round(g * (1 - g), 4) for g in sigmoid_output] # [0.16]

• Input range violation: Derivative undefined or negative outside $[0,1]$

sigmoid_derivative([-0.1, 1.1]) # Produces negative values—invalid!

• Using sigmoid input instead of output: Must use activation output $g$, not pre-activation $z$

• Floating-point precision: Use round() consistently; Python's print may show 0.16 as 0.159999999

• Mutable input modification: Don't alter original list—create new one

5. Time & Space Complexity

Scalability notes:

• Perfectly efficient for constraints (n ≤ 100)
• For large arrays, use NumPy: np.round(sigmoid_output * (1 - sigmoid_output), 4)
• No recursion, sorting, or heavy computation—purely linear

Gradient interpretation: Maximum gradient magnitude is $0.25$ at $g = 0.5$, vanishing at boundaries ($g=0,1$)—explains vanishing gradient problem in deep sigmoid networks.