📘

ReLU Derivative

Implement a function to compute the derivative of the Rectified Linear Unit (ReLU) activation function, a crucial component in Backpropagation. The ReLU function is defined as $f(x) = \max(0, x)$ , and its derivative is essential for training Deep Learning models.

The ReLU derivative is a piecewise function that depends on the input value $x$ . Understanding this derivative is vital for optimizing neural networks using Backpropagation, as it helps compute the gradients of the loss function with respect to the model's parameters.

To compute the ReLU derivative, follow these steps:

Evaluate the input value $x$ .
Apply the derivative formula based on the value of $x$ .

\frac{d}{dx}\text{ReLU}(x) = \begin{cases} 1 & x > 0 \\ 0 & x \leq 0 \end{cases}

This technique is widely used in Deep Learning models for image and speech recognition tasks.

Example:

Input:

relu_derivative([-1, 0, 1, 2])

Output:

[0, 0, 1, 1]

Reasoning:

For each input value, apply the ReLU derivative rule: if $x > 0$ , the derivative is 1; if $x \leq 0$ , the derivative is 0[5]
Evaluate each element: $-1 \leq 0$ → 0, $0 \leq 0$ → 0, $1 > 0$ → 1, $2 > 0$ → 1[5]
The output is the array of derivatives corresponding to each input: [0, 0, 1, 1]

Constraints:

Return derivative for each element

More from CV: Deep Learning

Implement Max Pooling with IndicesHard Depthwise Separable ConvolutionMedium Softmax Cross-Entropy GradientMedium Batch Normalization Forward PassHard Flatten Feature MapEasy

Background Knowledge

ReLU (Rectified Linear Unit) is a cornerstone activation function in deep neural networks, defined as $\text{ReLU}(x) = \max(0, x)$ . It introduces nonlinearity while being computationally efficient and mitigating the vanishing gradient problem seen in sigmoid/tanh functions, as its derivative is simple (1 for $x > 0$ , 0 otherwise). This piecewise linear form allows sparse activation—neurons "turn off" for negative inputs—promoting faster training and better generalization in backpropagation.

Backpropagation computes gradients of the loss with respect to weights via the chain rule, requiring derivatives of all operations, including activations. For ReLU, the derivative is a step function: $\frac{d}{dx}$ \text{ReLU} $(x) = 1$ if $x > 0$ , and $0$ if $x \leq 0$ . This "gates" gradients during training: positive pre-activations propagate errors backward fully, while negative ones block them, which can cause "dying ReLU" (neurons permanently outputting 0). Understanding this subgradient (at $x=0$ , it's typically set to 0) is key for implementing correct gradient flow in deep learning frameworks.

Algorithm/Approach

Implement the ReLU derivative as a piecewise conditional function matching the given specification. Use simple if-else logic or equivalent vectorized operations (e.g., in NumPy/PyTorch) to return 1 for positive inputs and 0 otherwise. This mirrors backpropagation's local gradient computation, ensuring compatibility with autograd systems while handling the discontinuity at zero.

Step-by-Step Strategy

Define the input: Accept a scalar, vector, or tensor $x$ representing pre-activation values.
Apply the condition: Check where $x > 0$ ; return 1 (or True/1.0) for those elements.
Handle non-positive cases: Set 0 for $x \leq 0$ (includes exactly zero to match the spec).
Vectorize for efficiency: Use boolean masking (e.g., (x > 0).astype(float)) to support batches without loops.
Test edge cases: Verify at $x=0$ , positives, negatives, and arrays to ensure subgradient choice aligns with standard implementations.

Common Pitfalls

Treating $x=0$ as differentiable (gradient undefined; convention is 0, but some frameworks allow 0.5—stick to problem spec).
Forgetting vectorization, leading to scalar-only code that fails on arrays.
Numerical instability near zero due to floating-point precision (use strict > comparison).
Confusing with smoothed variants (e.g., softplus); this is exact piecewise.
"Dying ReLU" awareness: derivative blocks gradients for negatives, but that's intended—don't modify unless specified.

Time & Space Complexity

Time: $O(n)$ where $n$ is input size (single pass comparison/masking).
Space: $O(n)$ for output tensor (in-place possible for $O(1)$ extra space). Constant for scalars. Ideal for backprop in large networks.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.