ReLU Activation

easyFREENeural NetworksDeep LearningCV: Deep Learning

Implement the ReLU activation function, a crucial component in Neural Networks. This function is used to introduce non-linearity in the model, enabling it to learn complex relationships between inputs and outputs.

The ReLU function, or Rectified Linear Unit, is defined as $max(0, x)$ , where $x$ is the input to the function. This function is widely used in Deep Learning due to its simplicity and computational efficiency.

The input $x$ is passed through the function.
The function returns $0$ if $x$ is negative, and $x$ if $x$ is positive.

\text{ReLU}(x) = \max(0, x)

This technique is widely used in image classification tasks.

Example

Input

relu([-2, -1, 0, 1, 2])

Output

[0, 0, 0, 1, 2]

Reasoning

The input is a list of five real numbers, [-2, -1, 0, 1, 2], which you can picture as the pre-activation outputs of five neurons. The output is a list of the same length, where entry $i$ is $\text{ReLU}(x_i) = \max(0, x_i)$ .
Negative entries get clipped: $\max(0, -2) = 0$ and $\max(0, -1) = 0$ . Their sign and size are both lost.
Zero sits right on the boundary, so $\max(0, 0) = 0$ . Either branch of the definition gives the same value.
Positive entries pass through unchanged: $\max(0, 1) = 1$ and $\max(0, 2) = 2$ .
Putting these together in the original order gives [0, 0, 0, 1, 2].
Notice that the output is never negative and keeps the input's position order. Every value $\le 0$ collapses to the same $0$ , and that flattening of the negative half is the non-linearity. An empty input has nothing to transform, so it gives an empty output.

Constraints

Apply element-wise to input list

Learn this in

Deep LearningCV Study Plan

More from CV: Deep Learning

Implement Max Pooling with Indiceshard Depthwise Separable Convolutionmedium Softmax Cross-Entropy Gradientmedium Batch Normalization Forward Passhard Flatten Feature Mapeasy

Concept primer

ReLU (Rectified Linear Unit) is an activation function used at each neuron in many neural networks, especially in computer vision models like CNNs. It is defined elementwise as $\text{ReLU}(x) = \max(0, x),$ which means it outputs the input if it is positive and 0 otherwise. This simple nonlinearity helps networks model complex, non‑linear relationships while being computationally cheap to evaluate.

Using ReLU has two important effects in deep learning. First, it is piecewise linear and non-saturating for positive values, which helps mitigate the vanishing gradient problem that often occurs with sigmoid or tanh activations in deep networks. Gradients do not shrink to near-zero in the positive region, which makes optimization with gradient descent more stable and efficient. Second, ReLU introduces sparsity: for negative inputs, the output is exactly zero, so many neurons become inactive for a given input, which can improve efficiency and sometimes generalization.

1. Background Knowledge

Activation function role In a neural network layer, you typically compute $z = W x + b$ and then apply an activation $a =$ \phi $(z)$ . Without a nonlinear activation $\phi$ , stacking many layers would still represent a linear function overall. ReLU provides this nonlinearity.
ReLU vs other activations Compared to sigmoid/tanh:
ReLU is cheaper to compute (just a comparison and a max).
It avoids saturation for positive values (gradient is 1 there), helping gradients flow in deep networks.
It can cause “dead neurons” if weights push many inputs to be negative, but for this problem you mainly just need to implement the basic function.

2. Algorithm / Approach Pattern

For coding problems like “Implement ReLU activation”:

Treat the input as a number, vector, or tensor.
Apply the same scalar operation to each element:
If the element is greater than 0 → keep it.
Otherwise → set it to 0.
Return the result in the same shape as the input.

This is a simple elementwise transformation: no loops are needed if your language/library supports vectorized max or comparison operations.

3. Step‑by‑Step Strategy

Understand the mathematical definition ReLU is $f(x) = \max(0, x)$ applied elementwise.
Check the function signature

Is the input a scalar, list, array, or tensor?
You must return the same type/shape.

Implement scalar ReLU Pseudocode:

def relu_scalar(x):
if x > 0:
return x
else:
return 0

or more compactly:

return max(0, x)

Extend to arrays/tensors

If allowed, use vectorized APIs:

def relu(x):
# x is a NumPy array / tensor
return np.maximum(0, x)

If you must use loops, iterate over each element and apply the scalar ReLU.

Test with simple examples

Input: $[-1, 0, 2, 3.5]$ → Output: $[0, 0, 2, 3.5]$
Include purely negative, purely positive, and mixed cases.

4. Common Pitfalls

Forgetting elementwise behavior Applying ReLU only to part of the structure, or accidentally reducing (e.g., taking a single max over the whole array) instead of doing it per element.
Changing the shape Ensure you do not flatten, squeeze, or otherwise alter dimensions; only values should change.
Type issues
If the input is integer type and negative values become 0, that is fine, but be careful if your platform distinguishes between integer and float in a way that affects downstream operations.
Avoid returning Python lists when the function expects arrays/tensors (or vice versa).
Equality vs strict inequality Remember that $x = 0$ should map to 0; using > vs >= will not change the output at 0, but make sure your logic is consistent.

5. Time & Space Complexity

Assuming the input has $n$ elements:

Time complexity:
Each element requires a constant-time comparison and selection.
Overall: $O(n)$ .
Space complexity:
If you create a new array for the output: $O(n)$ extra space.
If you modify the input in place: $O(1)$ extra space (beyond the input storage).

solution.py

Python 3.13

Test results