Activation Functions
Implement common activation functions and their derivatives.
Given a list of values and an activation name, compute both the activation output and its derivative:
- ReLU: f(x)=max(0,x), f′(x)=1 if x>0, else 0
- Sigmoid: f(x)=1+e−x1, f′(x)=f(x)(1−f(x))
- Tanh: f(x)=tanh(x), f′(x)=1−f(x)2
Return a tuple (outputs, derivatives), each rounded to 4 decimal places.
Example:
values = [-1, 0, 1] activation = "relu"
([0, 0, 1], [0, 0, 1])
- The input
valuesis[-1, 0, 1]and the chosenactivationis"relu", which has the function f(x)=max(0,x). - We apply the ReLU function to each value: f(−1)=max(0,−1)=0, f(0)=max(0,0)=0, f(1)=max(0,1)=1.
- Next, we calculate the derivatives: f′(−1)=0 since −1≤0, f′(0)=0 since 0 is not greater than 0, f′(1)=1 since 1>0.
- The final output is a tuple of the activation outputs and their derivatives, both rounded to 4 decimal places:
([0, 0, 1], [0, 0, 1]).
Constraints:
- values: list of floats
- activation: "relu", "sigmoid", or "tanh"
- Return tuple of two lists (outputs, derivatives)
- Round to 4 decimal places
Background Knowledge
Activation functions are crucial components in neural networks, as they introduce non-linearity to the model, enabling it to learn and represent more complex relationships between inputs and outputs. The most common activation functions are ReLU, Sigmoid, and Tanh. Each has its unique characteristics and is suited for different applications. For instance, ReLU is widely used in hidden layers due to its simplicity and computational efficiency, while Sigmoid and Tanh are often used in output layers for binary classification tasks.
The derivative of an activation function is essential for training neural networks using backpropagation, an algorithm that minimizes the loss function by adjusting the model's parameters. The derivative of the activation function is used to compute the gradient of the loss with respect to each parameter, which guides the optimization process. Understanding how to compute these derivatives is vital for implementing and training neural networks.
In the context of this problem, we need to implement the ReLU, Sigmoid, and Tanh activation functions and their derivatives. This involves understanding the mathematical formulas behind each function and how to compute their derivatives. For example, the derivative of ReLU is straightforward: it's 1 for positive inputs and 0 otherwise. In contrast, the derivatives of Sigmoid and Tanh involve the functions themselves, making their implementation slightly more complex.
Algorithm/Approach
The general approach to solving this problem involves defining functions for each activation type (ReLU, Sigmoid, Tanh) and their corresponding derivatives. We will then apply these functions to the given list of values and compute both the activation output and its derivative for each value. This can be achieved by iterating over the list of values and applying the appropriate activation function and its derivative based on the specified activation name.
Step-by-Step Strategy
- Define the activation functions (ReLU, Sigmoid, Tanh) and their derivatives based on their mathematical formulas.
- Initialize empty lists to store the activation outputs and derivatives for the given list of values.
- Iterate over the list of values and apply the specified activation function and its derivative to each value.
- Round the computed activation outputs and derivatives to 4 decimal places.
- Return a tuple containing the lists of activation outputs and derivatives.
Common Pitfalls
- Incorrect implementation of the activation functions or their derivatives.
- Failure to handle edge cases, such as division by zero in the Sigmoid function.
- Not rounding the results to the specified number of decimal places.
Time & Space Complexity
The time complexity of this solution is O(n), where n is the number of values in the input list, since we are applying the activation function and its derivative to each value once. The space complexity is also O(n), as we need to store the activation outputs and derivatives for each value.