RNN Forward Pass
Implement a simple RNN forward pass (Elman RNN) using NumPy.
At each time step t, the RNN computes: ht=tanh(Wxh⋅xt+Whh⋅ht−1+bh)
Where:
- xt is the input at time step t
- ht−1 is the previous hidden state
- Wxh is the input-to-hidden weight matrix
- Whh is the hidden-to-hidden weight matrix
- bh is the hidden bias
Input format:
- Line 1: input_size hidden_size seq_length (space-separated ints)
- Next input_size lines: W_xh matrix (hidden_size columns per line)
- Next hidden_size lines: W_hh matrix (hidden_size columns per line)
- Next line: b_h (hidden_size values)
- Next seq_length lines: one input vector per time step (input_size values)
- Next line: h_0 initial hidden state (hidden_size values)
Output: The final hidden state as a list, rounded to 4 decimal places.
Example:
2 3 2 0.1 0.2 0.3 0.4 0.5 0.6 0.1 0.0 0.1 0.0 0.1 0.0 0.1 0.0 0.1 0.01 0.02 0.03 1.0 0.5 0.5 1.0 0.0 0.0 0.0
[0.2729, 0.1567, 0.3878]
Time step 1: x=[1.0, 0.5] W_xh @ x = [0.11+0.40.5, 0.21+0.50.5, 0.31+0.60.5] = [0.3, 0.45, 0.6] W_hh @ h_0 = [0, 0, 0] (h_0 is zeros) h_1 = tanh([0.3+0+0.01, 0.45+0+0.02, 0.6+0+0.03]) = tanh([0.31, 0.47, 0.63]) h_1 = [0.3004, 0.4382, 0.5581]
Time step 2: x=[0.5, 1.0] Similarly compute W_xh @ x + W_hh @ h_1 + b_h, then tanh. Final h_2 = [0.2729, 0.1567, 0.3878]
Constraints:
- Use numpy for matrix operations
- Use np.tanh for activation
- Initial hidden state h_0 is provided
- Round final hidden state to 4 decimal places
Background Knowledge
The problem involves implementing a simple Recurrent Neural Network (RNN) forward pass, specifically an Elman RNN. RNNs are a type of neural network designed to handle sequential data, such as time series data or natural language processing tasks. The key characteristic of RNNs is their ability to maintain a hidden state that captures information from previous time steps, allowing them to keep track of context and make predictions based on that context.
In the context of the Elman RNN, the hidden state ht at time step t is computed using the previous hidden state ht−1, the current input xt, and the weights and biases of the network. The tanh activation function is used to introduce non-linearity into the model. The weights and biases are learned during training and are used to compute the hidden state at each time step. Understanding how to compute the hidden state and how to use it to make predictions is crucial to solving this problem.
The problem also involves linear algebra operations, such as matrix multiplication, which are used to compute the hidden state. Specifically, the input-to-hidden weight matrix Wxh and the hidden-to-hidden weight matrix Whh are used to compute the hidden state. The hidden bias bh is also added to the computation. Understanding how to perform these linear algebra operations using NumPy is essential to implementing the solution.
Algorithm/Approach
The general approach to solving this problem involves implementing the Elman RNN forward pass using NumPy. This involves:
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.