LSTM Gate Computation

hardPROCNNs & Sequence ModelsMachine Learning 2

Compute the LSTM gates for one timestep.

Given previous hidden state $h_{t-1}$ , previous cell state $c_{t-1}$ , and input $x_t$ , compute all four gates by concatenating $[h_{t-1}, x_t]$ into a combined vector, then:

Forget gate: $f = \sigma(W_f \cdot [h, x] + b_f)$
Input gate: $i = \sigma(W_i \cdot [h, x] + b_i)$
Candidate: $\tilde{c} = \tanh(W_c \cdot [h, x] + b_c)$
Cell state: $c_t = f \odot c_{t-1} + i \odot \tilde{c}$
Output gate: $o = \sigma(W_o \cdot [h, x] + b_o)$
Hidden state: $h_t = o \odot \tanh(c_t)$

Return (h_t, c_t) both as lists rounded to 4 decimal places.

Example

Input

h_prev = [0]
c_prev = [0]
x = [1]
W_f = [[0.5, 0.5]]
b_f = [0]
W_i = [[0.5, 0.5]]
b_i = [0]
W_c = [[0.5, 0.5]]
b_c = [0]
W_o = [[0.5, 0.5]]
b_o = [0]

Output

([0.1743], [0.2876])

Reasoning

Inputs: hidden size 1, input size 1. The memory starts empty ( $h_{t-1} = [0]$ , $c_{t-1} = [0]$ ) and $x_t = [1]$ . Each weight matrix is $1 \times 2$ : one row per hidden unit, one column per entry of $[h_{t-1}, x_t] = [0, 1]$ . All biases are 0.
Gates: every gate gets the same pre-activation, $0.5 \cdot 0 + 0.5 \cdot 1 + 0 = 0.5$ . So $f = i = o = \sigma(0.5) = \frac{1}{1 + e^{-0.5}} \approx 0.6225$ , and the candidate is $\tilde{c} = \tanh 0.5 \approx 0.4621$ .
Cell state: $c_t = 0.6225 \cdot 0 + 0.6225 \cdot 0.4621 \approx 0.2876$ . The old cell was empty, so all of the memory comes from the candidate.
Hidden state: $\tanh 0.2876 \approx 0.2800$ , so $h_t = 0.6225 \cdot 0.2800 \approx 0.1743$ .
Output: ([0.1743], [0.2876]), two lists of length hidden_size, $h_t$ first. Since $o \in (0, 1)$ , $|h_t| \le |\tanh c_t|$ , with equality only when $c_t = 0$ . Round only at the end: multiplying the rounded 0.6225 and 0.4621 gives 0.2877.

Constraints

h_prev, c_prev: 1D lists (hidden_size)
x: 1D list (input_size)
W_f, W_i, W_c, W_o: 2D lists (hidden_size x (hidden_size + input_size))
b_f, b_i, b_c, b_o: 1D lists (hidden_size)
Return tuple (h_t, c_t) rounded to 4 decimal places

Learn this in

CNNs & Sequence ModelsML Study Plan NLP FundamentalsML Study Plan Sequence ModelsNLP Study Plan

More from Machine Learning 2

Kernel PCAhard PCA via Covariance Matrixmedium Variance Explained Ratioeasy Feature Scalingeasy GAN Discriminator Losshard

Concept primerPREVIEW

Background Knowledge

Long Short-Term Memory (LSTM) networks are a type of Recurrent Neural Network (RNN) designed to handle the vanishing gradient problem that occurs when training traditional RNNs. LSTMs are capable of learning long-term dependencies in data, making them particularly useful for sequence modeling tasks such as language translation, text generation, and speech recognition. The core component of an LSTM is the memory cell, which allows the network to store and retrieve information over long periods of time.

The LSTM gates are the key to controlling the flow of information into and out of the memory cell. There are four gates in total: the forget gate, input gate, candidate gate, and output gate. Each gate is responsible for a specific function: the forget gate determines what information to discard from the previous cell state, the input gate determines what new information to add to the cell state, the candidate gate generates a new candidate value for the cell state, and the output gate determines what information to output based on the current cell state and hidden state.

The mathematical operations involved in computing the LSTM gates are based on linear algebra and activation functions. The sigmoid function, $\sigma(x) = \frac{1}{1 + e^{-x}}$ , is used to introduce non-linearity into the model, while the hyperbolic tangent function, $\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$ , is used to generate the candidate value for the cell state. The element-wise multiplication operator, $\odot$ , is used to compute the Hadamard product of two vectors.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Compute the combined vector by concatenating the previous hidden state and input
Compute the four LSTM gates using the combined vector and the corresponding weights and biases
Compute the new cell state using the forget gate, input gate, and candidate gate
Compute the new hidden state using the output gate and the new cell state

This approach requires a good understanding of linear algebra, activation functions, and the mathematical operations involved in computing the LSTM gates.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results