Autoencoder Bottleneck

easyFREEGenerative ModelsMachine Learning 2

Implement the forward pass of a simple autoencoder with one hidden (bottleneck) layer.

An autoencoder has:

Encoder: $z = \text{ReLU}(W_e \cdot x + b_e)$ — compress input to bottleneck
Decoder: $\hat{x} = W_d \cdot z + b_d$ — reconstruct from bottleneck

Given input $x$ , encoder weights/bias, and decoder weights/bias, compute both the bottleneck representation $z$ and the reconstruction $\hat{x}$ .

Return a tuple (z, x_hat) both as lists rounded to 4 decimal places.

Example

Input

x = [1, 2, 3]
W_e = [[0.5, 0.5, 0.5]]
b_e = [0]
W_d = [[1], [1], [1]]
b_d = [0, 0, 0]

Output

([3.0], [3.0, 3.0, 3.0])

Reasoning

Inputs: $x = [1, 2, 3]$ has $d_{in} = 3$ features. $W_e$ is a $1 \times 3$ matrix, so the bottleneck has a single unit ( $d_{hidden} = 1$ ) and $b_e = [0]$ . $W_d$ is $3 \times 1$ : it maps that one unit back to 3 features, with $b_d = [0, 0, 0]$ .
Encoder: the pre-activation is $0.5 \cdot 1 + 0.5 \cdot 2 + 0.5 \cdot 3 + 0 = 0.5 + 1 + 1.5 = 3$ . ReLU keeps positive values unchanged, $\max(0, 3) = 3$ , so $z$ = [3.0].
Decoder: each row of $W_d$ holds a single weight $1$ , so $\hat{x}_i = 1 \cdot 3 + 0 = 3$ for $i = 1, 2, 3$ , giving [3.0, 3.0, 3.0]. The decoder has no ReLU.
Output: the tuple $(z, \hat{x})$ with lengths $d_{hidden}$ and $d_{in}$ , every entry rounded to 4 decimal places.
Notice: the reconstruction is not $[1, 2, 3]$ . One number can only remember a weighted total of the input, so the differences between features are lost. A negative pre-activation would make that unit $0$ .

Constraints

x: 1D list (d_in)
W_e: 2D list (d_hidden x d_in), b_e: 1D list (d_hidden)
W_d: 2D list (d_in x d_hidden), b_d: 1D list (d_in)
Return (z, x_hat) both rounded to 4 decimal places

Learn this in

Introduction to Generative ModelsDiffusion Study Plan Latent Diffusion & Stable DiffusionDiffusion Study Plan Generative & Production MLML Study Plan Sequence ModelsNLP Study Plan

More from Machine Learning 2

Kernel PCAhard PCA via Covariance Matrixmedium Variance Explained Ratioeasy Feature Scalingeasy GAN Discriminator Losshard

Concept primer

Background Knowledge

An autoencoder is a type of neural network that is used for unsupervised learning, dimensionality reduction, and generative modeling. It consists of two main components: the encoder and the decoder. The encoder maps the input to a lower-dimensional representation, known as the bottleneck or latent representation, while the decoder maps this representation back to the original input space. The goal of an autoencoder is to learn a compact and informative representation of the input data.

The encoder and decoder are typically implemented using fully connected (dense) layers, and the activation function used in the encoder is often a non-linear function, such as the ReLU (Rectified Linear Unit). The ReLU function maps all negative values to 0 and all positive values to the same value, which helps to introduce non-linearity into the model. The decoder, on the other hand, typically uses a linear activation function, which means that the output is a linear transformation of the input.

In the context of this problem, we are given the encoder weights (W_e), encoder bias (b_e), decoder weights (W_d), and decoder bias (b_d), and we need to compute the bottleneck representation (z) and the reconstruction (x_hat) of the input (x). This requires applying the forward pass of the autoencoder, which involves propagating the input through the encoder and decoder layers.

Algorithm/Approach

The general approach to solving this type of problem involves applying the forward pass of the autoencoder, which can be broken down into the following steps:

Apply the encoder to the input (x) to compute the bottleneck representation (z)
Apply the decoder to the bottleneck representation (z) to compute the reconstruction (x_hat)

This approach requires using the given encoder weights, encoder bias, decoder weights, and decoder bias to compute the bottleneck representation and reconstruction.

Step-by-Step Strategy

To implement the solution, follow these steps:

Compute the bottleneck representation (z) by applying the encoder to the input (x): z = ReLU(W_e * x + b_e)
Compute the reconstruction (x_hat) by applying the decoder to the bottleneck representation (z): x_hat = W_d * z + b_d
Return the bottleneck representation (z) and the reconstruction (x_hat) as lists rounded to 4 decimal places

Common Pitfalls

When implementing the solution, watch out for the following common pitfalls:

Forgetting to apply the ReLU activation function in the encoder
Using the wrong weights or bias for the encoder or decoder
Not rounding the bottleneck representation and reconstruction to 4 decimal places

Time & Space Complexity

The time complexity of the solution is O(n), where n is the number of elements in the input (x), since we need to compute the bottleneck representation and reconstruction for each element. The space complexity is also O(n), since we need to store the bottleneck representation and reconstruction.

Example code to get you started:

import numpy as np

def autoencoder_forward_pass(x, W_e, b_e, W_d, b_d):
 # Compute the bottleneck representation (z)
 z = np.maximum(np.dot(W_e, x) + b_e, 0) # Apply ReLU activation function
 
 # Compute the reconstruction (x_hat)
 x_hat = np.dot(W_d, z) + b_d
 
 # Return the bottleneck representation and reconstruction as lists rounded to 4 decimal places
 return z.round(4).tolist(), x_hat.round(4).tolist()

solution.py

Python 3.13

Test results