Dense Layer Forward
Implement forward pass of a dense (fully connected) layer, a fundamental component in Neural Networks. This process is crucial for understanding how artificial neurons process inputs to produce meaningful outputs.
In the context of Neural Networks, a dense layer is where every input is connected to every output by a weight matrix W and a bias vector b. The forward pass involves computing the output of each neuron by taking the dot product of the input vector x with the corresponding weights and adding the bias term.
To compute the output, follow these steps:
- Initialize output vector y
- For each output neuron, compute the weighted sum of inputs
- Add the bias term to the weighted sum
This technique is widely used in image classification tasks.
Example:
dense_forward([[1,2],[3,4]], [1,1], [0,0])
[3, 7]
- Interpret inputs as: weight matrix W=[13​24​], input vector x=[11​], bias vector b=[00​].
- Compute Wx:
- First output: 1â‹…1+2â‹…1=3
- Second output: 3â‹…1+4â‹…1=7
- Add bias b (which is zero), so the final output remains [3,7].
Constraints:
- W is a 2D weight matrix
- x is input vector
- b is bias vector
- Return output rounded to 4 decimal places
- Background Knowledge
A dense (fully connected) layer is the basic building block of many neural networks. It takes an input vector x \in \mathbb{R}^{n_{\text{in}}} and produces an output vector y \in \mathbb{R}^{n_{\text{out}}} using a weight matrix W \in \mathbb{R}^{n_{\text{out}} \times n_{\text{in}}} and a bias vector b \in \mathbb{R}^{n_{\text{out}}}. Each output neuron computes a weighted sum of all input features plus a bias term. This linear transformation is written as:
y=Wx+bThis is the pre-activation step; in a full network it’s often followed by a nonlinear activation function (e.g., ReLU), but here you only care about the linear forward pass.
In practice, we often process a batch of inputs at once. If you stack batch examples as rows in a matrix X \in \mathbb{R}^{B \times n_{\text{in}}}, the dense layer applies the same weights and biases to every row. The batched forward pass is:
Y=XW⊤+bwhere Y \in \mathbb{R}^{B \times n_{\text{out}}} and b is broadcast (added to every row).
- Algorithm / General Approach
The pattern for a dense layer forward pass is:
- Represent parameters as:
- Weights: matrix connecting every input unit to every output unit.
- Biases: one bias per output unit.
- For each input example:
- Compute a matrix–vector (or matrix–matrix for a batch) multiplication for the linear part.
- Add the bias term to the result.
- Use efficient vectorized operations instead of explicit Python loops wherever possible.
So the algorithm is essentially:
- Linear transform via multiplication.
- Add bias with correct broadcasting.
- Return the result.
- Step-by-Step Strategy
Assume:
- Input shape (single example): (in_features,)
- Weight matrix shape: (out_features, in_features)
- Bias vector shape: (out_features,)
- Output shape: (out_features,) (or (batch_size, out_features) for batched inputs)
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.