Dense Layer Forward

mediumPRONeural NetworksDeep LearningCV: Deep Learning

Implement forward pass of a dense (fully connected) layer, a fundamental component in Neural Networks. This process is crucial for understanding how artificial neurons process inputs to produce meaningful outputs.

In the context of Neural Networks, a dense layer is where every input is connected to every output by a weight matrix $W$ and a bias vector $b$ . The forward pass involves computing the output of each neuron by taking the dot product of the input vector $x$ with the corresponding weights and adding the bias term.

To compute the output, follow these steps:

Initialize output vector $y$
For each output neuron, compute the weighted sum of inputs
Add the bias term to the weighted sum

\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}

This technique is widely used in image classification tasks.

Example

Input

dense_forward([[1,2],[3,4]], [1,1], [0,0])

Output

[3, 7]

Reasoning

Inputs: W is a $2 \times 2$ weight matrix $\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}$ where row $i$ holds the weights feeding output neuron $i$ , so it has one row per output and one column per input. x $= [1, 1]$ is the input vector and b $= [0, 0]$ gives one bias per output neuron.
Neuron 1: take row 1 of $W$ against $\mathbf{x}$ : $y_1 = 1 \cdot 1 + 2 \cdot 1 + 0 = 3$ .
Neuron 2: row 2 against $\mathbf{x}$ : $y_2 = 3 \cdot 1 + 4 \cdot 1 + 0 = 7$ .
Output: $\mathbf{y} = [3, 7]$ , a vector whose length equals the number of rows of $W$ , not the length of $\mathbf{x}$ . Rounding to 4 decimal places leaves these whole numbers unchanged.
Notice: with $\mathbf{x}$ all ones, each $y_i$ is just the sum of row $i$ of $W$ . A non-zero bias would simply shift each output, e.g. $\mathbf{b} = [1, 1]$ would give $[4, 8]$ .

Constraints

W is a 2D weight matrix
x is input vector
b is bias vector
Return output rounded to 4 decimal places

Learn this in

Deep LearningCV Study Plan

More from CV: Deep Learning

Implement Max Pooling with Indiceshard Depthwise Separable Convolutionmedium Softmax Cross-Entropy Gradientmedium Batch Normalization Forward Passhard Flatten Feature Mapeasy

Concept primerPREVIEW

Background Knowledge

A dense (fully connected) layer is the basic building block of many neural networks. It takes an input vector $x \in \mathbb{R}^{n_{$ \text{in} $}}$ and produces an output vector $y \in \mathbb{R}^{n_{$ \text{out} $}}$ using a weight matrix $W \in \mathbb{R}^{n_{$ \text{out} $} \times n_{$ \text{in} $}}$ and a bias vector $b \in \mathbb{R}^{n_{$ \text{out} $}}$ . Each output neuron computes a weighted sum of all input features plus a bias term. This linear transformation is written as:

y = Wx + b

This is the pre-activation step; in a full network it’s often followed by a nonlinear activation function (e.g., ReLU), but here you only care about the linear forward pass.

In practice, we often process a batch of inputs at once. If you stack batch examples as rows in a matrix $X \in \mathbb{R}^{B \times n_{$ \text{in} $}}$ , the dense layer applies the same weights and biases to every row. The batched forward pass is:

Y = XW^\top + b

where $Y \in \mathbb{R}^{B \times n_{$ \text{out} $}}$ and $b$ is broadcast (added to every row).

Algorithm / General Approach

The pattern for a dense layer forward pass is:

Represent parameters as:
Weights: matrix connecting every input unit to every output unit.
Biases: one bias per output unit.
For each input example:
Compute a matrix–vector (or matrix–matrix for a batch) multiplication for the linear part.
Add the bias term to the result.
Use efficient vectorized operations instead of explicit Python loops wherever possible.

So the algorithm is essentially:

Linear transform via multiplication.
Add bias with correct broadcasting.
Return the result.

Step-by-Step Strategy

Assume:

Input shape (single example): (in_features,)
Weight matrix shape: (out_features, in_features)
Bias vector shape: (out_features,)
Output shape: (out_features,) (or (batch_size, out_features) for batched inputs)

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results