📘

Inverted Dropout Forward

Problem Statement

Implement the forward pass of Inverted Dropout.

Background

Dropout is a regularization technique that randomly "drops" neurons during training to prevent overfitting. In Inverted Dropout:

Each neuron is dropped with probability $p$ (drop_prob)
Remaining neurons are scaled by $\frac{1}{1 - p}$ to keep expected values the same
During inference (test mode), no dropout is applied

The scaling ensures that expected output values are consistent between training and testing.

Your Task

Write a function dropout_forward(activations, drop_prob, mask, train_mode=True) that:

If train_mode=False: return activations unchanged
If train_mode=True: apply the given mask (0s and 1s) and scale by 1/(1-drop_prob)

The mask is provided as input (list of 0s and 1s) to make testing deterministic.

Output Format

Return a list of values after applying dropout. Round each value to 4 decimal places.

Example:

Input:

activations=[10, 20, 30], drop_prob=0.5, mask=[1, 0, 1], train_mode=True

Output:

[20.0, 0, 60.0]

Reasoning:

Scale = 1/(1-0.5) = 2. First: $10 \times 1$ ×2=20. Second: $20 \times 0$ =0. Third: $30 \times 1$ ×2=60.

Constraints:

0 < drop_prob < 1
len(activations) == len(mask)
mask contains only 0s and 1s
List length: 1 to 100 elements

More from Deep Learning 1

Early Stopping CheckEasy L2 Regularization PenaltyEasy Binary Cross-Entropy LossEasy Momentum Update StepMedium SGD Parameter UpdateEasy

1. Background Knowledge

Dropout is a regularization technique introduced by Srivastava et al. (2014) that randomly sets neuron activations to zero during training with probability $p$ (drop probability), preventing co-adaptation of features and reducing overfitting. Inverted Dropout, a practical variant, scales surviving neurons by $\frac{1}{1-p}$ during training to maintain consistent expected values between training and inference phases.

Key mathematical insight: For input $x_i$ , the dropout operation is:

y_i = \begin{cases} 0 & \text{with probability } p \\ \frac{x_i}{1-p} & \text{with probability } 1-p \end{cases}

This ensures $\mathbb{E}[y_i] = x_i$ , preserving expected output magnitude.

Prerequisites:

Understanding of neural network forward pass
Basic probability (Bernoulli masking)
List operations in Python
Concept of train vs. inference modes

2. Algorithm Approach

Deterministic masked forward pass with conditional scaling:

Inference mode (train_mode=False): Identity function $y = x$
Training mode (train_mode=True): Element-wise multiplication with mask, then scaling: $y_i = x_i \cdot mask_i \cdot \frac{1}{1-p}$

The provided mask (0s and 1s) simulates the random Bernoulli sampling, making the operation deterministic for testing.

Pseudocode:

if not train_mode:
 return activations
else:
 scale = 1 / (1 - drop_prob)
 return [a * m * scale for a, m in zip(activations, mask)]

3. Step-by-Step Strategy

Check training mode: If train_mode=False, return activations unchanged
Compute scale factor: scale = 1 / (1 - drop_prob)
Apply element-wise operations:

For each index $i$ : result[i] = activations[i] * mask[i] * scale

Round to 4 decimal places: round(value, 4) for each element
Return result list

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.