Compare Initialization Strategies
Problem Statement
Compare the effect of different initialization strategies on a network's first forward pass.
Background
Zero initialization kills the network (all neurons output the same value). Xavier works for Sigmoid/Tanh. Kaiming is best for ReLU networks.
Your Task
The starter code creates three identical models. Initialize them with three different strategies: zeros (both weights and biases), Xavier uniform (weights only), and Kaiming normal for ReLU (weights only). Apply each to its respective model. The forward passes and comparison are pre-filled.
Output Format
Returns a dictionary with "zeros_output_std", "xavier_output_std", "kaiming_output_std", and "zeros_dead".
Example
Input
None
Output
{'zeros_output_std': 0.0, 'xavier_output_std': 0.1729, 'kaiming_output_std': 0.6439, 'zeros_dead': True}Reasoning
- The function
compare_init_test()creates three copies of a neural network model withtorch.manual_seed(42)for reproducibility. - It initializes each model with different strategies:
model_zeroswith all weights set to 0,model_xavierwith Xavier uniform initialization, andmodel_kaimingwith Kaiming normal initialization. - The input
torch.ones(1, 10)is passed through each model, and the standard deviation of the output is calculated: sincemodel_zeroshas all weights as 0, its output will be 0, resulting in a standard deviation of 0.0 and a "dead" network. - The standard deviations of
model_xavierandmodel_kaimingoutputs are calculated as 0.1729 and 0.6439, respectively, after passing the input through each model and applying the respective initialization strategies.
Constraints
- Same architecture for all three
- Use torch.manual_seed(42) before each model
- Compare output statistics