Orthogonal Initialization for RNNs
Problem Statement
Apply orthogonal initialization to recurrent network weights and verify the orthogonality property.
Background
Orthogonal initialization creates matrices where W^T W = I. This preserves gradient norms during backpropagation, making it ideal for RNNs.
Your Task
The starter code creates a square nn.Linear(4, 4) layer. Apply orthogonal initialization to the layer's weights. The verification code (computing W^T W, checking diagonal, computing singular values) is pre-filled.
Output Format
Returns a dictionary with "weight_shape", "wtw_diagonal", "is_orthogonal", and "singular_values".
Example
Input
None
Output
{'weight_shape': [4, 4], 'wtw_diagonal': [1.0, 1.0, 1.0, 1.0], 'is_orthogonal': True, 'singular_values': [1.0, 1.0, 1.0, 1.0]}Reasoning
- We start by seeding the random number generator with
torch.manual_seed(42)to ensure reproducibility of the results. - A square matrix of size is created using
nn.Linear(4, 4)$, and thennn.init.orthogonal_` is applied to the weight to make it orthogonal, meaning where is the identity matrix. - The matrix product is computed, and its diagonal is found to be approximately since is close to the identity matrix, and the off-diagonal elements are less than .
- The singular values of the weight matrix are calculated and rounded to 4 decimals, resulting in , confirming that the matrix is orthogonal, and the function returns a dictionary with the specified information.
Constraints
- Use nn.init.orthogonal_
- Verify W^T W ≈ I
- Compute singular values with torch.linalg.svdvals