Orthogonal Initialization for RNNs
Problem Statement
Apply orthogonal initialization to recurrent network weights and verify the orthogonality property.
Background
Orthogonal initialization creates matrices where W^T W = I. This preserves gradient norms during backpropagation, making it ideal for RNNs.
Your Task
The starter code creates a square nn.Linear(4, 4) layer. Apply orthogonal initialization to the layer's weights. The verification code (computing W^T W, checking diagonal, computing singular values) is pre-filled.
Output Format
Returns a dictionary with "weight_shape", "wtw_diagonal", "is_orthogonal", and "singular_values".
Example:
None
{'weight_shape': [4, 4], 'wtw_diagonal': [1.0, 1.0, 1.0, 1.0], 'is_orthogonal': True, 'singular_values': [1.0, 1.0, 1.0, 1.0]}- We start by seeding the random number generator with
torch.manual_seed(42)to ensure reproducibility of the results. - A square matrix of size 4×4 is created using
nn.Linear(4, 4)$, and thennn.init.orthogonal_` is applied to the weight to make it orthogonal, meaning WTW=I where I is the identity matrix. - The matrix product WT@W is computed, and its diagonal is found to be approximately [1.0,1.0,1.0,1.0] since WTW is close to the identity matrix, and the off-diagonal elements are less than 0.001.
- The singular values of the weight matrix are calculated and rounded to 4 decimals, resulting in [1.0,1.0,1.0,1.0], confirming that the matrix is orthogonal, and the function returns a dictionary with the specified information.
Constraints:
- Use nn.init.orthogonal_
- Verify W^T W ≈ I
- Compute singular values with torch.linalg.svdvals
Background Knowledge
Orthogonal Initialization
Orthogonal initialization is a technique used to initialize the weights of neural networks, particularly recurrent neural networks (RNNs). The goal is to create a weight matrix W such that WTW=I, where I is the identity matrix. This property helps preserve the norm of the gradients during backpropagation, which can improve the stability and performance of the network.
Orthogonality and Linear Transformations
In linear algebra, an orthogonal matrix is a square matrix whose columns and rows are orthonormal vectors. This means that the matrix preserves the length and angle between vectors. When a matrix is orthogonal, its transpose is its inverse, i.e., WTW=I. This property is useful in neural networks because it helps to prevent the vanishing or exploding gradient problem.
Singular Value Decomposition (SVD)
The singular value decomposition (SVD) is a factorization technique that decomposes a matrix into three matrices: U, Σ, and V. The matrix U and V are orthogonal, and Σ is a diagonal matrix containing the singular values of the original matrix. The SVD is useful in analyzing the properties of a matrix, such as its rank, nullity, and orthogonality.
Algorithm/Approach
The approach to solving this problem involves the following general steps:
- Create a square matrix using nn.Linear
- Apply orthogonal initialization to the weight matrix
- Compute the matrix product WT@W and check its diagonal and off-diagonal elements
- Analyze the singular values of the weight matrix
- Return a dictionary containing the results of the analysis
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.