LoRA Weight Decomposition

easyFREEFine-Tuning MechanicsLLM 2: Training & Alignment

Implement LoRA (Low-Rank Adaptation) weight decomposition.

LoRA decomposes a weight update ΔW into two low-rank matrices: $\Delta W = A \times B$

where A has shape (d, r) and B has shape (r, d), with r << d being the rank.

Given d (dimension) and r (rank), initialize A from a normal distribution (seed 42) and B as zeros. Compute the effective weight update ΔW = A × B.

Input: d r (dimension, rank) Output: The ΔW matrix (d × d), rounded to 4 decimal places.

Since B is initialized to zeros, ΔW should be all zeros initially. Then set B = np.random.randn(r, d) * 0.01 (using the same seed state after A) and recompute.

Example

Input

3 2

Output

[[-0.0014  0.0006 -0.0112]
 [-0.0090  0.0082  0.0035]
 [ 0.0063 -0.0048 -0.0035]]

Reasoning

Initialize $A$ with shape $(d, r) = (3, 2)$ from a normal distribution with seed 42, and $B$ with shape $(r, d) = (2, 3)$ as zeros.
Compute the initial $\Delta W = A \times B$ , which results in a $(3, 3)$ matrix of all zeros, since $B$ is all zeros.
Update $B$ with $B = \text{np.random.randn}(r, d) \cdot 0.01$ , using the same seed state after $A$ , to get a new $(2, 3)$ matrix.
Recompute $\Delta W = A \times B$ using the updated $B$ to get the final $(3, 3)$ matrix, which is then rounded to 4 decimal places to produce the output.

Constraints

np.random.seed(42), A = np.random.randn(d, r), B = np.random.randn(r, d) * 0.01
Output the ΔW = A @ B matrix
Round to 4 decimal places

Learn this in

Mathematical FoundationsFoundations Fine-tuningLLM Study Plan Text ClassificationNLP Study Plan Parameter-Efficient Fine-TuningVLM Study Plan

More from LLM 2: Training & Alignment

RLHF Reward Shapinghard Implicit Reward from DPOmedium PPO Clipped Objectivemedium DPO Lossmedium KL Divergence Penaltyeasy

Concept primer

Background Knowledge

Low-Rank Adaptation (LoRA) is a technique used in fine-tuning large language models. It's based on the idea of decomposing weight updates into low-rank matrices, which helps reduce the number of parameters that need to be updated. This is particularly useful when fine-tuning pre-trained models, as it allows for more efficient adaptation to new tasks or datasets. The decomposition of the weight update ΔW into two low-rank matrices A and B is a key concept in LoRA, where A has shape (d, r) and B has shape (r, d), with r being the rank and r << d.

The rank of a matrix is a fundamental concept in linear algebra, referring to the maximum number of linearly independent rows or columns in the matrix. In the context of LoRA, the rank r is a hyperparameter that controls the complexity of the weight update. A lower rank means a more compact representation of the update, which can lead to faster computation and reduced memory usage. However, it may also limit the expressiveness of the update. The choice of rank depends on the specific application and the trade-off between computational efficiency and model performance.

In this problem, we're working with matrix multiplication, which is a basic operation in linear algebra. The product of two matrices A and B, denoted as A × B, results in a new matrix whose elements are computed by taking the dot product of rows from A with columns from B. This operation is fundamental to many machine learning algorithms, including LoRA. Understanding matrix multiplication and its properties is essential for implementing LoRA weight decomposition.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Initialize the matrices A and B according to the given specifications.
Compute the matrix product A × B to obtain the effective weight update ΔW.
Update the matrix B and recompute the matrix product to obtain the new weight update.

This approach requires a basic understanding of linear algebra, including matrix multiplication and initialization of matrices from specific distributions.

Step-by-Step Strategy

To implement the solution, follow these steps:

Import the necessary libraries, including NumPy for matrix operations.
Initialize the matrix A from a normal distribution with the specified seed.
Initialize the matrix B as zeros.
Compute the matrix product A × B to obtain the initial weight update ΔW.
Round the result to 4 decimal places.
Update the matrix B using the specified formula and seed state.
Recompute the matrix product A × B to obtain the new weight update.

Common Pitfalls

When implementing the solution, watch out for the following:

Ensure that the seed state is preserved when updating the matrix B.
Use the correct shapes for the matrices A and B.
Verify that the matrix product is computed correctly.

Time & Space Complexity

The time complexity of the solution is O(d^2 * r), where d is the dimension and r is the rank. This is because the matrix product A × B involves iterating over the elements of the matrices, which has a quadratic complexity in terms of the dimension. The space complexity is O(d^2 + d * r), which is the total memory required to store the matrices A, B, and ΔW.

solution.py

Python 3.13

Test results