📘

Merge a LoRA Update into the Base Weight

MediumVision-Language Models 📚 VLM 1: Vision-Language Models

Problem Statement

At inference LoRA is folded into the frozen weight so there is zero added latency: W' = W + (alpha/r) * B A. Implement the merge.

Background

LoRA scales its low-rank update by alpha/r (the "lora_alpha" over rank convention). Given the base weight W (d_out x d_in), factors A (r x d_in) and B (d_out x r), and alpha, the merged weight is

$W' = W + \frac{\alpha}{r}\, B A$

After merging, the adapter can be discarded — the model is a plain dense network again.

Your Task

Implement:

def merge_lora(W, A, B, alpha):

Return the merged d_out x d_in matrix as a nested list rounded to 4 decimals. Infer r from A's row count.

Input Format

W: d_out x d_in nested list.
A: r x d_in; B: d_out x r.
alpha (float).

Output Format

A d_out x d_in nested list rounded to 4 decimals.

Sample

W = [[1.0, 0.0], [0.0, 1.0]]
A = [[1.0, 1.0]]
B = [[1.0], [0.0]]
print(merge_lora(W, A, B, 2.0))

Output:

[[3.0, 2.0], [0.0, 1.0]]

Example:

Input:

W = [[1.0, 0.0], [0.0, 1.0]]
A = [[1.0, 1.0]]
B = [[1.0], [0.0]]
print(merge_lora(W, A, B, 2.0))

Output:

[[3.0, 2.0], [0.0, 1.0]]

Reasoning:

Infer the rank $r$ from the number of rows in matrix $A$ . Since $A$ is a $1 \times 2$ matrix, $r = 1$ .
Compute the low-rank update product $BA$ . Multiplying $B$ ( $2 \times 1$ ) by $A$ ( $1 \times 2$ ) yields a $2 \times 2$ matrix: $\begin{bmatrix} 1.0 \\ 0.0 \end{bmatrix} \begin{bmatrix} 1.0 & 1.0 \end{bmatrix} = \begin{bmatrix} 1.0 & 1.0 \\ 0.0 & 0.0 \end{bmatrix}$
Calculate the scaling factor $\frac{\alpha}{r}$ using the given $\alpha = 2.0$ and $r = 1$ : $\frac{2.0}{1} = 2.0$
Scale the product matrix by the factor to get the update term: $2.0 \times \begin{bmatrix} 1.0 & 1.0 \\ 0.0 & 0.0 \end{bmatrix} = \begin{bmatrix} 2.0 & 2.0 \\ 0.0 & 0.0 \end{bmatrix}$
Add this update to the base weight $W$ element-wise to obtain the merged weight $W'$ : $\begin{bmatrix} 1.0 & 0.0 \\ 0.0 & 1.0 \end{bmatrix} + \begin{bmatrix} 2.0 & 2.0 \\ 0.0 & 0.0 \end{bmatrix} = \begin{bmatrix} 3.0 & 2.0 \\ 0.0 & 1.0 \end{bmatrix}$
The final output is [[3.0, 2.0], [0.0, 1.0]]

Constraints:

W is d_out x d_in, A is r x d_in, B is d_out x r.
Scale the update by alpha / r where r = len(A).
Round to 4 decimals; avoid -0.0.

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention MassHard CHAIR-i and CHAIR-s Hallucination RatesMedium QLoRA Memory Footprint EstimateMedium POPE Precision, Recall, and F1Medium POPE Object Hallucination AccuracyEasy

1. Background Knowledge

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique where, instead of updating the full weight matrix $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ , we learn two smaller matrices $A \in \mathbb{R}^{r \times d_{\text{in}}}$ and $B \in \mathbb{R}^{d_{\text{out}} \times r}$ with rank $r \ll \min(d_{\text{out}}, d_{\text{in}})$ . The effective update is the product $BA$ , which has rank at most $r$ . During training, the forward pass computes $Wx + \frac{\alpha}{r}BAx$ ; at inference, the adapter is merged into the base weight so the model becomes a plain dense network with zero added latency.

The scaling factor $\frac{\alpha}{r}$ (where $\alpha$ is the hyperparameter lora_alpha) controls the magnitude of the update relative to the rank. A larger $\alpha$ or smaller $r$ amplifies the low-rank contribution. This convention keeps the effective scale stable as you change the rank.

Merging is a one-time offline operation: compute $W' = W + \frac{\alpha}{r}BA$ and discard $A$ and $B$ . The result is a full-rank dense matrix identical in shape to the original $W$ .

2. Algorithm Approach

This is a matrix multiplication followed by element-wise addition. The core operation is:

Compute the product $C = B \cdot A$ , which yields a $d_{\text{out}} \times d_{\text{in}}$ matrix.
Scale $C$ by the factor $\frac{\alpha}{r}$ .
Add the scaled matrix element-wise to $W$ .

Since the inputs are nested lists (not NumPy arrays), you implement the matrix product manually using triple-nested loops: for each output row $i$ and column $j$ , sum over the shared dimension $k$ (which ranges over $r$ ).

3. Step-by-Step Strategy

Infer the rank: $r = \text{len}(A)$ , i.e., the number of rows in $A$ .
Compute the product $BA$ :
For each row $i$ of $B$ (range $d_{\text{out}}$ ):
For each column $j$ of $A$ (range $d_{\text{in}}$ ):
Initialize a running sum to $0$ .
For each $k$ in range $r$ : accumulate $B[i][k] \times A[k][j]$ .
Store the result in a temporary matrix $C[i][j]$ .
Scale and merge:
Compute scale = alpha / r.
For each element, compute W[i][j] + scale * C[i][j].
Round: Apply round(value, 4) to every element of the result.
Return the nested list.

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.