Merge a LoRA Update into the Base Weight

mediumPROEfficient Adaptation & Hallucination MetricsVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

At inference LoRA is folded into the frozen weight so there is zero added latency: W' = W + (alpha/r) * B A. Implement the merge.

Background

LoRA scales its low-rank update by alpha/r (the "lora_alpha" over rank convention). Given the base weight W (d_out x d_in), factors A (r x d_in) and B (d_out x r), and alpha, the merged weight is

$W' = W + \frac{\alpha}{r}\, B A$

After merging, the adapter can be discarded — the model is a plain dense network again.

Your Task

Implement:

def merge_lora(W, A, B, alpha):

Return the merged d_out x d_in matrix as a nested list rounded to 4 decimals. Infer r from A's row count.

Input Format

W: d_out x d_in nested list.
A: r x d_in; B: d_out x r.
alpha (float).

Output Format

A d_out x d_in nested list rounded to 4 decimals.

Sample

W = [[1.0, 0.0], [0.0, 1.0]]
A = [[1.0, 1.0]]
B = [[1.0], [0.0]]
print(merge_lora(W, A, B, 2.0))

Output:

[[3.0, 2.0], [0.0, 1.0]]

Example

Input

W = [[1.0, 0.0], [0.0, 1.0]]
A = [[1.0, 1.0]]
B = [[1.0], [0.0]]
print(merge_lora(W, A, B, 2.0))

Output

[[3.0, 2.0], [0.0, 1.0]]

Reasoning

Infer the rank $r$ from the number of rows in matrix $A$ . Since $A$ is a $1 \times 2$ matrix, $r = 1$ .
Compute the low-rank update product $BA$ . Multiplying $B$ ( $2 \times 1$ ) by $A$ ( $1 \times 2$ ) yields a $2 \times 2$ matrix: $\begin{bmatrix} 1.0 \\ 0.0 \end{bmatrix} \begin{bmatrix} 1.0 & 1.0 \end{bmatrix} = \begin{bmatrix} 1.0 & 1.0 \\ 0.0 & 0.0 \end{bmatrix}$
Calculate the scaling factor $\frac{\alpha}{r}$ using the given $\alpha = 2.0$ and $r = 1$ : $\frac{2.0}{1} = 2.0$
Scale the product matrix by the factor to get the update term: $2.0 \times \begin{bmatrix} 1.0 & 1.0 \\ 0.0 & 0.0 \end{bmatrix} = \begin{bmatrix} 2.0 & 2.0 \\ 0.0 & 0.0 \end{bmatrix}$
Add this update to the base weight $W$ element-wise to obtain the merged weight $W'$ : $\begin{bmatrix} 1.0 & 0.0 \\ 0.0 & 1.0 \end{bmatrix} + \begin{bmatrix} 2.0 & 2.0 \\ 0.0 & 0.0 \end{bmatrix} = \begin{bmatrix} 3.0 & 2.0 \\ 0.0 & 1.0 \end{bmatrix}$
The final output is [[3.0, 2.0], [0.0, 1.0]]

Constraints

W is d_out x d_in, A is r x d_in, B is d_out x r.
Scale the update by alpha / r where r = len(A).
Round to 4 decimals; avoid -0.0.

Learn this in

Fine-tuningLLM Study Plan Parameter-Efficient Fine-TuningVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium POPE Object Hallucination Accuracyeasy

Concept primerPREVIEW

1. Background Knowledge

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique where, instead of updating the full weight matrix $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ , we learn two smaller matrices $A \in \mathbb{R}^{r \times d_{\text{in}}}$ and $B \in \mathbb{R}^{d_{\text{out}} \times r}$ with rank $r \ll \min(d_{\text{out}}, d_{\text{in}})$ . The effective update is the product $BA$ , which has rank at most $r$ . During training, the forward pass computes $Wx + \frac{\alpha}{r}BAx$ ; at inference, the adapter is merged into the base weight so the model becomes a plain dense network with zero added latency.

The scaling factor $\frac{\alpha}{r}$ (where $\alpha$ is the hyperparameter lora_alpha) controls the magnitude of the update relative to the rank. A larger $\alpha$ or smaller $r$ amplifies the low-rank contribution. This convention keeps the effective scale stable as you change the rank.

Merging is a one-time offline operation: compute $W' = W + \frac{\alpha}{r}BA$ and discard $A$ and $B$ . The result is a full-rank dense matrix identical in shape to the original $W$ .

2. Algorithm Approach

This is a matrix multiplication followed by element-wise addition. The core operation is:

Compute the product $C = B \cdot A$ , which yields a $d_{\text{out}} \times d_{\text{in}}$ matrix.
Scale $C$ by the factor $\frac{\alpha}{r}$ .
Add the scaled matrix element-wise to $W$ .

Since the inputs are nested lists (not NumPy arrays), you implement the matrix product manually using triple-nested loops: for each output row $i$ and column $j$ , sum over the shared dimension $k$ (which ranges over $r$ ).

3. Step-by-Step Strategy

Infer the rank: $r = \text{len}(A)$ , i.e., the number of rows in $A$ .
Compute the product $BA$ :
For each row $i$ of $B$ (range $d_{\text{out}}$ ):
For each column $j$ of $A$ (range $d_{\text{in}}$ ):
Initialize a running sum to $0$ .
For each $k$ in range $r$ : accumulate $B[i][k] \times A[k][j]$ .
Store the result in a temporary matrix $C[i][j]$ .
Scale and merge:
Compute scale = alpha / r.
For each element, compute W[i][j] + scale * C[i][j].
Round: Apply round(value, 4) to every element of the result.
Return the nested list.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.