LoRA Adapter Parameter Count

1. Background Knowledge

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique for large neural networks. Instead of updating every weight in a frozen matrix $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, LoRA learns a low-rank decomposition of the weight update $\Delta W$. The update is factored into two smaller matrices: $A \in \mathbb{R}^{r \times d_{\text{in}}}$ and $B \in \mathbb{R}^{d_{\text{out}} \times r}$, where $r \ll \min(d_{\text{in}}, d_{\text{out}})$. The effective weight becomes $W + B A$, and only $A$ and $B$ are trained.

The key insight is that a full-rank update requires $d_{\text{out}} \cdot d_{\text{in}}$ parameters, while the LoRA update requires only $r \cdot d_{\text{in}} + d_{\text{out}} \cdot r$ parameters. This is a massive reduction when $r$ is small (e.g., $r = 8$) relative to the matrix dimensions. For a $4096 \times 4096$ matrix, full fine-tuning needs $\approx 16.7$ million parameters, whereas a rank-8 LoRA adapter needs only $8 \times 4096 + 4096 \times 8 = 65{,}536$ parameters — roughly $0.4\%$ of the original.

This problem tests your understanding of how matrix dimensions translate into parameter counts. Each matrix contributes a number of trainable parameters equal to the product of its row and column dimensions. The total trainable parameter count is simply the sum of the parameter counts of all learned matrices.

2. Algorithm Approach

This is a direct computation problem. There is no search, iteration, or optimization involved. The approach is:

1. Identify the shapes of the two LoRA matrices: $A$ is $(r, d_{\text{in}})$ and $B$ is $(d_{\text{out}}, r)$.
2. Compute the number of elements (parameters) in each matrix by multiplying its dimensions.
3. Sum the two counts to get the total trainable parameters.

The formula is:

This can also be factored as $r \cdot (d_{\text{in}} + d_{\text{out}})$, which is algebraically equivalent and slightly more compact.

3. Step-by-Step Strategy