LoRA Trainable Parameter Savings

1. Background Knowledge

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique designed to reduce the memory and compute requirements of adapting large pretrained models. Instead of updating all weights in a model, LoRA freezes the original pretrained weight matrix $W$ and injects trainable low-rank decomposition matrices. The update is formulated as $W' = W + \frac{\alpha}{r} BA$, where $B$ and $A$ are low-rank matrices. This approach assumes that the update to the weights during fine-tuning has an "intrinsic low rank," meaning the changes can be captured by a small number of dimensions.

The core mathematical insight is that a full rank update to a matrix of shape $(d_{out}, d_{in})$ requires $d_{out} \times d_{in}$ parameters. In contrast, the LoRA update uses two matrices: $A$ of shape $(r, d_{in})$ and $B$ of shape $(d_{out}, r)$. The total number of trainable parameters for this single matrix becomes $r \cdot d_{in} + r \cdot d_{out}$. Since $r$ is typically much smaller than $d_{in}$ and $d_{out}$ (e.g., $r=8$ vs $d=4096$), the parameter count drops dramatically. The scaling factor $\alpha/r$ is applied during inference or training but does not add trainable parameters.

In Vision-Language Models (VLMs) and Large Language Models (LLMs), this technique is applied to specific linear layers, such as query and value projections in attention mechanisms. By repeating this pattern across multiple layers, the total adapter size remains small (often just a few megabytes) compared to the billions of parameters in the base model. This allows for efficient storage of multiple task-specific adapters on a single GPU without reloading the entire base model weights.

2. Algorithm Approach

The problem requires calculating three values: the total number of trainable parameters in the LoRA adapter, the total number of parameters if full fine-tuning were used, and the percentage of parameters saved.

The approach is purely arithmetic based on the dimensions provided:

1. Iterate through the list of matrix shapes provided for a single layer.
2. Calculate the LoRA parameter count for each matrix using the formula $r(d_{in} + d_{out})$.
3. Calculate the full fine-tuning parameter count for each matrix using the formula $d_{in} \times d_{out}$.
4. Aggregate these counts by multiplying by the number of layers (num_layers).
5. Compute the percentage as $100 \times \frac{\text{LoRA Params}}{\text{Full Params}}$.

This is a straightforward accumulation pattern where you sum up contributions from each component (each matrix in each layer) to get the global totals.

3. Step-by-Step Strategy

1. Initialize Accumulators: Create variables total_lora_params and total_full_params initialized to 0. These will store the cumulative counts.
2. Loop Through Shapes: Iterate over each [d_out, d_in] pair in the layer_shapes list.

• Extract d_out and d_in from the pair.
• Calculate the LoRA parameters for this specific matrix: lora_count = rank * (d_in + d_out).
• Calculate the full parameters for this specific matrix: full_count = d_out * d_in.
• Add lora_count to total_lora_params and full_count to total_full_params.

3. Scale by Layers: Since the layer_shapes list represents the matrices adapted inside one layer, and this pattern repeats for num_layers, multiply both total_lora_params and total_full_params by num_layers.
4. Calculate Percentage: Compute the percentage using the formula:

5. Round the Result: Round the percent value to 4 decimal places as required.
6. Return Results: Return a list containing [total_lora_params, total_full_params, percent]. Ensure the first two are integers and the last is a float.

4. Common Pitfalls

• Confusing Dimensions: Ensure you correctly identify d_out and d_in from the input list [d_out, d_in]. Swapping them in the LoRA formula $r(d_{in} + d_{out})$ yields the same result due to commutativity, but swapping them in the full fine-tuning formula $d_{out} \times d_{in}$ also yields the same result. However, conceptually, it is important to track which is which.
• Forgetting num_layers: The input layer_shapes defines the matrices per layer. A common mistake is calculating the stats for just one layer and forgetting to multiply by num_layers.
• Rounding Errors: Use Python's round() function with ndigits=4 to ensure the percentage is formatted correctly. Floating-point arithmetic can sometimes lead to tiny precision errors, but round() handles the final output requirement.
• Integer vs. Float Division: In Python 3, division / always returns a float. Ensure that the final percentage is a float, but the parameter counts remain integers.
• Alpha Parameter: Do not include the scaling factor $\alpha$ in the parameter count. It is a hyperparameter applied during computation, not a trainable weight.

5. Time & Space Complexity

• Time Complexity: $O(N)$, where $N$ is the number of matrix shapes in layer_shapes. The algorithm iterates through the list once, performing constant-time arithmetic operations for each element. The number of layers num_layers is used in a single multiplication at the end, so it does not affect the iteration count.
• Space Complexity: $O(1)$. The algorithm uses a fixed number of variables to store the accumulators and intermediate results. It does not allocate any additional data structures that grow with the input size.