low-rank

Definition

The property that a large matrix can be approximated by the product of two much smaller ones, so its effective degrees of freedom are far fewer than its shape suggests. This is the premise behind LoRA: weight updates during fine-tuning appear to lie in a low-dimensional subspace, so only that subspace needs to be learned and stored. It is a lossy approximation, and the chosen rank is the dial that trades capacity against cost.

Explainers using this term

• Paper Walkthrough: Normalized Low-Rank Adaptation — Why Normalizing LoRA's Entry Matrix WorksNormalized Low-Rank Adaptation
• A Practical Map of Image Generation — SD, ControlNet, and Applying LoRALoRA: Low-Rank Adaptation of Large Language Models
• Singular Value Decomposition and Low-Rank Approximation — the Math Behind LoRALoRA: Low-Rank Adaptation of Large Language Models