low-rank

Definition

The property that a large matrix can be approximated by the product of two much smaller ones, so its effective degrees of freedom are far fewer than its shape suggests. This is the premise behind LoRA: weight updates during fine-tuning appear to lie in a low-dimensional subspace, so only that subspace needs to be learned and stored. It is a lossy approximation, and the chosen rank is the dial that trades capacity against cost.

Explainers using this term

• QLoRA — Fine-Tuning a 65B Model on One GPULoRA: Low-Rank Adaptation of Large Language Models
• LoRA from Scratch — Fine-Tuning a Giant by Training Two Thin MatricesLoRA: Low-Rank Adaptation of Large Language Models
• LoRA vs. Full Fine-Tuning — What You Give Up, What You GetLoRA: Low-Rank Adaptation of Large Language Models