#low-rank

3 articles

01 ·Training & Alignment·FREE·PAPER·17 min read LoRA from Scratch — Fine-Tuning a Giant by Training Two Thin Matrices Full fine-tuning of a 7B model needs over 100 GB of optimizer state for a job whose actual content is small. LoRA's claim is that the *change* a model needs is low-rank, so it can be stored as a product of two thin matrices, ΔW = BA. This article builds that from the cost problem, takes the equations apart one line at a time, explains why B starts at zero and what α/r is for, counts the parameters for a real model, and ends on the three decisions that actually matter: the rank, the target layers, and what merging does at inference time. 02 ·Training & Alignment·FREE·PAPER·10 min read LoRA vs. Full Fine-Tuning — What You Give Up, What You Get LoRA squeezes a model's weight update into two thin matrices, ΔW = BA. This article starts from zero: why that is enough, what you actually trade away against full fine-tuning, how to pick the rank, and where merging quietly breaks. 03 ·Linear Algebra·★ MEMBER·PAPER·10 min read Singular Value Decomposition and Low-Rank Approximation — the Math Behind LoRA Starting from the 'rotate, stretch, rotate' picture, this article builds Singular Value Decomposition (SVD) from zero: matrices as stacks of rank-1 layers, why real-world data needs only a few of them, and how that single fact lets LoRA fine-tune a giant model with 0.4% of the parameters.