#adapters

1 articles

01 ·Training & Alignment·FREE·PAPER·17 min read LoRA from Scratch — Fine-Tuning a Giant by Training Two Thin Matrices Full fine-tuning of a 7B model needs over 100 GB of optimizer state for a job whose actual content is small. LoRA's claim is that the *change* a model needs is low-rank, so it can be stored as a product of two thin matrices, ΔW = BA. This article builds that from the cost problem, takes the equations apart one line at a time, explains why B starts at zero and what α/r is for, counts the parameters for a real model, and ends on the three decisions that actually matter: the rank, the target layers, and what merging does at inference time.