#finetune

2 articles

01 ·Training & Alignment·FREE·PAPER·17 min read LoRA from Scratch — Fine-Tuning a Giant by Training Two Thin Matrices Full fine-tuning of a 7B model needs over 100 GB of optimizer state for a job whose actual content is small. LoRA's claim is that the *change* a model needs is low-rank, so it can be stored as a product of two thin matrices, ΔW = BA. This article builds that from the cost problem, takes the equations apart one line at a time, explains why B starts at zero and what α/r is for, counts the parameters for a real model, and ends on the three decisions that actually matter: the rank, the target layers, and what merging does at inference time. 02 ·Training & Alignment·FREE·PAPER·10 min read LoRA vs. Full Fine-Tuning — What You Give Up, What You Get LoRA squeezes a model's weight update into two thin matrices, ΔW = BA. This article starts from zero: why that is enough, what you actually trade away against full fine-tuning, how to pick the rank, and where merging quietly breaks.