#peft

6 articles

01 ·Training & Alignment·★ MEMBER·PAPER·7 min read Paper Deep Dive: Transformers Stop Thinking Too Early — How a One-Layer Rank-8 LoRA Stretches 3-Line Reference Following to 50 Pretrained LLMs follow in-context variable references for only two or three lines. Add a rank-8 LoRA (about 65K parameters) at one early layer with every weight frozen, and Qwen3-8B goes from 15.5% to 99% on 24-line chains. We walk through the mid-layer 'relay' behind it and the sharp limit on where the fix can go. 02 ·Training & Alignment·FREE·PAPER·19 min read QLoRA — Fine-Tuning a 65B Model on One GPU A 65-billion-parameter model in 16-bit weights is 130 GB before you have trained anything. QLoRA gets that onto a single 48 GB card by storing the frozen weights in a 4-bit format shaped like the weights themselves, quantizing the quantization constants, and paging the optimizer to CPU — while gradients still flow through those 4-bit weights into a small 16-bit adapter. 03 ·Training & Alignment·FREE·PAPER·17 min read LoRA from Scratch — Fine-Tuning a Giant by Training Two Thin Matrices Full fine-tuning of a 7B model needs over 100 GB of optimizer state for a job whose actual content is small. LoRA's claim is that the *change* a model needs is low-rank, so it can be stored as a product of two thin matrices, ΔW = BA. This article builds that from the cost problem, takes the equations apart one line at a time, explains why B starts at zero and what α/r is for, counts the parameters for a real model, and ends on the three decisions that actually matter: the rank, the target layers, and what merging does at inference time. 04 ·Time Series·★ MEMBER·PAPER·13 min read Paper Review: MetroLLM-Bench — What Happens When an LLM Runs the Ticket Machine. Why a 2.6 GB 4B Model Tied GPT-5.6, and Why Fine-Tuning Hurt at 27B A 955-case benchmark that asks how well a language model can serve as the decision layer of a transit ticket kiosk. We read, from the paper alone, how a 4B QLoRA student beat GPT-5.6 on the deterministic tier and why PEFT gains shrink with model size until they turn negative at 27B. 05 ·Training & Alignment·FREE·PAPER·10 min read LoRA vs. Full Fine-Tuning — What You Give Up, What You Get LoRA squeezes a model's weight update into two thin matrices, ΔW = BA. This article starts from zero: why that is enough, what you actually trade away against full fine-tuning, how to pick the rank, and where merging quietly breaks. 06 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Walkthrough: Normalized Low-Rank Adaptation — Why Normalizing LoRA's Entry Matrix Works Rescaling LoRA's down-projection so every column has unit length improves convergence, stability and forgetting resistance at zero extra cost. A ground-up reading of Normalized Low-Rank Adaptation (NoRA) through the lens of a hidden preconditioner.