#lora

22 articles

01 ·Training & Alignment·★ MEMBER·PAPER·7 min read Paper Deep Dive: Transformers Stop Thinking Too Early — How a One-Layer Rank-8 LoRA Stretches 3-Line Reference Following to 50 Pretrained LLMs follow in-context variable references for only two or three lines. Add a rank-8 LoRA (about 65K parameters) at one early layer with every weight frozen, and Qwen3-8B goes from 15.5% to 99% on 24-line chains. We walk through the mid-layer 'relay' behind it and the sharp limit on where the fix can go. 02 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Explained: Post-Training Leaves Behavioral Shadows on Unrelated Decisions — capability transfer through one unrelated word Ask a code-tuned model to pick between 'jacket' and 'tie' a few thousand times, and another model can inherit part of its coding gain. A from-scratch walkthrough of Active Taskless Distillation (ATD): the math, the controls, and the limits. 03 ·★ MEMBER·PAPER·7 min read Paper Explained: SpatialBlock — Teaching Vision-Language Models Spatial Sense with 15,000 Block-Stacking Puzzles A paper that improves spatial reasoning in vision-language models using nothing but 15,000 synthetic block-stacking questions — no real-scene 3D labels. We walk through how the data is built, the two training recipes, what actually helped, and where it falls short. 04 ·Training & Alignment·FREE·PAPER·19 min read QLoRA — Fine-Tuning a 65B Model on One GPU A 65-billion-parameter model in 16-bit weights is 130 GB before you have trained anything. QLoRA gets that onto a single 48 GB card by storing the frozen weights in a 4-bit format shaped like the weights themselves, quantizing the quantization constants, and paging the optimizer to CPU — while gradients still flow through those 4-bit weights into a small 16-bit adapter. 05 ·Training & Alignment·FREE·PAPER·17 min read LoRA from Scratch — Fine-Tuning a Giant by Training Two Thin Matrices Full fine-tuning of a 7B model needs over 100 GB of optimizer state for a job whose actual content is small. LoRA's claim is that the *change* a model needs is low-rank, so it can be stored as a product of two thin matrices, ΔW = BA. This article builds that from the cost problem, takes the equations apart one line at a time, explains why B starts at zero and what α/r is for, counts the parameters for a real model, and ends on the three decisions that actually matter: the rank, the target layers, and what merging does at inference time. 06 ·Audio & Speech·★ MEMBER·PAPER·10 min read Paper Walkthrough: X-AuT — Pull Audio-Encoder Layers Out Two at a Time, Then Backfill from a Bigger Teacher Delete whole layers from a speech LLM's audio encoder and the decoder starts guessing 'end of sentence' too early, dropping characters. X-AuT picks which layers to remove with short trial runs, then backfills in three stages: distillation from a larger teacher, supervision on the student's own decoding context, and LoRA finetuning. Qwen3-ASR-0.6B goes from 18 to 16 layers with macro-average error falling 5.61% → 5.27%, and 14 layers land at 5.75% — single-run numbers, read here through the equations and the recipe. 07 ·★ MEMBER·PAPER·13 min read Paper Explained: Reason Through the Latent! — Visual information being in a latent state doesn't mean the answer uses it Several latent visual reasoners keep their accuracy even when you corrupt the latent state — the answer is not actually flowing through it. CVRR (arXiv:2609.06746) starts recurrence from the question state after the image has been absorbed, then closes every other image-conditioned route before decoding, making latent computation necessary while keeping performance. A walkthrough grounded only in the paper's text, including its causal interventions. 08 ·★ MEMBER·PAPER·17 min read Paper Walkthrough: Miles v0.1 — Anatomy of a Post-Training Stack Built to Survive Production A read-through of Miles, a full-stack system for agentic RL post-training of frontier MoE models: how it hunts down the numerical drift between the rollout engine and the trainer, and what a 744B model actually measured on 64 GPUs. 09 ·★ MEMBER·PAPER·9 min read Paper Walkthrough: Marigold V2 — Turning an Image Editor Into a State-of-the-Art Depth Estimator on One 32GB GPU How Marigold V2 repurposes an image-editing diffusion transformer (Qwen-Image-Edit) into a state-of-the-art monocular depth estimator using QLoRA and two new losses — explained from first principles through normalization, single-step inference, iREPA, and SinkLoss. 10 ·Training & Alignment·FREE·PAPER·10 min read LoRA vs. Full Fine-Tuning — What You Give Up, What You Get LoRA squeezes a model's weight update into two thin matrices, ΔW = BA. This article starts from zero: why that is enough, what you actually trade away against full fine-tuning, how to pick the rank, and where merging quietly breaks. 11 ·RAG & Retrieval·FREE·8 min read RAG vs Fine-Tuning — Which One, and When The two main ways to make an LLM better, compared on four axes: knowledge freshness, cost, hallucination, and data requirements. From metaphor to math to interactive demos to the mistakes that break production systems. 12 ·Inference & Serving·★ MEMBER·PAPER·10 min read Paper Explained: Compile by Training — Turning a Natural-Language Spec into a Function That Runs Locally "Sort my email into urgent and later" — a spec that vague, turned into a small function you can run locally after about a minute of training. A ground-up walkthrough of Compile by Training: analogy, mechanism, equations, measured numbers. 13 ·Training & Alignment·★ MEMBER·PAPER·12 min read Paper Walkthrough: StudentSim — Training a Simulator That Is Actually *That* Student How to build the practice students an AI tutor needs, by training on real learner records. From the two-axis definition of behavioral fidelity and guidance responsiveness, through the pooled-then-specialized pipeline, to using the simulator as a reward model. 14 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Walkthrough: Normalized Low-Rank Adaptation — Why Normalizing LoRA's Entry Matrix Works Rescaling LoRA's down-projection so every column has unit length improves convergence, stability and forgetting resistance at zero extra cost. A ground-up reading of Normalized Low-Rank Adaptation (NoRA) through the lens of a hidden preconditioner. 15 ·Time Series·★ MEMBER·PAPER·12 min read H3-World: Turning Language Understanding into World Control — Paper Explained H3-World turns a 33B video generator into a controllable world model without bolting on an action module: express actions as sentences, bind each sentence to a video-latent interval, and use attention routing to stop the crosstalk. Only 0.199% of parameters are trained. 16 ·Agents·★ MEMBER·PAPER·14 min read Paper Walkthrough: Code World Model — Putting a Coding Agent in Charge of the World A ground-up reading of Code World Model, which hands the *evolution* of a world to executable code and its *appearance* to a video model — including the proxy interface that joins them, and the limits the authors admit to. 17 ·Generative Models·FREE·PAPER·11 min read A Practical Map of Image Generation — SD, ControlNet, and Applying LoRA A single map for anyone about to touch image generation for the first time: the four boxes of latent diffusion, where the prompt actually takes effect, locking down composition with ControlNet, fine-tuning with LoRA, and the licensing layers that quietly cause the most damage. 18 ·Training & Alignment·★ MEMBER·PAPER·9 min read Continual Learning and Catastrophic Forgetting — Why Models Can't Just Keep Learning Why a model that learns something new tends to lose what it already knew, explained from a single fact: weights are a shared resource. Walks through EWC, replay, and swappable LoRA adapters with equations and interactive figures, then explains why production teams still just mix the old data back in and retrain. 19 ·Linear Algebra·FREE·7 min read The Linear Algebra Under LoRA and RAG — Eigenvalues, Low Rank and Vector Search, Hands On A matrix is a deformation of space, an eigenvector is a direction that survives it, SVD generalises the idea, and the dot product is the definition of 'similar'. Four interactive figures and four equations show that LoRA's ΔW=BA and RAG's vector search stand on the same floor. A column meant to be dragged, not just read. 20 ·Linear Algebra·★ MEMBER·PAPER·10 min read Singular Value Decomposition and Low-Rank Approximation — the Math Behind LoRA Starting from the 'rotate, stretch, rotate' picture, this article builds Singular Value Decomposition (SVD) from zero: matrices as stacks of rank-1 layers, why real-world data needs only a few of them, and how that single fact lets LoRA fine-tune a giant model with 0.4% of the parameters. 21 ·Agents·FREE·PAPER·8 min read Paper Walkthrough: Macaron-V1 — A Frozen Base plus a Mixture of LoRAs, Built to Keep Learning After Launch A guided tour of Mind Lab's open agent-model family Macaron-V1, straight from the paper: the Mixture-of-LoRA architecture that stacks four specialist adapters on a frozen 744B base and picks one per turn, and the recursive self-improvement loop that trains the model and its harness as a versioned pair. 22 ·Paper Deep-Dives·FREE·PAPER·12 min read LoRA Paper Explained: Low-Rank Adaptation of Large Language Models (Hu et al., 2021) — Why Low Rank Is Enough The LoRA paper (Hu et al., 2021) explained, grounded strictly in its own text: what BA actually means, the 21x amplification factor the authors measured, why r=1 sufficed, and the questions they left open.