#optimization
15 articles
01
·Inference & Serving·★ MEMBER·PAPER·8 min read
Paper Walkthrough: Normalized Low-Rank Adaptation — Why Normalizing LoRA's Entry Matrix Works
Rescaling LoRA's down-projection so every column has unit length improves convergence, stability and forgetting resistance at zero extra cost. A ground-up reading of Normalized Low-Rank Adaptation (NoRA) through the lens of a hidden preconditioner.
02
·Agents·★ MEMBER·PAPER·11 min read
Paper Walkthrough: AutoSaddler — Growing a Harness That Doesn't Break, from Agent Failure Logs
A ground-up walkthrough of AutoSaddler, which automatically optimizes the harness around an LLM agent — prompts, tools, and middleware — by repeatedly diagnosing failure traces and generating structured patches. It beat the base harnesses on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 by 9.0, 9.6, and 10.0 points.
03
·Training & Alignment·FREE·PAPER·11 min read
Diagnosing Broken Training — Telling Divergence, NaN, and Plateaus Apart
Training breaks in exactly three ways: it diverges, it goes NaN, or it stalls. Built around a symptom-to-cause table, this article shows why divergence is a threshold effect (with the math and an interactive figure), how to pin down where a NaN was born, and how to isolate the cause of a plateau — assuming no prior knowledge.
04
·Calculus & Optimization·★ MEMBER·11 min read
Jacobians and Hessians — Multivariable Calculus, Drawn
The Jacobian is a magnifying glass at a point; the Hessian is how sharply the ground curves there. From local linearization to what eigenvalues say about the terrain, why Newton's method is fast on paper but absent in practice, and how to get Hessian information in a billion dimensions without ever building the matrix.
05
·Compilers & Runtimes·FREE·11 min read
Compilers From Scratch — How Source Becomes Machine Code
One line of source code, followed all the way down through lexing, parsing, semantic analysis, intermediate representation, optimization, and code generation, until it comes out as a single machine instruction. Along the way: why bugs surface at -O2, and why your benchmark loop disappears.
06
·Complexity·★ MEMBER·12 min read
Approximation Algorithms — Trading Exactness for a Guarantee
The craft of giving up on the optimal answer while attaching a price tag that reads "never worse than X times optimal". We build up the approximation ratio, carry a greedy proof all the way to the end, and see why the triangle inequality flips the entire conclusion for the traveling salesman problem.
07
·Search & Optimization·★ MEMBER·11 min read
Simulated Annealing and Genetic Algorithms — What to Do When Exact Solving Breaks Down
Why search methods that guarantee nothing end up running real delivery routes and factory schedules. From the three ingredients of local search, through temperature in annealing and populations in genetic algorithms, to the harder question of when you should not reach for them at all.
08
·Search & Optimization·★ MEMBER·11 min read
Simulated Annealing and Genetic Algorithms — What to Do When Exact Solving Breaks Down
Why search methods that guarantee nothing end up running real delivery routes and factory schedules. From the three ingredients of local search, through temperature in annealing and populations in genetic algorithms, to the harder question of when you should not reach for them at all.
09
·Search & Optimization·★ MEMBER·11 min read
Linear Programming from Scratch — The Workhorse of Optimization
The oldest and most widely deployed tool for choosing the best option under limited stock, budget and time. From the three-part recipe for writing a model down, to why the answer always sits at a corner, to what duality tells you a kilo of flour is really worth — built up from zero.
10
·Search & Optimization·★ MEMBER·11 min read
Linear Programming from Scratch — The Workhorse of Optimization
The oldest and most widely deployed tool for choosing the best option under limited stock, budget and time. From the three-part recipe for writing a model down, to why the answer always sits at a corner, to what duality tells you a kilo of flour is really worth — built up from zero.
11
·Calculus & Optimization·★ MEMBER·PAPER·12 min read
Beyond SGD — Adam, Second-Order Methods, and Constrained Optimization
What exactly is momentum accumulating? What does each of Adam's four lines do? What did AdamW fix? And why does nobody train an LLM with second-order methods that are supposedly faster? Metaphor, equations, live figures, code, and production practice — no prerequisites assumed.
12
·Training & Alignment·★ MEMBER·PAPER·10 min read
Learning Rate Schedules — Why Warmup and Why Cosine
The learning rate is not a fixed number — it is a curve you design across the whole run. Why we deliberately start slow (warmup), why we come down along a cosine, and what else has to move when batch size changes. Equations, live figures, PyTorch code, and the mistakes that actually break runs.
13
·Calculus & Optimization·★ MEMBER·PAPER·9 min read
Convexity and Optimization — Why Deep Learning Works Even Though It Isn't Convex
Optimization textbooks teach a stark divide: convex problems are solvable, non-convex ones come with no guarantees. So why does deep learning — whose loss surface is provably non-convex — work at all? From convex sets and functions to saddle points and flat minima, this article connects the whole story in the language of landscapes.
14
·Search & Optimization·★ MEMBER·8 min read
Dynamic Programming From Scratch — On Remembering Subproblems
Why naive recursion explodes exponentially, what memoization and table-filling actually change, and Fibonacci, knapsack and edit distance taken apart in order — ending at the places edit distance shows up in real AI systems, from ASR word error rate to diffusion step schedules.
15
·Machine Learning Basics·FREE·8 min read
Loss Functions and Optimization — How a Model Learns From Being Wrong
Why MSE and cross-entropy have the shapes they do, what the gradient actually points at, and one step of gradient descent taken apart with equations, a draggable figure, and ten lines of numpy — divergence included.