#softmax
5 articles
01
·Distillation & Compression·FREE·PAPER·8 min read
The Math of Distillation — Why Soft Answers Teach More
Why is the distillation loss KL(teacher||student), what is the temperature T actually doing, and where does that mysterious T² factor in every implementation come from? A ground-up derivation of why a soft distribution carries more teaching signal than a correct answer.
02
·Calculus & Optimization·★ MEMBER·11 min read
Matrix Calculus from Scratch — Derive the Backward Pass Yourself
Where does the transpose in ∂L/∂W = XᵀG actually come from? Matrix calculus is not a formula sheet to memorize — it is one move: rotate dX to the right inside a trace. From denominator layout and shape-checking to the gradients of a linear layer and softmax + cross-entropy, ending with a double-precision gradient check.
03
·Numerical Computing·★ MEMBER·10 min read
Numerical Pitfalls — Cancellation, Rounding, and logsumexp
Where "the loss went nan three hours into the run" actually comes from, built up from nothing: how rounding enters, how the condition number amplifies it, and why subtracting two close numbers is so destructive. It all converges on logsumexp — the one trick sitting inside every softmax and cross-entropy implementation.
04
·Information Theory·FREE·9 min read
Entropy and Cross-Entropy — Where the Loss Function Comes From
The cross-entropy loss you write in every classifier wasn't handed down by decree. Starting from the idea of surprise, we walk a single path through entropy and cross-entropy to that familiar one-liner — and finish with why squared error is the wrong tool for the job.
05
·CNNs & Image Recognition·FREE·8 min read
Image Classification from Scratch — The Invention of the Convolution
Feed a photograph straight into a fully connected layer and you get over a hundred million parameters — and a model for which the same cat, moved a few pixels, is an entirely different input. The convolution solves both with one idea: slide a small window. Kernels, stride, padding and pooling, ending with softmax turning scores into probabilities.