#swiglu

1 articles

01 ·Deep Learning Basics·FREE·PAPER·13 min read Activation Functions from Scratch — Why Nonlinearity Is Non-Negotiable Without an activation function, a hundred stacked layers can do exactly what one layer does. Starting from that one-line proof, this article traces why sigmoid was abandoned, why ReLU won, and why today's LLMs settled on SiLU and SwiGLU — with an interactive plot where you can drag the input and watch the slope.