Activation Functions from Scratch — Why Nonlinearity Is Non-Negotiable

Without an activation function, a hundred stacked layers can do exactly what one layer does. Starting from that one-line proof, this article traces why sigmoid was abandoned, why ReLU won, and why today's LLMs settled on SiLU and SwiGLU — with an interactive plot where you can drag the input and watch the slope.

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Change money three times, and it's still one multiplication

Convert yen to dollars, dollars to euros, euros to pounds. Ignoring fees, the amount you end up with can be predicted by a single number: your starting yen times one fixed rate. Three conversions collapse into one. Adding more currency windows buys you nothing.

A neural network layer, left to itself, suffers the same fate. What a layer does is multiply its input by weights, add them up, and add an offset — a scaling and a shift, nothing more. Stack a hundred of those and mathematically you still have one. The only boundary such a network can draw is a straight one.

So we wedge one not-straight step between the conversions. That is an activation function. The moment it goes in, stacking layers starts to mean something, and the network can bend its decision boundaries. Deep learning is deep in any useful sense only because this small function sits between the layers.

This article starts with the one-line proof of why nonlinearity is required, then follows the story forward: sigmoid falling out of favour, ReLU taking over, and today's large language models converging on GELU, SiLU and SwiGLU. You will get to drag the input around and feel the slope for yourself. Almost no prerequisites needed.

Where the activation function actually sits

First, the location. A single layer looks like this:

$$z = Wx + b, \qquad a = \phi(z)$$

One symbol at a time. $x$ is the input vector (a list of numbers), $W$ is the weight matrix, $b$ is the bias vector (the offset), and $z$ is the value before the activation is applied — practitioners call it the pre-activation, or just "the logits". Then $\phi$ is the activation function, and $a$ is what gets handed to the next layer.

Written out in words: multiply each input number by its weight, add the results up, add an offset on top to get $z$, push $z$ through the function $\phi$, and hand the result to the next layer. Nothing in that sentence is harder than a weighted average with a fixed number added at the end.

The easy thing to miss is that $\phi$ acts element by element. If $z$ is a list of a thousand numbers, the same function is applied independently to each of the thousand. It never looks at its neighbours. That makes activation functions almost free compared to the matrix multiply next door — and yet they largely determine the character of the network. Cheap and decisive is an unusual combination, and it explains why so much attention has been paid to such a small piece.

What happens without nonlinearity (the one-line proof)

Remove the activation and stack two layers, feeding the first output straight into the second:

Put in words: equation (1) says that two layers' worth of computation can be replaced exactly by one pre-multiplied weight matrix $W' = W_2W_1$ and one bias $b' = W_2b_1 + b_2$. All a layer knows how to do is scale and shift, and a scale of a scale is still just a scale — so the second layer's weights can be folded into the first layer's before the input ever arrives. Just like two currency conversions folding into a single rate, the extra layer has bought nothing. Ten layers, a hundred layers — the conclusion is unchanged.

Turned around: the only thing we ask of an activation function is that it breaks this collapse. As long as the layers can no longer be folded together, expressive power accumulates with depth. Theory backs this up — provided the activation is not a polynomial, a single hidden layer of sufficient width can approximate any continuous function arbitrarily well (the universal approximation theorem). But that theorem only promises that a good set of weights exists; it promises nothing about training ever finding it. The reason practitioners argue about activation functions is not expressive power. It is how well gradients flow.

The intuition: enough straight segments will draw anything

"Nonlinear" evokes smooth curves, but the most-used activation in history, ReLU, is a bent straight line that simply zeroes out negatives. Is that enough? Add up enough bent lines and you can get arbitrarily close to a circle, a wave, anything — the same reason a polygon starts to look like a circle once you add enough vertices.

A ReLU network in fact represents a piecewise linear function: the input space is chopped into many regions, and within each region the output is a flat plane. Adding layers multiplies the number of creases, which is what buys finer shapes. It looks curved because the creases are dense, not because any individual piece bends.

The first generation: sigmoid, tanh, and saturation

Early neural networks used the sigmoid:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Spelled out in words: whatever real number goes in, what comes out is squeezed into the gap between 0 and 1. The $e^{-z}$ term is the squeezer — for large positive $z$ it shrinks towards nothing, so the fraction climbs towards 1; for large negative $z$ it blows up, so the fraction collapses towards 0. Close to 1 for large positive inputs, close to 0 for large negative ones, rising gently through the middle. It felt intuitive, like a biological neuron expressing a probability of firing.

tanh is the same shape stretched to span $-1$ to $1$ (they are literally related by $\tanh(z) = 2\sigma(2z) - 1$). Because its output is centred on zero, it tends to be better behaved than sigmoid.

The problem is that both go flat at the ends. Training pushes error backwards through the layers, and how much survives the trip is the product of the slopes it passes through. The sigmoid's slope never exceeds 0.25. Across ten layers that is $0.25^{10}$ — about one in a million. Almost nothing reaches the early layers. This is vanishing gradient, and it is one of the main reasons deep networks refused to train until around 2010. The mechanism by which error travels backwards is covered in Backpropagation from Scratch.

In the figure below, watch the function curve (the bold line) and its slope (the faint gold line) at the same time. Drag horizontally across the plot to move the input, and the output and slope at that point are printed as numbers. Leave it on sigmoid, pull the input out to about $\pm4$, and you can watch the slope collapse to nearly zero.

ReLU: the winner that just throws away negatives

What took over in the 2010s was ReLU (Rectified Linear Unit):

$$\mathrm{ReLU}(z) = \max(0, z)$$

That is the entire function in words: if the number is negative, write down zero; if it is positive, leave it exactly as it was. The $\max$ is doing nothing more exotic than picking whichever of the two things in the brackets is bigger.

Three things made it work. First, the slope on the positive side is exactly 1, so no matter how many layers the gradient passes through, it is multiplied by one each time and does not shrink geometrically. Second, there is no exponential to evaluate, so it is fast. Third, zeroing negatives makes the output sparse (many elements exactly zero), which encourages a division of labour where each unit responds to a specific kind of input.

There is a price. The slope on the negative side is zero, so if some unit's $z$ ends up negative for every input it will ever see, its gradient is zero forever and it never recovers. This is the dying ReLU problem. It tends to strike right after the learning rate is raised too far, and it shows up in the most annoying way possible: the loss stops improving without ever diverging.

A family of patches grew up around this hole.

In practice, though, none of these displaced ReLU. Plain ReLU is still widely used in convolutional vision models. The trap is usually not ReLU itself but forgetting to match the initialisation. ReLU discards half its inputs, which shrinks the variance of what comes out. He initialisation (Kaiming initialisation) accounts for that by starting the weights larger; leave a tanh-era Xavier initialisation in place and the signal thins out in the deeper layers.

GELU and SiLU: the activation as a gate

Since the Transformer, smoothed-out versions of ReLU's kink have become the norm. The two you will meet are GELU and SiLU (also known as Swish):

Read equation (2) as a sentence which says: take the input, work out a second number saying how much of it deserves to pass, and multiply the two together. Both functions decide what fraction of the input to let through, as a continuous number between 0 and 1. $\Phi$ is the cumulative distribution function of the standard normal distribution (the probability that a standard normal draw is at most $z$), and $\sigma$ is the sigmoid from earlier; both approach 1 for large $z$ and 0 for very negative $z$. Where ReLU makes a hard all-or-nothing cut, these replace it with a gatekeeper who smoothly varies how much gets through.

Why does smoothing help? For one thing, the slope no longer jumps discontinuously at $z=0$, which gives the optimiser a friendlier surface. For another, a small output survives on the negative side. Both GELU and SiLU dip slightly negative just below zero, and the slope there is not zero — so a unit can climb back out, and dying-ReLU-style permanent death becomes structurally unlikely.

One implementation note. GELU's $\Phi$ involves the error function, so a fast approximation is common:

$$\mathrm{GELU}(z) \approx 0.5\,z\left(1 + \tanh\!\left(\sqrt{2/\pi}\,(z + 0.044715\,z^3)\right)\right)$$

Intimidating as it looks, it is a line which says "replace the exact $\Phi$ with a similarly shaped curve built out of tanh". None of the constants carry a meaning of their own: $\sqrt{2/\pi}$ and $0.044715$ are just the numbers that make the tanh curve sit almost exactly on top of $\Phi$. The values are very close but not identical, which means mixing the exact and approximate versions makes the same weights produce slightly different outputs. As we will see below, this is a classic porting bug.

In the figure below, switch the tab to GELU and drag the input into the negative region. Where ReLU would be perfectly flat, GELU's slope line is still lifted off zero.

SwiGLU: the shape today's LLMs actually use

Most current large language models do not use an activation function on its own. They rebuild the feed-forward layer inside the Transformer block into a gated form:

$$\mathrm{SwiGLU}(x) = \big(\mathrm{SiLU}(xW) \odot xV\big)W_2$$

The symbols: $W$, $V$ and $W_2$ are three separate weight matrices, and $\odot$ is elementwise multiplication (multiply matching positions). Read it in words: make two copies of the input with two different weight matrices, turn one copy into a row of dials set somewhere between 0 and 1, multiply the other copy by those dials position by position, and send what survives through a final matrix. Two branches are built from the same input $x$; only one of them is pushed through SiLU to become an "openness" value near 0 to 1, and that openness is multiplied into the raw values of the other branch. In other words, the branch that carries the value and the branch that decides how much of it passes are split apart.

The extra branch means a naive implementation has 1.5× the parameters, so implementations typically shrink the hidden width to about $\tfrac{2}{3}$ so that comparisons are made at matched parameter counts. The design spread after Shazeer (2020) benchmarked gated variants for the Transformer, and many major open models have used it since.

Seeing it in code

Every one of these is a few lines:

import numpy as np

def relu(z):  return np.maximum(0.0, z)
def leaky(z, a=0.01):  return np.where(z > 0, z, a * z)
def sigmoid(z):  return 1.0 / (1.0 + np.exp(-np.clip(z, -60, 60)))
def silu(z):  return z * sigmoid(z)
def gelu(z):  return 0.5 * z * (1 + np.tanh(np.sqrt(2/np.pi) * (z + 0.044715 * z**3)))

The np.clip is the practical trick. If $z$ reaches something like $-800$, np.exp overflows and you get warnings or inf. Clipping at $\pm60$ does not change the sigmoid's value at all — it has already pinned to 0 or 1 — so the clip is free. This species of stabilisation — absent from the formula, mandatory in the code — turns up all over deep learning implementations.

A quick guide to choosing

• Convolutional vision models: start with ReLU. Assume He initialisation and a normalisation layer alongside it.
• Transformers: GELU or SiLU. If you are reproducing an existing model, match whichever that model uses — exactly.
• Regression output layer: nothing at all (identity). Put a ReLU here and the model can never output a negative number.
• Binary classification output: conceptually a sigmoid, but in code you should pass raw logits to the loss instead (see below).
• Multi-class output: softmax — and again, let the loss function handle it.

How this shows up on the job

You rarely get to choose an activation function. Outside of designing a model from scratch, there is almost never a reason to touch the default. The reason you still need to understand them is that when something goes wrong, this is often where the fault lives.

Who touches this, and when. The ML engineer running training, when isolating why the loss stopped falling. The engineer porting a researcher's implementation into production code, when the outputs refuse to match the paper. The inference engineer, when deciding what to quantise or fuse into a kernel.

Parameter names you will actually type. In PyTorch: nn.ReLU, nn.LeakyReLU(negative_slope=0.01), nn.SiLU, nn.GELU(approximate='none' | 'tanh'). For Hugging Face models the real switch is hidden_act in config.json, holding a string like "gelu" or "silu". Initialisation helpers usually take the activation as an argument — torch.nn.init.kaiming_normal_(w, nonlinearity='relu') — and leaving that at its default is exactly how initialisation and activation end up mismatched.

Four ways this bites people.

First, double sigmoid. Applying torch.sigmoid at the end of the model and then handing the result to BCEWithLogitsLoss (which applies a sigmoid internally). Nothing errors, training even sort of progresses, and accuracy simply plateaus below where it should. The rule: if the loss has WithLogits in its name, give it raw, un-activated values.

Second, the GELU version mismatch. Mix the exact and tanh-approximate forms and identical weights produce slightly different outputs. Negligible in isolation, but the discrepancy compounds through a deep stack, and it is a textbook cause of "we ported it and lost a few points of accuracy". This is why Hugging Face keeps "gelu" and "gelu_new" as distinct entries — when moving a checkpoint, match the activation down to the variant.

Third, missing a dying ReLU. If the loss stalls right after you raised the learning rate, measure the fraction of zeros in each layer's output over one batch. If a layer is over 90% zeros all the time, it is largely dead. The fixes are to lower the learning rate, switch to Leaky ReLU or SiLU, or revisit the initialisation.

Fourth, ordering. The standard convolutional block is Conv → normalisation → activation. Swap the last two and the activation either undoes the scale the normalisation just established or vice versa, and convergence behaviour changes. The normalisation side of that story is in A History of Normalization Layers.

If it comes up in an interview. "Why does ReLU mitigate vanishing gradients?" is a staple. The core of the answer is one sentence: the slope on the positive side is exactly 1, so the per-layer multiplications do not shrink the gradient. Following it with "though ReLU alone was not enough — it took residual connections and normalisation before networks of dozens of layers actually trained" shows you know the order in which the history happened.

Summary

• Without an activation function, any number of stacked layers equals one layer — this is the part that makes depth mean something
• sigmoid and tanh go flat at the tails, so gradients thin out layer by layer (vanishing gradient)
• ReLU has slope exactly 1 on the positive side and is fast, at the cost of units that can die on the negative side
• GELU and SiLU smooth ReLU's kink and act as gatekeepers that continuously decide how much passes
• Current LLMs mostly use SwiGLU-style layers, which split the value branch from the gate branch

If you want the next step, Neural Networks from Scratch picks up directly from here and shows how layers with these activations stack into a working network.