Neural Networks from Scratch — From One Neuron to Many Layers

What a single neuron actually computes, and why stacking layers is pointless without an activation function — shown with a one-line proof that composing linear maps just gives you another linear map. Metaphor, math, an interactive figure, then fifteen lines of numpy.

One credit officer in a room

Imagine a single officer deciding whether to sign a new supplier. She looks at three numbers and nothing else: annual volume, years of relationship, and count of late payments. In her head each carries a weight — "volume is worth 3 points, years are worth 2, every late payment costs 5" — and she adds them up. If the total clears her bar, she approves.

That is a neuron. A neural network is that officer duplicated a few hundred times across a row, and then stacked a few rows deep. The hard part is deciding the weights; the officer herself is almost embarrassingly simple.

This article assumes you already know what a loss function and gradient descent are — turning "how wrong the model is" into a single number, then nudging parameters in the direction that shrinks it. If that is hazy, read loss functions and optimization first and the second half here will land better.

What a single neuron computes

The officer's job, written as math, is this and nothing more.

$$y = \sigma(w^\top x + b)$$

One symbol at a time:

• $x$ is the input vector — in our example, $x = (\text{volume}, \text{years}, \text{late payments})$.
• $w$ is the weight vector, the same length as $x$, holding how much each input matters.
• $w^\top x$ is the dot product: $w_1 x_1 + w_2 x_2 + \dots + w_d x_d$, multiply-and-add, nothing more ($d$ is the input dimension).
• $b$ is the bias, a constant added to everything, which sets how strict the bar is.
• $\sigma$ is the activation function, a non-linear function applied to the total.
• $y$ is the output — this neuron's verdict.

What the formula says, in plain words: add the inputs up with weights, shift the total, then push it through one function. The $w^\top x + b$ part is just $ax + b$ with more variables.

Why you need an activation function at all

Is $\sigma$ really necessary? Emitting the raw total feels like it ought to be enough. Let us check. Suppose $\sigma$ does nothing at all (the identity function), and write out a two-layer network — layer one's output fed straight into layer two.

$$y = W_2 (W_1 x + b_1) + b_2 = (W_2 W_1) x + (W_2 b_1 + b_2)$$

What this says, in plain words: expand the brackets and the two layers collapse into one. $W_2 W_1$ is just a matrix product, so call it $W'$; call $W_2 b_1 + b_2$ simply $b'$. The right-hand side is $W' x + b'$ — precisely the shape of a single layer.

Stack a hundred layers and you get the same result: one matrix that happens to be a hundred matrices multiplied together. Without an activation function, depth buys you exactly nothing. That is what "composing linear maps only ever gives you a linear map" means.

A concrete case makes it vivid. Take XOR: input $(0,0)$ gives 0, $(0,1)$ gives 1, $(1,0)$ gives 1, $(1,1)$ gives 0. Plot those four points and try to separate the zeros from the ones with a single straight line. You cannot — the two zeros sit on one diagonal and the two ones sit on the other. Slip a non-linear function in between, though, and it becomes solvable. Non-linearity is what gives a model the freedom to fold space.

What extra layers buy you

In theory, a single hidden layer with enough units can approximate any continuous function to arbitrary precision — the universal approximation theorem, proved in the late 1980s and early 1990s. What the theorem pointedly does not promise is how many units you would need, or whether training could ever find them.

We go deep anyway because, empirically, the same function is usually cheaper to build tall and thin than short and wide. Deeper layers can reuse the outputs of earlier ones as parts. In vision networks, early layers respond to simple things like intensity edges while later layers respond to textures and object parts assembled from them. Layer one builds the parts, layer two combines them, layer three combines those — and that is where depth earns its keep.

Forward propagation, all at once, in matrices

A layer is a bundle of $m$ neurons looking at the same input. Stack each neuron's weight vector $w$ as a row and you get a weight matrix $W$, which turns a whole layer into one matrix-vector product. For three layers:

$$h^{(1)} = \sigma(W^{(1)} x + b^{(1)})$$

$$h^{(2)} = \sigma(W^{(2)} h^{(1)} + b^{(2)})$$

$$\hat{y} = W^{(3)} h^{(2)} + b^{(3)}$$

The symbols: $h^{(1)}$ is the first layer's output — a hidden layer value, "hidden" because it is neither the input nor the final answer. $W^{(l)}$ and $b^{(l)}$ are the weight matrix and bias vector of layer $l$. $\hat{y}$ ("y-hat") is the model's prediction, hatted to distinguish it from the true label $y$. And $\sigma$ is applied element-wise — the same function on each component separately, never mixing them.

What those three lines say, in plain words: multiply, shift, bend — three times. Because we compute them in order from input to output, this is called the forward pass.

Tracking dimensions makes it concrete. With a 784-dimensional input (a 28×28 grayscale image flattened into a row) and 128 units in the first layer, $W^{(1)}$ is a 128×784 matrix. In practice you push $N$ examples at once as a matrix $X$, so one matrix product handles the whole batch.

From sigmoid to ReLU

The activation function of choice for a long time was the sigmoid.

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Here $z$ is the input (the value of $w^\top x + b$) and $e$ is Euler's number, about 2.718. What it says, in plain words: squash any real number into the range 0 to 1. Large positives go to nearly 1, large negatives to nearly 0, with a smooth ramp in between.

In deep networks it has a fatal flaw. Its derivative is

$$\sigma'(z) = \sigma(z)\,(1 - \sigma(z))$$

and since $\sigma(z)$ lives between 0 and 1, that product peaks at 0.25, at $z = 0$. Push $z$ far in either direction and it collapses toward zero.

Why does that matter? Training propagates the error from the output back toward the input to compute each layer's gradient — the instruction telling every parameter which way and how far to move. Each layer you travel back multiplies in another activation derivative. Ten layers means ten factors of at most 0.25, and $0.25^{10}$ is roughly one in a million. The instruction reaching the earliest layers is effectively zero. This is vanishing gradients; the backpropagation article works through the mechanics.

The replacement that took over is ReLU (Rectified Linear Unit).

$$\mathrm{ReLU}(z) = \max(0, z)$$

Zero if negative, unchanged if positive — a bent straight line, and that is the whole function. But its derivative in the positive region is exactly 1, so gradients do not shrink as they travel back. It also costs one comparison rather than an exponential. That swap is one reason deep networks became trainable in practice.

It has its own failure mode. On the negative side both the output and the derivative are zero, so a neuron whose input stays negative receives zero gradient and never updates again — dying ReLU. Leaky ReLU keeps a small slope on the negative side, and GELU smooths the corner, to soften that.

The forward pass in numpy

Everything above amounts to this much code.

import numpy as np

def relu(z):
    return np.maximum(0, z)

def forward(X, params):
    h = X                          # (N, 784)  a whole batch at once
    for W, b in params[:-1]:       # hidden layers: linear, then non-linear
        h = relu(h @ W + b)
    W, b = params[-1]              # no activation on the output layer
    return h @ W + b               # (N, 10)  one score per class

h @ W + b is the matrix product; relu is the non-linearity. Those two lines, repeated, are what a neural network is. The output layer skips its activation because folding softmax into the loss function is numerically more stable — a standard implementation trick, not a modelling choice.

Three things that matter in practice

1. Never initialise weights to zero. Start everything at zero and every neuron in a layer emits the same value and receives the same gradient. The symmetry never breaks, so a thousand units do the work of one. Random initialisation is mandatory; He initialisation is the default for ReLU-family activations, Xavier (Glorot) for sigmoid and tanh.

2. Hidden layers and the output layer play different roles. For hidden layers, a ReLU-family activation is a fine default. The output layer is dictated by the task: no activation for regression, sigmoid for binary classification, softmax for multi-class. Get this wrong and a structurally correct model still will not learn.

3. Before adding layers, suspect your input scaling and learning rate. When training stalls, the instinct is to go deeper. The actual cause is far more often unscaled inputs or a learning rate that is too large or too small. Depth is the last resort, not the first.

Summary

• A neuron is a weighted sum plus a bias plus an activation: $y = \sigma(w^\top x + b)$ is the whole story
• Without an activation, any stack of layers collapses into a single $W'x + b'$
• The forward pass is "multiply, shift, bend," repeated
• Sigmoid's derivative peaks at 0.25 and gets multiplied at every layer, so gradients vanish; ReLU's positive-side derivative of 1 avoids that

Next up is the machinery that sends those gradients back through the layers: backpropagation, which the chain rule alone fully explains. Even modern components like attention decompose into the same linear-transform-plus-non-linearity pattern you just saw. The foundation does not change.