RNNs and LSTMs from Scratch — Why Learn Them in the Transformer Era

Start from one idea — read a sequence one step at a time while carrying a state — then work out why multiplying the same matrix over and over kills the gradient, and what the three LSTM gates actually fixed. Ends with why Transformers took over, and where this recurrent idea still wins.

The metaphor: reading a novel with a single notepad

You read a long novel from page one. All you carry is the text in front of you and one sheet of notes holding the story so far. Read the next page, reconcile it with your notes, rewrite the notes. When you finish, your entire understanding of the book lives on that one sheet.

That is the whole of a recurrent neural network. Read the sequence from the front, one element at a time, and update a state — the notepad — at every step. The notepad has a fixed size chosen in advance; it does not grow whether you read 100 pages or 1,000. That single design choice produces both the strength and the weakness of RNNs.

The definition: apply the same function to yourself, repeatedly

Write the input at time $t$ as $x_t$ and the state at that moment as $h_t$. The plainest RNN is one line.

$h_{t-1}$ is the previous state (the initial $h_0$ is usually the zero vector), $W_h$ is the matrix that carries a state into the next state, $W_x$ brings the input into the state space, $b$ is a bias, and $\tanh$ squashes values into $-1$ to $1$. If you need an output, read it off the state with $y_t = W_y h_t$.

Put in words, that single line says: the new notepad is the old notepad and the page you have just read, mixed together and then squeezed back into the range $-1$ to $1$ so no entry can run away. Two multiplications, one addition, repeated once per step. There is nothing else in it.

The thing not to miss: there is no separate set of weights per time step. The same $W_h, W_x, b$ are reused at every step. That is why the parameter count does not depend on sequence length, and why one network handles a five-word sentence and a five-hundred-word one. Variable length comes from weight sharing, nothing else.

import numpy as np

def rnn_step(x, h, W_x, W_h, b):
    return np.tanh(x @ W_x + h @ W_h + b)    # advance the state by one step

def run(xs, h, W_x, W_h, b):
    for x in xs:                             # through the sequence, one at a time
        h = rnn_step(x, h, W_x, W_h, b)
    return h                                 # the notepad, once you have finished

Training: unroll it in time and it is just a deep network

Redraw the recurrence side by side, once per time step, and a 50-step sequence becomes a 50-layer deep network. The only difference is that every layer shares the same weights. So ordinary backpropagation applies unchanged, and applying it along the time axis has a name: BPTT, backpropagation through time. The name is grand; the content is the chain rule on an unrolled diagram.

Why the gradient vanishes: the same matrix, over and over

Trace how a loss at the final step $T$ depends on a much earlier state $h_t$ and the chain rule hands you a product.

$D_k$ is a diagonal matrix holding the slope of the activation at step $k$. The structure matters more than the symbols: the same $W_h$ appears $(T-t)$ times in that product.

The left-hand side is a sensitivity — nudge a state far in the past, and how much does the notepad at the end move? — and the right-hand side, which says how you get that number, is the effect of one single step multiplied by itself $(T-t)$ times. Think of a whisper passed down a line where each person repeats it at 0.8 of the volume: ten people later, what arrives is $0.8^{10}$ of the original. Same structure.

Shrink it to a scalar and the point becomes obvious. Multiplying $a$ by itself $n$ times gives $a^n$, which decays exponentially to zero when $|a|<1$ and blows up exponentially when $|a|>1$. Matrices behave the same way, governed by the largest singular value of $W_h$ — the most it can stretch a vector. If that value times the activation slope sits below one, gradients die exponentially; above one, they explode. Nothing keeps it politely near one.

The activation is no help either. The slope of $\tanh$ peaks at 1 at the origin, and the sigmoid $\sigma$ peaks at $0.25$. Push either into saturation and the slope falls to almost nothing. So $D_k$ is usually a diagonal of numbers below one, and every multiplication shrinks.

The consequence is that a plain RNN can learn dependencies ten steps back but is hopeless at linking a word 100 steps back to the current one. The intuition — "keep rewriting the notepad and the earliest entries fade under the overwrites" — is exactly what that product says.

The opposite failure exists too. When the singular value exceeds one, gradients swell exponentially, one update blows the weights apart, and the loss becomes NaN. That is the exploding gradient, and the fix is blunt: if the norm of the gradient vector exceeds a threshold, rescale it down to the threshold. Gradient clipping is near-mandatory equipment when training RNNs.

LSTM: build a clear road for the gradient

The LSTM (Long Short-Term Memory), proposed in 1997, did not fix this by tweaking the activation. It rebuilt the path the information travels along.

The key move is splitting the state in two: the hidden state $h_t$ that goes to the outside world, and the cell state $c_t$, an internal memory. The cell update is this and nothing else.

$\odot$ is elementwise multiplication. $f_t$ is the forget gate (how much of the old memory to keep), $i_t$ the input gate (how much of the new candidate to admit), and $g_t$ the candidate content itself.

The line, in words: the memory now is whatever the model decided to keep, plus whatever it decided to write down. The old memory is not stirred back through a matrix; it is faded by a dial and the new material is added on top. That addition is the whole point of the next paragraph.

Here is the crux. In a plain RNN, $h_{t-1}$ always had to pass through the dense matrix $W_h$ before reaching the next step. The LSTM cell state is multiplied by $f_t$ and added, with no matrix in the way. So $\partial c_t / \partial c_{t-1}$ is simply the diagonal $f_t$. In dimensions where the forget gate stays near one, the gradient passes through dozens of steps without decaying or rotating. Think of it as doing along the time axis what residual connections do in a deep stack.

The gates are all built from the previous state and the current input.

$$f_t = \sigma(W_f[h_{t-1}, x_t] + b_f),\quad i_t = \sigma(W_i[h_{t-1}, x_t] + b_i)$$

Those two lines, in words: look at the previous notepad $h_{t-1}$ and the page just read, $x_t$, and decide on the spot how much of the old memory to throw away ($f_t$) and how much of the new material to let in ($i_t$) — each as a dial set somewhere between 0 and 1.

$$o_t = \sigma(W_o[h_{t-1}, x_t] + b_o),\quad g_t = \tanh(W_g[h_{t-1}, x_t] + b_g)$$

The other two work from exactly the same ingredients, which says how much to show the outside world right now ($o_t$) and what the new material would be ($g_t$). All four read the same input; only the weights differ — four judgements pulled in parallel out of one look at the same thing.

$[h_{t-1}, x_t]$ is the two vectors concatenated, and $\sigma$ is the sigmoid. The final output is

where $o_t$, the output gate, chooses which part of the stored memory to show the outside world at this instant.

That last line, in words: what leaves the cell is the internal memory tidied back into $-1$ to $1$, with only the dimensions the output gate has opened allowed through. The memory itself ($c_t$) and the summary shown outside ($h_t$) are deliberately two different things, and that separation is what this line is for.

The sigmoid in the gates is not an arbitrary choice. Its output lies between 0 and 1, so a gate behaves like a valve: 0 shuts it completely (discard, admit nothing, show nothing), 1 opens it fully. As the figure above lets you confirm, the sigmoid is gentle in the middle and saturates at both ends — a continuously adjustable opening, which is precisely what was wanted. This is one of the rare cases where the choice of activation function is the architectural idea.

def lstm_step(x, h, c, W, b):
    z = np.concatenate([x, h]) @ W + b       # all four gates in one matmul
    f, i, o, g = np.split(z, 4)
    f, i, o = sigmoid(f), sigmoid(i), sigmoid(o)
    c = f * c + i * np.tanh(g)               # the addition — the clear road
    h = o * np.tanh(c)
    return h, c

A standard implementation trick is to initialise the forget-gate bias $b_f$ to a positive value, tilting the model towards "do not forget" at the start, so the cell state does not collapse to zero every step early in training.

The GRU is the trimmed version: two gates instead of three, with $c$ and $h$ merged into one. Fewer parameters and faster training, at a small cost in expressiveness — that has been the usual basis for choosing between them.

Why Transformers took over

The reason is less about accuracy than about parallel training.

First, sequentiality. Computing $h_t$ requires $h_{t-1}$, so a sequence of length $N$ forces $N$ sequential steps and leaves a GPU — a parallel machine — largely idle. Self-attention processes every position in a single matrix multiplication, so training is fully parallel within a sequence.

Second, distance. In an RNN, information from step 1 passes through 99 transformations to reach step 100. The LSTM cell state softens this but does not remove the penalty. Self-attention connects any two positions in one step.

Third, and as a consequence of the first two, the architecture suited scaling to large models and large datasets. Being parallelisable simply meant you could train more.

There is a bill to pay. Self-attention costs compute and memory quadratic in sequence length, and the KV cache grows as generation proceeds. An RNN's inference memory is one state vector, no matter how long the sequence.

Where recurrent models still win

Very long sequences. Quadratic cost simply does not survive hundreds of thousands of steps. An RNN's cost is linear in length.

Streaming. One state update per incoming sample keeps a current answer available at all times. Real-time speech recognition, sensor anomaly detection, always-on keyword spotting — inputs that never end — are exactly where a constant-cost update matters. Doing the same with a Transformer means re-cutting a window or managing a cache that keeps growing.

Embedded and edge. A fixed-size state means the memory ceiling is known in advance. A growing cache makes worst-case sizing awkward. On a device with kilobytes of RAM, that predictability decides the design.

And the idea is not a museum piece. Recent state-space architectures make the recurrence linear so it can be computed in parallel during training and run as a constant-memory recurrence at inference — surgically removing the one weakness, parallelism, while keeping the rest. That lineage descends directly from what this article covers. Learning RNNs is not learning history.

How this shows up on the job

The people who actually touch RNNs and LSTMs are those doing time-series anomaly detection and sensor processing, those running speech or keyword spotting on edge devices, and those maintaining existing systems. Do not dismiss the last group: a great many demand-forecasting and audio pipelines built between roughly 2016 and 2020 are still LSTMs today.

In PyTorch the knobs are nn.LSTM with hidden_size (how big the notepad is), num_layers (how many stacked), bidirectional, batch_first and dropout. In the training loop, torch.nn.utils.clip_grad_norm_(params, max_norm) is almost always present. Variable-length batches go through the pack_padded_sequence / pad_packed_sequence pair. In Keras, LSTM(units, return_sequences=..., stateful=...) plays the same role.

Four traps worth knowing about.

Feeding padding straight in. Pad short sequences with zeros to build a batch and the model dutifully reads the padding and updates its state on it. Either pack the batch or at minimum extract the state at the last valid step. Get this wrong and accuracy quietly drops on data with uneven lengths.

Using bidirectional=True for streaming. A bidirectional LSTM also reads backwards, which means it sees the future. Powerful for offline batch processing, structurally impossible for real-time inference — and the classic way to discover this is at deployment, after offline validation looked great.

Not managing carried-over state. A stateful setup passes the state across batches. Forget to reset it at an epoch or dataset boundary and memories from unrelated sequences leak in.

Turning off gradient clipping. Training runs fine for a few hundred steps and then the loss becomes NaN. It is nearly always this.

Two questions come up repeatedly in interviews and design reviews. "Why is an LSTM resistant to vanishing gradients?" — because the cell state has a path that skips the dense matrix and is merely multiplied by the forget gate and added. And "compare an RNN and a Transformer in compute and memory as a function of sequence length $N$." — training is $N$ sequential steps and unparallelisable for the RNN, $N^2$ but parallel for self-attention; at generation time the RNN's state is constant while the Transformer's cache grows in proportion to $N$. Those two axes explain both why Transformers won and why recurrence has not disappeared.

Summary

• An RNN is nothing more than "update a fixed-size state with the same function, one step at a time". Weight sharing is what makes variable length possible
• Vanishing gradients are the product of the same matrix and sub-one activation slopes, repeated once per time step
• The LSTM carved out an additive path through the cell state so gradients can travel undecayed. The three gates work as valves thanks to the sigmoid's 0-to-1 opening
• Transformers won mainly on parallel training and short paths; the bill is quadratic cost and a growing cache
• On very long sequences, streaming, and embedded targets, a constant-memory recurrence still pays

Next we turn to the field this recurrent idea grew up in — time-series forecasting, starting from the classical methods.