Building the Assumption into the Structure — Convolution and Recurrence

Textbook › Part III Deep Learning: what assumption you add

CHAPTER 5

Building the Assumption into the Structure — Convolution and Recurrence

FREE13 min

Chapter 2 listed "lower the capacity" among the remedies for overfitting — with the catch that lowering it too far leaves you short. Squeezed from both sides, it was a poor bargain.

This chapter is the third way: not reducing capacity but reducing it only in the right directions, by building what you know about the data into the structure itself. This is what Chapter 1 meant by "from Chapter 5, which assumption to build into the structure depends on the shape of the data."

1. A fully connected layer pays for assuming nothing

A fully connected layer mapping a $28 \times 28$ image to 128 features has $784 \times 128 = 100{,}352$ weights (plus 128 biases).

That layer knows nothing about how the pixels are arranged. Shuffle the pixels into a fixed random order before training and it reaches the same accuracy, because nothing in the structure says the arrangement means anything.

It pays twice for that.

Parameter count. At $224\times224$ input it becomes $6{,}422{,}656$ (still to 128).
A shifted image is a different object. Move the cat one pixel right and every incoming number is different. The only way to cope is to have seen shifted examples.

2. Convolution — three assumptions at once

Slide one small window of shared weights across every position. In one dimension:

y_i \;=\; \sum_{j} k_j \, x_{i+j-m}

(1)

Three assumptions are built in:

Locality — distant pixels are not directly related (nothing outside the window is seen)
Weight sharing — the same feature is detected with the same weights wherever it appears
Translation equivariance — shift the input and the output shifts with it

f\big(\mathrm{shift}_s(x)\big) \;=\; \mathrm{shift}_s\big(f(x)\big)

(2)

The parameter counts change:

layer	weights + biases
fully connected $784 \to 128$	100,480
convolution $3\times3\times1 \to 32$ ch	320
convolution $5\times5\times1 \to 32$ ch	832

About 314 times fewer — and a convolution's count does not depend on the input size. At $224\times224$ or $1024\times1024$ it is still 320, while the fully connected layer goes to $6{,}422{,}656$ and $134{,}217{,}856$ .

Small windows still reach far once stacked. With $3\times3$ over $L$ layers, one output sees $1 + 2L$ pixels (its receptive field):

layers	1	2	4	8
receptive field (per side)	3	5	9	17

One pixel further each way per layer. Depth is also the mechanism for covering a wide area with small windows.

3. How true is "shifted gives the same answer"?

Test equation (2) by computing it two ways. A: shift, then apply. B: apply, then shift.

FIG 1Above, the input; below, the same output computed in both orders. Overlapping curves mean equivariance holds. With convolution and wrap-around edges they overlap at any shift. Switch to fully connected and the two separate. Switch the edges to zero-padding and it breaks exactly on the shifts where the bump straddles the boundary

Measured (length 32, kernel $(0.25, 0.5, -0.25)$ ):

shift	bump at	conv (zero-pad)	fully connected
1	11–	0	0.8583
7	17–	0	0.6100
18	28–	0.1500	—
22	0–	0.0750	—
25	3–	0	—

(The output's own scale is 0.5155, so the fully connected gap of 0.86 means "a different thing entirely.")

Wrap-around is always exactly zero; zero-padding breaks precisely on the shifts where the bump straddles the edge. Equivariance is not something convolution has by itself — it holds only once the edge handling is specified. Real image models use zero-padding, so equivariance is broken near the borders — a fact textbook diagrams tend to hide.

Building an assumption into the structure means paying for it in weights. When the assumption holds, the same job gets done with far fewer parameters and shifts you never saw are handled from the start. When it does not, the assumption becomes a mistake you cannot train away — a convolution over shuffled pixels is meaningless.

4. Sequences — sharing weights along time

Sequences of varying length need a different assumption: the same rule applies at every time step. Written as weight sharing,

h_t \;=\; \varphi\big(W h_{t-1} + U x_t\big)

(3)

$W$ is one single matrix, independent of $t$ , which is what lets any length be processed.

In exchange, the product from Chapter 3 reappears along time:

\frac{\partial h_T}{\partial h_1} \;=\; \prod_{t} W^{\top} \mathrm{diag}\big(\varphi'(z_t)\big)

(4)

Same shape as stacking layers, except the same $W$ is applied $T$ times. Equation (6) of Chapter 0-1 ( $A^k = P\Lambda^k P^{-1}$ ) applies directly, and eigenvalues come out raised to the $T$ .

Dropping the activation to isolate it, $\rho^T$ appears cleanly ( $\rho$ = largest absolute eigenvalue of $W$ ):

$T$	$\rho=0.8$	$\rho=1.0$	$\rho=1.2$
10	$8.2\times10^{-2}$	0.761	4.71
50	$4.9\times10^{-6}$	0.345	$3.1\times10^{3}$
100	$7.2\times10^{-11}$	0.354	$2.9\times10^{7}$

matching $1.2^{100} = 8.3\times10^{7}$ in order of magnitude. Below 1 it vanishes, above 1 it explodes — in the linear case.

Put $\tanh$ back and it changes:

$\rho$	$\lVert$ gradient $\rVert$ at $T=100$	mean $\tanh'$
0.8	$1.3\times10^{-12}$	0.966
1.0	$9.9\times10^{-5}$	0.927
1.2	$1.4\times10^{-1}$	0.834
2.0	$8.2\times10^{4}$	0.488

At $\rho = 1.2$ it does not explode. Raising $\rho$ enlarges $z$ , which drops $\tanh'$ and cancels it. Chapter 3's conclusion — what matters is the product of (weight size) × (activation slope) — holds here in exactly the same form.

5. Gates — multiplying by a learned "how much gets through"

If the trouble is a product repeated $T$ times, then make the factors something you can push toward 1. Learn how much passes:

c_t \;=\; f_t \circ c_{t-1} \;+\; i_t \circ g_t

(5)

$f_t$ (forget gate) and $i_t$ (input gate) lie in $[0,1]$ , and $\circ$ is the Hadamard product — the second of the two places Chapter 0-1 named as where it takes the leading role.

The gradient along this path is

$\frac{\partial c_T}{\partial c_1} \;=\; \prod_t f_t$

$W$ does not appear at all. Not a product of weight matrices but a product of gate values, and if $f_t$ stays near 1 the signal survives any $T$ .

$f$	$T=20$	$T=100$
0.90	0.122	$2.7\times10^{-5}$
0.95	0.358	$5.9\times10^{-3}$
0.99	0.818	0.366
1.00	1.000	1.000

The plain RNN gave $9.9\times10^{-5}$ at $T=100$ even with $\rho=1.0$ . At $f=0.99$ it is 0.366 — more than three thousand times more.

The shape $c_t = f\circ c_{t-1} + \cdots$ should look familiar: it is the additive path of the residual connection $y = x + F(x)$ from Exercise 3 of Chapter 0-2. Connect everything multiplicatively and the product decays it away; run one additive path through and that route passes at a factor of 1. LSTMs and residual connections are doing the same single thing.

A common misreading: "convolution is for images, recurrence is for sequences"

Look at which assumption got built in rather than at the tool's name:

structure	assumption built in	axis shared
fully connected	none	(nothing shared)
convolution	only nearby things relate / the rule is the same at every position	space
recurrence	the rule is the same at every time step	time
attention	where to look is decided per input	(weights shared, connections variable)

Convolution in one dimension serves audio and text (and is used there); recurrence can scan an image. The criterion is not the kind of data but whether the assumption holds.

Build in an assumption that does not hold and the capacity you gave up is simply lost. Tabular data is the classic case — where column order means nothing, a convolution gives a different answer when you reorder the columns, precisely the opposite of the property it was supposed to encode.

Nor does dropping assumptions buy freedom for free. Vision Transformers dropped convolution's assumptions — and need far more data in return. As Chapter 2's tables showed, assumptions (capacity limits) and data volume trade against each other. Drop an assumption and you pay for it in data.

Exercises

The answers are folded away. Work them first.

Exercise 1. Input images of $64\times64$ . (a) Parameters for a fully connected map to 256 features. (b) For a $3\times3\times1\to256$ ch convolution. (c) What happens to each at $256\times256$ input?

Show the answer

(a) $64^2 \times 256 + 256 = 1{,}048{,}832$ .

(b) $9 \times 256 + 256 = 2{,}560$ — about 410 times fewer.

(c) Fully connected: $256^2 \times 256 + 256 = 16{,}777{,}472$ (16× more). Convolution: still 2,560. A convolution's count depends only on the window size and channel count, not on the input size — the direct consequence of reusing the same weights everywhere.

Exercise 2. Stacking $3\times3$ convolutions. (a) How many layers for a 31-pixel receptive field? (b) With $5\times5$ ? (c) For a given receptive field, which uses fewer parameters: one large window or several small ones (single channel)?

Show the answer

(a) 15. From $1 + 2L = 31$ .

(b) The field is $1 + 4L$ , so $L = 7.5$ → 8 layers (reaching $1+32 = 33$ ).

(c) Several small ones. For a 9-pixel field: one $9\times9$ costs 81, four $3\times3$ cost $9\times4 = 36$ . And with activations between the layers, the stack expresses more shapes at the same receptive field. That is why $3\times3$ dominates — at the price of more layers, which worsens the product problem of Chapter 3.

Exercise 3. A length-200 sequence through a plain RNN whose $W$ has largest absolute eigenvalue 0.9. (a) Roughly what factor does the gradient pick up (treating it as linear)? (b) With a gate holding $f = 0.995$ ? (c) Why is (b) possible at all?

Show the answer

(a) $0.9^{199} \approx 7\times10^{-10}$ — effectively zero. The start of the sequence cannot influence the loss at the end.

(b) $0.995^{199} \approx 0.37$ — over a third survives.

(c) Because what gets multiplied changed. In the plain RNN the factor is $W^{\top}\mathrm{diag}(\varphi')$ , and $W$ is a single matrix for the whole sequence, forcing the same contraction at every step. The gated path multiplies only $f_t$ , which is computed from the input, per step and per component, so it can say "keep this ( $f\approx1$ )" here and "forget it ( $f\approx0$ )" there. Being able to push toward 1 only where it matters is the difference.

What this chapter does not say

The full LSTM/GRU equations are not written out. Equation (5) isolates the cell-state path; the real thing adds an output gate and the construction of $f_t, i_t, g_t$ . Those are left to the article.
Pooling, stride and dilation are not covered; the receptive-field arithmetic assumes stride 1. Pooling adds translation invariance (not equivariance), which is a different story.
Attention gets one row in a table. Deciding where to look per input is the subject from Chapter 6 onward.
The RNN measurements are single runs at width 64 with one fixed initialisation. The form — $\rho$ multiplied by the slope — is what holds; the values move with the setup.
Every number was computed and checked (parameter counts, receptive fields, the two-order gap for wrap/zero-pad/fully-connected, the linear RNN's $\rho^T$ against $1.2^{100}$ , the gradients and mean $\tanh'$ with the activation in place, and the gate's $f^T$ ).