Video Compression from Scratch — Motion Compensation and the GOP

Compressing thirty separate photographs a second gets nowhere near today's delivery bitrates. Why frame differencing breaks, what a motion vector actually carries, I/P/B frames and the GOP, and what CBR, VBR and CRF each hold constant — ending at the keyframe interval a delivery engineer has to choose.

The metaphor: "same as before, but the man on the left took a step right"

Imagine narrating a security camera feed down a phone line. Nobody describes the whole picture thirty times a second. You say "car park, three white cars" once, and after that it is "same as before, but the man on the left took a step right". Video compression is, at bottom, that idea.

Check the orders of magnitude. 1920×1080 is about 2.07 million pixels; at three bytes each that is 6.2 MB per frame and 186 MB every second at 30 fps — call it 1.5 Gbps. Encode each frame as an independent JPEG and, even at a tenth of the size, you are still near 150 Mbps. Real 1080p delivery usually fits in a pipe two orders of magnitude narrower. Stacking JPEGs does not come close.

What closes the gap is temporal redundancy. Consecutive frames are a thirtieth of a second apart and most pixels keep the value they had. Still-image compression removes redundancy inside a picture; video removes it between pictures, where there is far more of it.

The obvious idea: send the difference

The simplest move is to send the subtraction.

In words: at every pixel position, take the value it has now, subtract the value that same position had one frame ago, and send only what is left over.

$f_t$ is the frame at time $t$ and $d_t$ is the difference — a picture of "what changed since last time". With a tripod-mounted camera and one person walking, it is almost entirely black, glowing only around the person, and a long run of zeros compresses very well.

It collapses the instant the camera moves. Pan by a single pixel and every edge in the frame turns up in the difference. When a subject moves, both the place it left and the place it arrived appear. The content has not changed. It has only shifted.

Motion compensation: sending "where it came from"

If it has only shifted, shift it back before subtracting. That is motion compensation.

Split the frame into blocks (in H.264 the basic unit is a 16×16 macroblock) and, for each one, search the previous frame for the closest-looking patch. The offset you find is the motion vector $(m_x, m_y)$.

Which says, in everyday terms: look up the previous frame not at the same address, but at the one offset by $(m_x, m_y)$ — the spot this block's contents were sitting in a moment ago — and subtract that instead. What survives is only the part that the shift alone cannot account for.

The second term on the right is the prediction — the previous frame, shifted. The $r_t$ on the left is the residual. A 256-pixel block has become two numbers plus whatever little is left over.

The important part is that how you search is outside the standard. The standard fixes how a vector is transmitted and how sub-pixel positions are interpolated (most codecs predict at half- and quarter-pixel precision); finding the vector is the encoder's business. An exhaustive search over radius $S$ costs $(2S+1)^2$ comparisons per block, which becomes impossible very quickly (the complexity article calls this the quadratic wall), so real encoders use hierarchical searches and local searches seeded from neighbouring vectors. When two encoders differ in quality at the same standard and the same bitrate, this is mostly where the difference lives.

The residual is also just an image

So what is $r_t$? An image. If the prediction was good, it is a flat, empty-looking image whose pixels sit near zero.

Which means the image toolkit applies unchanged. The residual is transformed block by block (from H.264 onward with an integer-defined approximation of the DCT, so encoder and decoder never disagree about the inverse), quantized, then entropy coded — the same path as the JPEG article.

Only the input differs. Video compresses well not because its transform or its entropy coder is special, but because the picture entering the transform is already close to empty.

I, P and B frames, and the GOP

There are three kinds of frame. An I-frame (intra) references nothing else and stands on its own; it is essentially still-image compression, and it is the largest. A P-frame (predicted) references the past. A B-frame (bi-predictive) references both past and future.

Why does B help? First, occlusion. The wall revealed from behind a walking person is not in the previous frame at all, so looking backwards there is nothing to predict from — but in a later frame that wall is plainly visible. Second, averaging: combining a forward and a backward prediction cancels part of the error in each, shrinking the residual further.

The price is reordering. To reference the future you must decode it first, so a display order of I B B P is transmitted and decoded as I P B B. The decoder needs a buffer to put frames back in order, and that buffer is latency. Cutting B-frames for live streaming is a latency decision, not a quality one.

A GOP (group of pictures) runs from one I-frame to just before the next, and its length — the keyframe interval — is the number you touch most often in production. Reference structures have grown more elaborate over the generations, with multiple reference frames and pyramid arrangements in some implementations.

What a scene change does

The moment there is a cut, temporal correlation vanishes. Motion search finds no similar patch anywhere, the residual is about as large as the picture itself, blocks fall back to intra prediction one after another, and the bit count balloons. A P-frame whose prediction fails can cost more than an honest I-frame.

So encoders detect scene changes and place an I-frame there, judging from inter-frame difference statistics or from the cost the motion search reports. When detection fires, the GOP is cut shorter than configured. Useful by default — and squarely in the way when you need to choose keyframe positions yourself.

Rate control: what is being held constant

Rate control is the design of where to spend quantization parameter (QP). QP is the coarseness of the quantizer, the video equivalent of JPEG's "quality", except that video can vary it per frame and per block. CBR, VBR and CRF differ in what they hold constant.

CBR holds the bitrate constant. It suits a fixed pipe or broadcast channel that must not be exceeded. The cost is quality: busy scenes are forced into the same bit budget, QP rises, and the picture visibly falls apart.

VBR targets an average bitrate and redistributes bits between scenes, taking from quiet passages to pay for busy ones, so quality is steadier. In practice you pair it with a ceiling (-maxrate) and a buffer size (-bufsize) so the instantaneous rate never outruns the network.

CRF (constant rate factor) holds quality constant and lets the bitrate fall where it may. It is a mode offered by encoders such as x264 and x265, where a smaller number means higher quality and a larger file (x264 defaults to 23; the same number means something different in another encoder). It matches how people actually watch, but you cannot know the output size in advance.

Behind all three sits a buffer constraint: the decoder's input buffer must neither starve nor overflow. That, in practice, is why -maxrate and -bufsize accompany even CRF and VBR.

How this shows up on the job

Five things a delivery engineer actually sets.

Align the keyframe interval with the segment duration. HLS and DASH cut video into segments of a few seconds, and players start decoding at a segment boundary, so every segment must begin with an I-frame. Set the keyframe interval to the segment duration. In ffmpeg that means matching -g (GOP length in frames) with -keyint_min and stopping the encoder from inserting keyframes on its own (-sc_threshold 0 or scenecut=0 for x264, or -force_key_frames to place them explicitly). Get it wrong and the packager splits mid-GOP, wasting the bits in the truncated part. Worse, if the rungs of an ABR ladder do not share keyframe positions, the picture jumps or stalls every time the player switches rendition.

The GOP length trade-off itself. A longer GOP means fewer I-frames and fewer bits for the same quality. It also means fewer seek points: the player must go back to the previous I-frame, so seeking and joining mid-stream both get slower. It is a trade between coding efficiency and how fast you can seek and start, with no universally correct answer. A few seconds matched to the segment duration for VOD, shorter for low-latency live, is the usual landing spot.

Recognise when two-pass earns its keep. A first pass measures the complexity of every frame and a second distributes bits accordingly. It pays when the final size or average bitrate is fixed up front — a contracted bandwidth, a target rate per ladder rung, an archive quota. When size is free, one CRF pass is usually enough, and live encoding cannot use two-pass at all because it cannot see the future.

Do not let re-encoding accumulate. Encode an already-encoded video and quantization error stacks up generation over generation. The second-generation encoder also treats the first generation's blocking and ringing as real detail and faithfully spends bits on it, so you pay more and look worse. The rule is one line: keep the master in a high-quality (ideally intermediate or lossless) format and build every delivery version from that master. Edit → export → burn in subtitles → export again is exactly where this goes wrong.

Judge quality with metrics and with your eyes. Bitrate alone says nothing about quality, so PSNR, SSIM and VMAF all get used (VMAF especially when designing a ladder). But metrics are metrics: the frames right after a cut, and dark gradients, still need a human look.

The design-review question you will get is "why do live streams use fewer B-frames?". Because referencing the future swaps display order and coding order, and that reordering is latency — answer that and you are fine.

Summary

• Video beats a stack of still images because it removes temporal redundancy
• Plain frame differencing breaks the moment anything moves; if it only shifted, shift it back and then subtract — motion compensation. How vectors are searched sits outside the standard, which is where encoders differ
• The residual is just an image, so it rejoins the same transform, quantization and entropy coding as JPEG
• B-frames pull occluded regions out of the future, but the reordering they need becomes latency
• At a scene change prediction fails completely, so the encoder inserts an I-frame
• CBR holds the rate constant, CRF the quality, VBR the average
• In production: match the keyframe interval to the segment duration, keep it identical across the ladder, and never re-encode a re-encode

Next, what actually changed from one generation to the next — H.264 through AV1.